JP2001197296A - Image processing method and image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing method and an image processor for opening the closed eyes of a portrait image, correcting the failure and outputting an image having the eyes opened. SOLUTION: When the prescribed image processing is applied to the inputted image data and these data are outputted, the image processing includes correction to open the closed eyes in the image.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing method and apparatus for correcting a face image of a person with closed eyes into a face image with opened eyes and outputting the corrected face image.

[0002]

2. Description of the Related Art In photography, sometimes a person blinks at the moment of photography or closes his eyes by flash photography, resulting in a photograph with closed eyes. In particular, in the case of a portrait photograph, eyes are an important factor, and a photograph with closed eyes is not preferable in terms of photographic quality. It is good if re-taking is possible, but if re-taking is difficult, it is desirable to correct such a shooting failure. In such a case,
Conventionally, a skilled person has performed retouch correction using a pen.

[0003]

However, the conventional retouching method using retouching using a pen relies on the manual work of a skilled worker, and therefore takes time and effort, and the cost of photographs is high. There was a problem that would be. The present invention has been made in view of the above-described conventional problems, and performs processing to open eyes on a human image with closed eyes, corrects a failure in photographing, and outputs an image with opened eyes. An object of the present invention is to provide an image processing method and apparatus capable of performing the above.

[0004]

According to a first aspect of the present invention, there is provided an image processing method for performing predetermined image processing on input image data and outputting the image data. An image processing method, wherein the image processing includes an eye correction process of correcting a person image in a state in which the eyes are closed with an image of the eyes in a state in which the eyes are closed. It is.

[0005] Preferably, the eye correcting process is a synthesizing process for synthesizing an image of an eye with closed eyes with an image of an open eye of the same person or a person other than the same person. Preferably, a plurality of samples are prepared in advance, and the synthesis process is to synthesize the selected eye from the closed eye.

It is preferable that the synthesizing process includes a process of adjusting the size, angle, eyelid, pupil and color around the eye and density of the image to be synthesized so as to match the image to be synthesized.

The color density around the eyes of the image to be synthesized is
It is preferable that the adjustment is automatically performed based on at least one of image features such as the position of both eyes, the distance between both eyes, the size of eyes, and the size of a face.

Preferably, the adjustment is manually performed by an operator based on a menu for changing the eye size, angle, color, density, and aspect ratio of the image to be synthesized. Here, the aspect ratio refers to the ratio of the length and width of the eye portion.

Further, the image in which the eyes are closed and the image in which the eyes are open are compared, and the movement of a point on the eyelid is estimated based on the characteristic of the shape of the eye image. It is preferable to perform a correction process. Also, using the image with the eyes closed as an input signal and the image with the eyes open as a teacher signal, the image conversion from the closed eyes to the open eyes is learned, and the image conversion based on the learned conversion is performed. It is preferable to perform the eye correction processing for opening the closed eyes.

It is preferable that the degree of opening the eyes can be adjusted in the eye correction processing.

[0011] Further, by designating the length of the eyelashes, the direction of the eyelashes, the single eyelid, and the double eyelid, it is possible to correct an image in which the eyes are closed to an image in which predetermined eyes are open. preferable.

Further, it is preferable that the eye correction processing further has a retouch function.

[0013] It is preferable that the eye correction process uses features of an open eye and surroundings when correcting an image of a closed eye when only one eye is closed.

The peripheral features include the color and size of the pupil, the length of the eyelashes, the skin color around the eyelids and the eyes, the single eyelid, the double eyelid, the position of the eyes, the size and shape of the eyes. It is preferable to use at least one piece of information. Also,
It is preferable that the position of the pupil, the position of the pupil, and the shape of the pupil can be adjusted to adjust the line of sight of both eyes.

[0015] Similarly, in order to solve the above-mentioned problems,
A second aspect of the present invention is an image processing apparatus that receives image data from an image input unit, performs predetermined image processing, and outputs the image data to an image output unit. The present invention also provides an image processing apparatus comprising eye correction processing means for correcting the closed eyes to an image of an open eye. Further, the second aspect of the present invention
In the above-mentioned image processing apparatus, the image processing apparatus may further include an image display device, wherein the eye correction processing unit causes the image display device to open the closed eyes while the eye correction process is being performed. The present invention provides an image processing apparatus characterized in that control is performed so that an enlarged image around the image can be displayed.

[0016]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an image processing method and apparatus according to the present invention will be described in detail based on preferred embodiments shown in the accompanying drawings.

FIG. 1 is a block diagram showing an embodiment of a digital photo printer to which an image processing apparatus for implementing the image processing method of the present invention is applied. A digital photo printer (hereinafter referred to as a photo printer) 10 shown in FIG.
A scanner (image reading device) 12 that photoelectrically reads an image captured on the film F, and an image that performs image processing on image data (image information) read by the scanner 12 and operation and control of the entire photo printer 10 Processing device 14 and a photosensitive material (photographic paper) using a light beam modulated according to image data output from the image processing device 14
And an image recording device (printer) 16 for exposing the image to an image, developing the image, and outputting a (finished) image as a print. The image processing apparatus 14 includes an operation system 18 having a keyboard 18a and a mouse 18b for inputting various conditions such as input, setting, selection and instruction of processing, and instructions for color / density correction, and the like. Images,
A monitor 20 for displaying various operation instructions, setting / registration screens for various conditions, and the like, and performing image verification while the operator views them, is connected. The image processing apparatus 14 is also connected to image data input / output means 40 such as image data recording / reproducing means from a medium and image data distribution means via communication means such as a network. The image data may be input through the image data input / output means 4.
0 may be output.

The scanner 12 is a device that photoelectrically reads an image photographed on a film F or the like one frame at a time.
A variable aperture 24, a diffusion box 26 for making the reading light incident on the film F uniform in the surface direction of the film F, a carrier 28 of the film F, and an imaging lens unit 30
, An image sensor 32 having a three-line CCD sensor corresponding to reading of image densities of R (red), G (green), and B (blue), an amplifier (amplifier) 33, and an A / D (analog / digital). ) A converter 34.

In the photo printer 10, a dedicated carrier 28 that can be freely mounted on the main body of the scanner 12 includes a film F such as a new photographic system (advanced photo system) or a 135 size negative (or reversal) film.
Are prepared in accordance with the type and size of the film, the form of the film such as strips and slides, and the like, and can be adapted to various films and processes by replacing the carrier 28. An image (frame) photographed on a film and provided for printing is conveyed by the carrier 28 to a predetermined reading position. Also, as is well known, a magnetic recording medium is formed on the film of the new photographic system, and a cartridge ID, a film type, etc. are recorded. And various data such as the model of the developing machine can be recorded. The carrier 28 corresponding to the film (cartridge) of the new photo system includes:
The magnetic information reading means is arranged, reads the magnetic information when the film is transported to the reading position, and sends these various kinds of information to the image processing device 14.

When reading an image photographed on the film F in such a scanner 12, a uniform reading light emitted from the light source 22 and adjusted in light amount by the variable aperture 24 and the diffusion box 26 is transmitted by the carrier 28. Incident on the film F located at the predetermined reading position,
By transmitting the light, projection light carrying an image photographed on the film F is obtained. The color image signal is not limited to a signal input by reading the light transmitted through the film as described above, and may be a reflection original or an image captured by a digital camera. Further, the image data may be input from the image data input / output unit 40 via a medium or a network.

In the illustrated example, the carrier 28 is a 24
It corresponds to a long film F (strips) such as a 35-size film or a cartridge of a new photo system, and while positioning the film F at a predetermined reading position,
For example, three lines CC of RGB of the image sensor 32
The film F is conveyed in such a manner that the longitudinal direction of the film F coincides with the sub-scanning direction orthogonal to the extending direction (main scanning direction) of the D sensor. The film F is positioned at the reading position by the carrier 28, and is conveyed in the sub-scanning direction, and receives the reading light. The reading light is regulated by a slit extending in the main scanning direction. As a result, the film F is two-dimensionally scanned, and an image of each frame captured on the film F is read.

The projection light of the film F is imaged on the light receiving surface of the image sensor 32 by the imaging lens unit 30. The image sensor 32 is a so-called three-line color CCD sensor having a line CCD sensor for reading an R image, a G image, and a B image, and each line CCD sensor extends in the main scanning direction as described above. are doing. The projection light of the film F is transmitted to the image sensor 3
2, the light is separated into three primary colors of R, G, and B and read photoelectrically. Each output signal of R, G and B output from the image sensor 32 is amplified by an amplifier 33 and sent to an A / D converter 34, where the A / D converter 34 outputs, for example, a 12-bit RGB digital signal. After being converted into image data, it is output to the image processing device 14.

In the scanner 12, when reading an image photographed on the film F, a pre-scan (first image reading) for reading at low resolution and a fine scan (first image reading) for obtaining image data of an output image are performed. (2nd image reading). Here, the pre-scan is performed under preset pre-scan reading conditions so that the image sensor 32 can read all images of the film F to be scanned by the scanner 12 without saturation. On the other hand, the fine scan is performed under the fine scan reading conditions set for each frame from the pre-scan data so that the image sensor 32 is saturated at a density slightly lower than the minimum density of the image (frame). The prescan and fine scan output image signals are basically the same image data except that the resolution and the output image signal level are different.

Note that the scanner 12 used in the photo printer 10 is not limited to one that performs such a slit scan reading, and may be one that performs a planar reading that reads the entire surface of one frame of film image at a time. Good. In this case, for example, an area sensor such as an area CCD sensor is used, and means for inserting R, G, and B color filters are provided between the light source 22 and the film F, and inserted into the optical path of the light emitted from the light source 22. The reading light transmitted through the color filters is irradiated onto the entire surface of the film F, and the transmitted light is imaged on the area CCD sensor to read the entire image of the film by sequentially switching the R, G, and B color filters. Accordingly, the image photographed on the film F is separated into three primary colors and read.

As described above, the digital image data signal output from the scanner 12 is output to the image processing device 14 that performs the image processing method of the present invention. FIG. 2 is a block diagram of the image processing apparatus (hereinafter, referred to as a processing apparatus) 14. The processing device 14 includes a LOG conversion unit 42, a pre-scan data processing unit 44, a fine scan data processing unit 46, and a setup unit 48. In addition,
FIG. 2 mainly shows parts related to image processing. The processing device 14 also includes a CPU that controls and manages the entire photo printer 10 including the processing device 14, an operation of the photo printer 10, and the like. A memory or the like for recording necessary information is provided, and the operation system 18 and the monitor 20
Each part is connected via a PU or the like (CPU bus).

The R, G, and B image signals input from the scanner 12 to the processing unit 14 perform correction of read image data such as DC offset correction, dark time correction, defective pixel correction, and shading correction as necessary. After that, it is input to the LOG converter 42. The LOG conversion unit 42 performs a logarithmic conversion process using, for example, a look-up table (LUT) to convert the digital image data into gradations and convert the image data into digital image density data.

The pre-scan image data and the fine scan image data converted into digital image density data by the LOG conversion section 42 are stored (stored) in respective memories. The pre-scan image data is read by the pre-scan data processing unit 44, and the fine scan image data is read by the fine scan data processing unit 46. Note that image data can also be input to the processing device 14 from the image data input / output unit 40 via a medium, a network, or the like.

The pre-scan data processing section 44 performs various image processing required for displaying on the monitor 20 on the pre-scan image data. The pre-scan data processing section 44 converts the pre-scan image data into gradations of color and density. 50, an eye correction processing unit 52, and a color reproduction conversion unit 54. The gradation conversion unit 50 can reproduce a color image on the CRT display screen of the monitor 20 described later with desired image quality for the image data read by the scanner 12 according to the image processing conditions set by the setup unit 48 described later. In this way, predetermined image processing such as color conversion, density conversion, and gradation correction is performed by a lookup table (hereinafter, represented by LUT) or matrix (hereinafter, represented by MTX) calculation. The eye correction processing unit 52 performs processing on the monitor display image.
The eye correction processing for opening the closed eyes is performed. The color reproduction conversion unit 54 thins out the image data for monitor display that has been subjected to the image processing as necessary in order to match the resolution of the monitor 20, and uses the 3D (three-dimensional) LUT or the like to display the image data on the monitor 20. The data is converted into corresponding image data and displayed on the monitor 20. The processing conditions in the gradation conversion unit 50 are set by a setup unit 48 described later.

On the other hand, the fine scan data processing section 46
Is a portion for performing various image processing necessary for outputting fine scan image data from the image recording device 16 as a color print, processing for opening closed eyes of the present invention, and other image processing; Enlargement / reduction processing unit 56, gradation conversion unit 58, image storage unit 60, eye image conversion unit 62, sharpness processing unit 64, and color reproduction conversion unit 66
have. The enlargement / reduction processing section 56 performs enlargement / reduction processing on an input image according to the image output means, and the gradation conversion section 58 converts the gradation of color and density of image data. The image storage unit 60 is for storing an image with the eyes opened for use in the eye image conversion process. The eye image conversion unit 62 converts the image of the output image with the eyes closed with the eyes open based on the data of the eye correction processing performed by the prescan data processing unit 44. Also, the sharpness processing unit 64
It emphasizes the edges of the image and sharpens the image.
The color reproduction conversion unit 66 converts image data subjected to various types of image processing into image data corresponding to image recording by the printer 16 using, for example, a 3DLUT.

The fine scan data processing section 46
Are set by the setup unit 48. The setup unit 48 sets various image processing conditions in the fine scan data processing unit 46. The setup unit 48 includes a parameter calculation unit 68 and a key correction unit 70. The setup unit 48 sets the scanning conditions of the fine scan using the prescanned image data and the like and supplies the conditions to the scanner 12, and also creates the image processing conditions of the prescan data processing unit 44 and the fine scan data processing unit 46. (Calculation).

More specifically, the parameter calculation section 68 creates a density histogram from the prescanned image data, and calculates an average density, LATD (large area transmission density), highlight (lowest density), shadow (highest density), and the like. Is calculated. From the calculated image feature amount, the scanning conditions of the fine scan, such as the light amount of the light source 22, the aperture value of the variable aperture 24, the image sensor 3, and the like, are set so that the image sensor 32 is saturated at a density slightly lower than the minimum density of the image.
2 is set. The fine scan reading condition may change all the elements corresponding to the output level of the image sensor with respect to the prescan reading condition, or change only one element such as the aperture value. Alternatively, only a plurality of factors such as the aperture value and the accumulation time may be changed. Further, the parameter calculation unit 68 sets image processing conditions such as the above-described color balance adjustment and gradation adjustment according to the density histogram, the image feature amount, and an instruction from an operator performed as necessary.

The key correction unit 70 includes an adjustment amount such as density (brightness), color, contrast, sharpness, and saturation set by a key (not shown) provided on the keyboard 18a and the operation system 18, and a mouse 18b. The amount of adjustment of the image processing condition (for example, LU)
A correction amount of T or the like is calculated and parameters are set. The set parameters indicating the image processing conditions are:
The data is sent to the pre-scan data processing unit 44 and the fine scan data processing unit 46. Further, the key correction unit 70
The image processing conditions set for each part are corrected (adjusted) or the image processing conditions are reset according to the adjustment amount calculated in step (1).

FIG. 3 shows an outline of the eye correction processing section 52.
As shown in FIG. 3, the eye correction processing unit 52 includes an eye opening processing unit 7.
2. It includes an eye feature specifying unit 74 and a manual adjusting unit 76. The eye opening processing section 72 is a part that forms the center of the eye correction processing. There are various methods for opening eyes,
As described below, the eye opening processing unit 72 has a configuration corresponding thereto. The eye feature specifying unit 74 is a part for setting information such as the position and size of the eyes, which is the premise of the eye opening process.
The manual adjustment section 76 is a section where the operator manually performs each of the above adjustments while watching the display on the monitor 20.

The first eye correction method is to synthesize an image of an open eye with an image of a closed eye. At this time, as shown in FIG. 4, the eye opening processing unit 72 includes an eye opening image storage unit 78 for storing an open eye image to be synthesized, and an eye image synthesizing unit 80 for actually performing image synthesis. In.

Hereinafter, the first eye correction processing method will be described. This is because, for example, when an image of the same person whose eyes are open is captured in another frame in one film currently being processed, the image of the open eyes is closed. It is intended to be combined with an image. Therefore, one film is pre-scanned, and for example, the operator designates the film while viewing the display on the monitor 20, and
The image with the eyes open is stored in the eye opening image storage unit 78. In addition, the image with the opened eyes stored in the open-eye image storage unit 78 to be combined with the image of the closed eyes is not limited to the image of the same person on the same film, and may be an image of another film. Even if there is, it may be an eye opening image of a person other than the same person, that is, another person. When the operator examines the image on the display of the monitor 20 to determine the image processing conditions, and there is an image in which the eyes are closed, the eye opening processing is performed according to the order of the user or the judgment of the operator. Is executed.

When the eye opening process is started, first, the eye characteristic amount specifying unit 7
In 4, the size of the eyes is estimated from the size of the eyelids, the distance between the left and right eyes, the size of the face, and the like of the image in which the eyes are closed. Further, the eye feature specifying unit 74 estimates the angle of the eye from the positional relationship between the left and right eyes and the eye and the angle of the face. Based on the information, the eye feature specifying unit 74 selects, from the eye opening image storage unit 78, an image with open eyes that is close to this condition as an image to be synthesized, and sets the size of the image of the open eye to this. Match. Also, the angle of the open eye image to be synthesized
The angle should be adjusted to the eye angle of the combined image. further,
The eye feature quantity specifying unit 74 matches the color and density of the image with the eyes open to be combined based on the color and density around the eyelids of the image with eyes closed, which is the image to be combined.

Next, in the eye image synthesizing section 80, the image with the eyes closed, which has been selected above and processed so as to be appropriately synthesized, is synthesized with the image with the eyes closed. This actually removes the periphery 82 of the closed eye and fits the periphery 84 of the open eye instead, as shown in FIG. Thereafter, in the manual adjustment unit 76, an image adjustment such as an eye position, a color, a density, and the like is performed by an operator giving an instruction while watching the display on the monitor 20.

The above processing may be performed automatically by computer processing where possible, but all processing may be performed manually by the operator while watching the monitor 20. The image of the open eye to be synthesized is not only selected from the image of one film being processed as described above, but also the same customer's eye can be identified by the customer code from the past image data in the database. An open image or a similar person, for example, an image with open eyes such as a sibling may be pulled. In addition, in the synthesis, the adjustment of the size and color density of the image of the open eye to be synthesized with the image of the closed eye, which is the image to be synthesized, is performed by adjusting the color density around the eyes among the features of the image to be synthesized. The eye feature amount specifying unit 74 may automatically adjust as described above based on features such as the position of both eyes, the distance between both eyes, the size of the closed eyes, and the size of the face. Alternatively, the adjustment may be manually performed by an operator based on a menu for changing the size, angle, color, density, and aspect ratio of the combined image.

In this manner, the position data and the image data of the synthesized image (part) of the open eyes synthesized by the eye image synthesis unit 80 of the eye opening processing unit 72 of the eye correction processing unit 52 and adjusted by the manual adjustment unit 76 are obtained. From the eye correction processing unit 52 of the pre-scan data processing unit 44 to the image storage unit 60 and the eye image conversion unit 6 of the fine scan data processing unit 46.
Sent to 2. Note that, instead of the image data of the synthesized image of the opened eyes obtained by the eye correction processing unit 52, the difference between the image data of the synthesized part of the eyes before and after the eye correction by the eye correction processing unit 52 is stored in the image storage unit 60 or the eye. The image data may be sent to the image conversion unit 62, or the image data of the open eyes used for synthesis in the eye image synthesis unit 80, the parameters used for synthesis, and the adjustment parameters used for adjustment in the manual adjustment unit 76. May be sent.

Next, a second eye correction processing method will be described. The second eye correction process performs a conversion based on the characteristics of a two-dimensional shape. Specifically, for example, from the image with the eyes closed, the half-eyed image, and the image with the eyes open, the movement of the symmetry point is modeled, and morphing is performed based on the movement of each modeled part, such as morphing. This is to perform image conversion.

The eye opening processing section 17 in the second eye correction processing
2 is applied to the eye correction processing unit 52 shown in FIG. 3 instead of the eye opening processing unit 72 shown in FIG. 4, and as shown in FIG. 6, an image storage unit 86, a model creation unit 88, an eye image A conversion unit 90 is included. The image storage unit 86 is an image of the same person or a person other than the same person (another person) with closed eyes, a half-eye image, and opened eyes for data for modeling the movement of a point on the eyelid. The stored image and the like are stored. The image data may be stored from the image of one film currently being processed, or if that alone is insufficient, the same customer or another person, such as a sibling, a parent or another person with similar facial features, etc. May be read from the database.

The model creating section 88 reads, for example, an image 92 with closed eyes, an image 94 with half eyes, and an image 96 with eyes opened, as shown in FIG. 7, for the same person or another person from the image storage section 86. And create a model based on these. For example, in the image 92 with the eyes closed, the points P0 and Q0 on the eyelid are designated, and how these points move is tracked. At this time, it is assumed that the points P0 and Q0 are respectively P1 and Q1 in the half image 94, and P2 and Q2 in the image 96 with the eyes opened. From these points, these points become P0 → P1 → P2,
And the state during the movement of Q0 → Q1 → Q2 is estimated.
The other points on the eyelids are also considered to move in the same way, whereby the shape of the eyes during the transition from the closed state to the fully opened state is also estimated, and when the closed eyes are opened,
A model is built of how each point on the eyelid moves.

The eye image conversion unit 90 converts the closed eye image into the open eye image based on the created model. According to this model, it is possible to display an arbitrary state from a state in which the eyes are closed to a state in which the eyes are completely opened, and the eyes can be gradually opened, and the expression according to the scene can be displayed. Can be made. Also, once a model is created, by storing this model, the next time the same customer or another similar person has the opportunity to process a person image with closed eyes, It is possible to easily correct the image with the eyes closed using the model.

In this way, the position data and image data of the converted image (part) to the open eyes converted by the manual adjustment unit 76 and converted by the eye image conversion unit 90 of the eye opening processing unit 172 of the eye correction processing unit 52 Is sent from the eye correction processing unit 52 of the pre-scan data processing unit 44 to the image storage unit 60 and the eye image conversion unit 62 of the fine scan data processing unit 46. In addition, instead of the image data of the converted image to the opened eyes, the difference between the pre-scanned pre-scanned image data and the corrected eye image data of the converted image portion of the eye may be sent, or The image data of a plurality of eyes used in the model creation by the model creation unit 86, the parameters used in the model creation, the conversion parameters used in the conversion to the eyes opened in the eye image conversion unit 90, and the adjustment by the manual adjustment unit 76 The used adjustment parameters may be sent.

Next, a third eye correction processing method will be described. This is to collect the data of the same person or another person's eyes with eyes closed and images with eyes open, use the images with eyes closed as input data, open eyes The conversion rule is learned using the image in the state where the image is in the teaching data, the image is converted based on the conversion rule, and the image with the eyes closed is modified.

The eye opening processing section 27 in the third eye correction processing
2 is applied to the eye correction processing unit 52 shown in FIG. 3 instead of the eye opening processing unit 72 shown in FIG. 4, and as shown in FIG. 8, an image storage unit 98, a conversion rule learning unit 100, An image conversion unit 102 is included. The image storage unit 98 is for storing an image of the same person or another person with their eyes closed and an image with their eyes open. The conversion rule learning unit 100 reads the image from the image storage unit 98, uses the image 104 of the same person or another person with closed eyes as input data (signal) as shown in FIG. The conversion rule is learned by using the image 106 in the present state as teacher data (signal). For example, 1
In a film of a book, when there is a similar person in a plurality of frames, for example, if information indicating the identity or similarity of a person in different frames is included in shooting information or the like, even if the scene is different, Open-eye and closed-eye images of the same person or a similar person can be associated with each other, thereby converting the closed-eye image to an open-eye image. Learn the rules when doing The eye image conversion unit 102 converts a closed eye image into an open eye image according to a conversion rule established as a result of learning.

The system for performing the above-mentioned learning is not particularly limited, but specifically, for example, an approximation using a neural network, a GA (generic algorithm), a high-order polynomial, or the like can be considered.

Next, a fourth eye correction processing method will be described. In this method, image conversion is performed based on the result of the dynamic motion analysis of the human eye and eyelid opening / closing motion. The eye opening processing unit 372 in the fourth eye correction processing is applied to the eye correction processing unit 52 shown in FIG. 3 instead of the eye opening processing unit 72 shown in FIG. 4, and as shown in FIG. -Eyelid data extraction unit 108, mechanical motion analysis unit 110, eye image conversion unit 1
12 is included.

The eye / eyelid data extraction unit 108 designates a plurality of points on the eyes and eyelids for the next dynamic motion analysis. The mechanical movement analysis unit 110 includes the eye / eyelid data extraction unit 1
A dynamic motion analysis is performed on the point designated by 08 to create an image with the eyes open.
Specifically, the eyelids and eyelash movements are analyzed and formalized based on the positions and movements of the structure, tissue, and muscles of the human eye. The eyelids and eyelashes are recognized from the image with the eyelids closed, or the eyelids and eyelashes are specified by an operator's instruction, and an operation of opening the eyes is performed based on the equation of motion to obtain an image with the eyes opened.

The eye image conversion unit 112 may combine the above-obtained image with the eyes open with the image with the eyes closed, or may convert the eyelids based on the analysis result. You may make it open gradually and write a pupil between the opened eyelids. This method seems to be difficult to formulate the equation of motion, but once formulated, the image with the eyes open can be determined fairly accurately just by giving the points on the eyelids.

The converted eyes (parts) obtained by the eye correction processing unit 52 applying the eye opening processing units 272 and 372 for implementing the third and fourth eye correction processing methods (parts)
The position data and image data of the pre-scan data processing unit 44 are obtained in the same manner as those obtained by the eye correction processing unit 52 that applies the eye opening processing units 72 and 172 that execute the second eye correction processing method. It is sent from the correction processing unit 52 to the image storage unit 60 and the eye image conversion unit 62 of the fine scan data processing unit 46. Similarly, instead of the image data of the converted image for the open eyes, a difference between the image data before and after the eye correction may be sent, or the conversion learning unit 100 or the dynamic motion analysis unit 110 Alternatively, the parameters used in step (1), the conversion parameters used in the eye image conversion unit 90, and the adjustment parameters used for adjustment in the manual adjustment unit 76 may be sent.

In the above-described eye correction processing method, by making the position and shape of the pupil and the pupil adjustable, the eyes of both eyes can be adjusted, and as a result, the eyes of both eyes can be matched. Also, in the eye correction processing method, by specifying the eyelash length, eyelash direction, single eyelid, double eyelid, and the like, an eye opening image suitable for the eye opening image is created or synthesized, and a desired image is formed. Can be converted to further,
In the above eye correction processing method, by providing a so-called retouch function in which an operator makes a manual correction, more fine-tuned adjustment becomes possible.

In each of the eye correction processing methods described above, the correction is performed on the monitor 20 while the eye correction processing unit 52 of the pre-scan data processing unit 44 is performing the eye opening processing such as the eye opening processing or the retouching. It is preferable to control so that an enlarged image around the eye to be displayed can be displayed. In FIG.
Display screen 1 for performing the combining process by the eye correction processing unit 52
20 shows an example. The display screen 120 shown in FIG. 5 includes an enlarged image 122 of the periphery of the eye to be corrected shown in FIG.
Are displayed on the right side of the figure, and five representative samples 124a, 124b, 124c, 12
4d and 124e, on the lower side in the figure, a registration reference button 126a, another frame reference button 126b, and an edit button 1
Selection buttons such as 26c and a retouch button 126d are displayed as a GUI (graphic user interface).

As described above, the display screen 120 of the monitor 20
Can display an enlarged image 122 of the periphery of the eye to be corrected when performing the eye opening process. However, the enlarged display range of the periphery of the eye on the display screen 120 may be designated. A face may be extracted from a frame image and automatically determined based on the size of the face. By pressing the registration reference button 126a, as shown in FIG.
The representative eye candidates (open eye samples 124a to 124e) registered in advance are displayed on the display screen 120, and an open eye sample to be synthesized can be selected. It should be noted that a new eye sample may be registered in the open eye sample list registered in advance, or a registered eye sample may be deleted from the list. When combining the open eyes, the open eyes to be combined may be automatically adjusted based on information around the eyes to be combined, for example, skin color density and eye size.

Next, by depressing the other frame reference button 126b, another frame in the same item can be displayed and referenced, and the same person or a similar person's eyes with the eyes open can be selected from the images. Can be taken out and combined with the closed eyes to be corrected. In addition, by the edit button 126c,
For one or both eyes, eye size, angle, color, aspect ratio, single eyelid, double eyelid, eyelash length, orientation,
The position of the pupil, the position of the pupil, and the like can be changed, and the line of sight can be adjusted. Further, the retouching operation (dot drawing, line drawing, color painting, etc.) can be performed by the retouch button 126d.

In this way, the position data and image data of the synthesized image and the converted image (part) of the open eyes obtained by the eye correction processing unit 52 that executes the above-described eye correction processing method, and the image data before and after the eye correction The image data of the open eyes used for the difference or synthesis or conversion, the learning parameters, the analysis parameters, the conversion parameters, and the adjustment parameters are converted from the eye correction processing unit 52 of the prescan data processing unit 44 to the image of the fine scan data processing unit 46. The data is sent to the storage unit 60 and the eye image conversion unit 62 and stored. On the other hand, in the fine scan data processing section 46, the eye image conversion section 62
Represents the fine scan data that has been subjected to the enlargement / reduction by the enlargement / reduction processing unit 56 and the color density gradation conversion by the gradation conversion unit 58,
The difference between the position data and image data of the synthesized image and the converted image (part) of the open eye stored in the image storage unit 60 and the image data before and after the eye correction or the open eye used for synthesis or conversion. Using the image data, the learning parameters, the analysis parameters, the conversion parameters, the adjustment parameters, and the like, a process of turning the image of the closed eye into the previously set appropriate open eye is performed.

Here, when the position data and the image data of the synthesized image and the converted image of the opened eyes are stored in the image storage unit 60, the position data and the image data are low-resolution pre-scan data. Is based on
The eye image conversion unit 62 sets the same resolution as the fine scan data by a process such as interpolation, and then performs a synthesis process or a conversion process of the eye correction image data in accordance with the position data around the eye to be corrected of the fine scan data. To correct closed eyes to open eyes. Next, when a difference between the position data of the eye-corrected image and the image data before and after the eye correction is stored in the image storage unit 60, the position data and the difference data are based on the low-resolution pre-scan data. Therefore, the eye image conversion unit 62 sets the resolution to the same as that of the fine scan data by a process such as interpolation, and then adds the difference data in accordance with the position data around the eye to be corrected in the fine scan data. To correct closed eyes to open eyes.
Since the difference data is 0 except for the periphery of the eye to be corrected, the difference data is not limited to the periphery of the eye to be corrected, and may be taken for the entire frame image.
Can be simplified.

Further, when image data of open eyes used for synthesizing or converting, learning parameters, analysis parameters, conversion parameters, and adjustment parameters are stored in the image storage section 60, By making the configuration substantially the same as that of the eye correction processing unit 52, the same processing as the eye opening processing by the eye correction processing unit 52 can be performed to correct the closed eyes to the open eyes. of course,
When the image data of the opened eyes is low-resolution pre-scan data, it goes without saying that the eye image conversion unit 62 sets the same resolution as the fine scan data by performing processing such as interpolation. The data stored in the image storage unit 60 and the configuration and function of the eye image conversion unit 62 are not particularly limited, and may be determined according to the type or method of the eye opening processing performed by the eye correction processing unit 52 or data obtained. What is necessary is just to set suitably.

As described above, the fine scan data in which the closed eyes are corrected to the open eyes is subjected to the sharpening processing by the sharpness processing unit 64 and the conversion processing to the output device data by the color reproduction conversion unit 66. The image data is output to the printer 16 for output as a color print as output image data corresponding to the output device, and to the image data input / output means 40 for recording on a medium or distributing via communication means.

In any of the first to fourth eye correction methods described above, when one eye of an image to be corrected is open, information on the image of the open eye may be used. it can. The information of the eye image includes the features of the eye and its surroundings, that is, the color and size of the eyes that are open, the length of the eyelashes, the color of the eyelids and the skin around the eyes, the single eyelid, The type of weight and the like, the position of the eyes, the size and shape of the eyes, and the like can be considered. This information can be used to correct the closed eyes with the eyes open, for example, to make the eyes the same color, the same eyelash length, and the same eye size. By performing such processing as eyelashing, the eye correction processing for opening the meditated eyes can be performed more easily and more accurately.

As described above, in the eye correction processing, when one eye is open, it is very effective to use the features of the open eye and its surroundings. At this time, if the eye correction processing is to be performed more easily, the open eye can be inverted and combined with the eye that is closing.

As described above, according to the present embodiment, it is possible to easily convert an image with eyes closed to an image with eyes open, and to correct shooting failure. In addition, the present invention can be applied to other processing such as closing an open mouth, or turning a face to the front when a small child turns to the side during shooting. is there. As described above, the image processing method and apparatus of the present invention have been described in detail with reference to various embodiments. However, the present invention is not limited to the above examples, and various improvements may be made without departing from the gist of the present invention. Of course, changes may be made.

[0063]

As described above, according to the present invention, a process of opening eyes is performed on a human image with eyes closed to correct a photographing failure, and an image with eyes open is corrected. Can be obtained.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating an embodiment of a digital photo printer to which an image processing apparatus that performs an image processing method of the present invention is applied.

FIG. 2 is a block diagram schematically showing an embodiment of the image processing apparatus shown in FIG.

FIG. 3 is a block diagram schematically illustrating an embodiment of an eye correction processing unit of the image processing apparatus illustrated in FIG. 2;

4 is a block diagram schematically illustrating an example of an eye opening processing unit that performs a first eye correction process according to the embodiment of the eye correction processing unit illustrated in FIG. 3;

FIG. 5 is an explanatory diagram showing an example of a state of an eye opening process in a first eye correction process of the embodiment.

6 is a block diagram schematically illustrating an example of an eye opening processing unit that performs a second eye correction process according to the present embodiment and is applied to the eye correction processing unit illustrated in FIG. 3;

FIG. 7 is an explanatory diagram showing an example of an eye opening process in a second eye correction process of the embodiment.

8 is a block diagram schematically illustrating an example of an eye opening processing unit that performs a third eye correction process according to the present embodiment, which is applied to the eye correction processing unit illustrated in FIG. 3;

FIG. 9 is an explanatory diagram illustrating an example of an eye opening process in a third eye correction process according to the embodiment;

10 is a block diagram schematically illustrating an example of an eye opening processing unit that performs a fourth eye correction process according to the present embodiment, which is applied to the eye correction processing unit illustrated in FIG. 3;

FIG. 11 is an explanatory view schematically showing an embodiment of a display screen of an image display device which performs a synthesizing process for implementing the image processing method of the present invention.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 Digital photo printer 12 Scanner 14 Image processing device 16 Image recording device 18 Operation system 18a Keyboard 18b Mouse 20 Monitor 22 Light source 24 Variable aperture 26 Diffusion box 28 Carrier 30 Imaging lens unit 32 Image sensor 34 A / D converter 40 Image data Input / output means 42 LOG conversion unit 44 prescan data processing unit 46 fine scan data processing unit 48 setup unit 50, 58 gradation conversion unit 52 eye correction processing unit 54, 66 color reproduction conversion unit 56 scaling processing unit 60, 86, 98 Image storage unit 62, 90, 102, 112 Eye image conversion unit 64 Sharpness processing unit 68 Parameter calculation unit 70 Key correction unit 72, 172, 272, 372 Opening processing unit 74 Eye feature value specifying unit 76 Manual adjustment unit 78 Opening Image Part 80 th image synthesizing unit 88 modeling unit 100 conversion rule learning unit 108 eyes, eyelid data extraction unit 110 Mechanical motion analysis unit 120 display screen

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification code FI Theme coat ゛ (Reference)

Claims (17)

[Claims] 1. An image processing method for performing predetermined image processing on input image data and outputting the image data, wherein the image processing is performed on a person image in a state where eyes are closed. An image processing method comprising: an eye correction process for correcting an image of an eye in a state where the eyes are open. 2. The image processing method according to claim 1, wherein the eye correcting process is a synthesizing process for synthesizing an image of the same person with open eyes on an image of which the eyes are closed. 3. The image processing method according to claim 1, wherein the eye correcting process is a synthesizing process of synthesizing an image of an open eye of a person other than the same person with an image of a closed eye. 4. The eye correction process according to claim 1, wherein a plurality of open eye samples are prepared in advance, and a selected eye is synthesized from the plurality of open eye samples. Image processing method. 5. The synthesizing process includes a process of adjusting an eye size, an angle, eyelids, pupils, and colors and densities around an eye in an image to be synthesized so as to match an image to be synthesized.
5. The image processing method according to any one of 4. 6. The adjustment based on at least one of image characteristics such as color density around eyes of the combined image, positions of both eyes, distance between eyes, size of eyes, and size of face. 6. The image processing method according to claim 5, wherein the image processing is automatically performed. 7. The eye size, angle,
6. The image processing method according to claim 5, wherein the adjustment is manually performed by an operator based on a menu for changing a color, a density, and an aspect ratio. 8. An image in which the eyes are closed and an image in which the eyes are open are compared, and the movement of a point on the eyelid is estimated based on the characteristic of the shape of the eye. The image processing method according to claim 1, wherein the correction processing is performed. 9. An image conversion from a closed eye to an open eye is learned by using the image with the closed eyes as an input signal and the image with the open eyes as a teacher signal, and the learned conversion is performed. The image processing method according to claim 1, wherein the eye correction processing for opening the eyes that are closed based on the eye correction processing is performed. 10. The image processing method according to claim 8, wherein the degree of opening the eyes can be adjusted in the eye correction processing. 11. An eyelash-closed image can be corrected to a predetermined eye-opened image by designating eyelash length, eyelash direction, single eyelid, and double eyelid. The image processing method according to claim 8. 12. The image processing method according to claim 1, wherein said eye correction processing further has a retouch function. 13. The method according to claim 1, wherein the eye correction process uses features of an open eye and surroundings when correcting an image of a closed eye when only one eye is closed. Image processing method. 14. The peripheral features include the color and size of the pupil, the length of the eyelashes, the skin color around the eyelids and the eyes, the single eyelids,
14. The image processing method according to claim 13, wherein at least one of information of double, eye position, eye size and shape is used. 15. The image processing method according to claim 1, wherein the pupil and the position and shape of the pupil can be adjusted to adjust the line of sight of both eyes. 16. Image data is input from image input means.
An image processing apparatus for performing predetermined image processing and outputting the processed image to an image output unit, wherein an eye for correcting a person image with eyes closed is changed to an image of eyes with eyes closed with eyes closed. An image processing apparatus comprising correction processing means. 17. The image processing apparatus according to claim 16, further comprising an image display device, wherein said eye correction processing means opens the closed eye, and executes said eye correction processing. An image processing apparatus for controlling an image display device to display an enlarged image around the eyes.