JP2009223566A - Image processor, image processing method, and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem that size imbalance between a right eye and a left eye in a face in an image gives poor impression of the image. <P>SOLUTION: This image processor is provided with: a detection part for detecting information showing the sizes of the right and left eyes in the face in an object image; a correction amount determination part for determining correction amounts corresponding to the size of at least one of the right and left eyes on the basis of the detected information; and a deformation part for deforming an image with respect to at least one of a region including the left eye and a region including the right eye on the basis of the determined correction amounts. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program.

An image processing technique for deforming an image for a digital image is known (see Patent Document 1). In Patent Document 1, a partial area (an area representing a cheek image) on a face image is set as a correction area, the correction area is divided into a plurality of small areas according to a predetermined pattern, and set for each small area. Image processing for deforming the shape of a face by enlarging or reducing the image at a specified magnification is disclosed.
JP 2004-318204 A

  When the image includes a face, the eyes in the face are the part that particularly draws the attention of the observer of the image. Here, the size of the left and right eyes of a human may be different from each other. Further, in an image generated by capturing a human expression in an instant, for example, an image including a face photographed by a digital still camera or the like, such a difference in the size of the left and right eyes often appears remarkably. An image with a bad balance between the left and right eye sizes, for example, a photograph of a face with one eye slightly closed, has a poor impression on the viewer, and is desired for the display result on the monitor and the print result by the printer. It was not a thing. Note that the conventional image processing for image deformation is specialized in correcting cheek lines, and it is possible to correct the imbalance of the size of the left and right eyes in the image as described above. There wasn't.

  The present invention has been made in view of the above problems, and an image processing apparatus capable of obtaining an image that gives a good impression to an observer by correcting the balance of the size of the left and right eyes of a face in an image. An object of the present invention is to provide an image processing method and an image processing program.

  In order to achieve the above object, an image processing apparatus according to the present invention includes a detection unit that detects information indicating the size of left and right eyes of a face in a target image, and at least one of left and right based on the detected information. A correction amount determination unit that determines a correction amount for the eye size, and a deformation unit that performs image deformation processing on at least one of the region including the left eye and the region including the right eye based on the determined correction amount It is set as the structure provided with. According to the present invention, since at least one of the region including the left eye and the region including the right eye is deformed by the correction amount, a face image with a good impression in which the difference in the size of the left and right eyes is corrected is obtained. can get.

  The correction amount determining unit determines a correction amount for enlarging the size of the smaller one of the left and right eyes, and the deforming unit is an area including the smaller eye based on the correction amount for enlarging. It is also possible to perform a deformation process for enlarging. According to this configuration, since the smaller eye is enlarged, a good face image in which the left and right eyes are represented in an appropriate size can be obtained.

  The correction amount determination unit increases the size of the smaller eye as the ratio of the size of the smaller eye to the size of the larger eye among the left and right eyes is higher as the correction amount to be enlarged. The amount of correction that approaches the size of the larger eye may be determined. According to this configuration, when there is a large size difference between the left and right eyes in the untransformed image, the size of the smaller eye is not forced to approach the size of the larger eye. Therefore, it is possible to avoid the inconvenience that excessive enlargement is applied to the smaller eye, resulting in an unnatural image.

The correction amount determination unit may calculate an average value of the size of the left and right eyes, and may determine a correction amount that substantially enlarges the size of the smaller eye to the average value. According to this configuration, it is possible to perform appropriate enlargement that is not excessive for the smaller eye.
The correction amount determination unit may determine a correction amount that substantially enlarges the size of the smaller eye to the size of the larger eye. According to this configuration, the size of the left and right eyes can be made substantially uniform in the transformed image.

  The difference in the size of the left and right eyes mainly appears in the height direction of the eyes. Therefore, the deforming unit may deform the image of the region targeted for the deformation process along a substantially height direction of the eyes included in the region. According to this configuration, it is possible to obtain a good face image in which the difference in size in the height direction between the left and right eyes is corrected.

  The difference in size between the left and right eyes is often caused by the difference in the position of the upper end of the eye (the lower end of the eyelid). Therefore, the deforming unit determines, based on the correction amount, a pixel that is located above the face from a line that divides the region and is located below the eye among pixels in the region that is the target of the deformation process. The deformation process may be performed by moving the According to this configuration, the positions of the lower ends of the left and right eyes are substantially maintained before and after the deformation, and the deformation is performed so that the size of the left and right eyes with respect to the lower end is balanced. Therefore, it is possible to obtain a face image in which the difference in size between the left and right eyes is corrected very naturally.

  The image processing apparatus may include a face direction estimation unit that estimates a face direction in the target image. The correction amount determination unit may change the correction amount according to the face orientation estimated by the estimation unit. According to this configuration, the degree of deformation of each region including the left and right eyes can be changed according to the orientation of the face in the target image.

  In addition to the above-described image processing apparatus, the technical idea of the present invention is an image processing method invention including each processing step performed by each unit included in the above-described image processing apparatus, and each unit included in the above-described image processing apparatus. Can be understood as an invention of an image processing program for causing a computer to execute a function corresponding to the above. It is also possible to grasp the invention of the printing apparatus that also serves as the above-described image processing apparatus and the invention of the digital still camera that also serves as the above-described image processing apparatus.

Examples of the present invention will be described in the following order.
1. Schematic configuration of image processing apparatus:
2. Left and right eye size detection process:
3. Correction amount determination process:
4). Transform and print processing:
5. Other examples:

1. Schematic configuration of image processing apparatus:
FIG. 1 schematically shows a configuration of a printer 10 as an example of an image processing apparatus of the present invention. The printer 10 is a color inkjet printer that supports so-called direct printing, in which an image is printed based on image data acquired from a recording medium (for example, a memory card MC). The printer 10 includes a CPU 11 that controls each unit of the printer 10, an internal memory 12 configured by, for example, a ROM and a RAM, an operation unit 14 configured by buttons and a touch panel, a display unit 15 configured by a liquid crystal display, A printer engine 16, a card interface (card I / F) 17, and an I / F unit 13 for exchanging information with an external device such as a PC, a server, or a digital still camera are provided. Each component of the printer 10 is connected to each other via a bus.

  The printer engine 16 is a printing mechanism that performs printing based on print data. The card I / F 17 is an I / F for exchanging data with the memory card MC inserted into the card slot 172. Image data is stored in the memory card MC, and the printer 10 can acquire the image data stored in the memory card MC via the card I / F 17. In addition to the memory card MC, various media can be used as recording media for providing image data. The printer 10 can be connected to a PC, a server, or the like via the I / F unit 13 with a cable, and can input print data from the PC, the server, or the like.

  The internal memory 12 stores an image transformation unit 20, a display processing unit 31, and a print processing unit 32. The image deformation unit 20 is a computer program (image processing program) for executing an image deformation process and the like described below under a predetermined operating system. The display processing unit 31 is a display driver that controls the display unit 15 to display a processing menu or a message on the display unit 15. The print processing unit 32 is a computer program for generating print data from image data, controlling the printer engine 16 and printing an image based on the print data. The CPU 11 implements the functions of these units by reading and executing these programs from the internal memory 12.

  The image deformation unit 20 includes a face region detection unit 21, a face organ detection unit 22, a correction amount determination unit 23, a deformation region setting unit 24, a deformation region division unit 25, and a deformation processing execution unit 26 as program modules. And a face orientation estimating unit 27. The facial organ detection unit 22 includes a binocular size detection unit 221. The functions of these units will be described later. Furthermore, the internal memory 12 stores various data such as the dividing point arrangement pattern table 41, the face template 14b, and the eye template 14c. The printer 10 may be a so-called multifunction machine having various functions such as a copy function and a scanner function in addition to the print function.

FIG. 2 is a flowchart showing an outline of the image deformation printing process executed by the image processing apparatus (printer 10).
In step S (hereinafter, step notation is omitted) 100, the image transformation unit 20 includes information (left and right eyes) indicating the size of the left and right eyes of the face in the image (target image) to be subjected to image processing. Size).
In S300, the image deformation unit 20 determines a correction amount for the size of at least one of the left and right eyes based on the detected size of the left and right eyes.
In S500, the image deformation unit 20 performs image deformation processing on at least one of the region including the left eye and the region including the right eye in the target image based on the determined correction amount.

  In S700, the print processing unit 32 generates print data based on the image data representing the target image after the deformation process, and executes printing based on the generated print data. However, in the process of S100, the printer 10 fails to detect the face area from the target image (S110 in FIG. 3 to be described later) or detects the left and right eye areas even if the face area is successfully detected (FIG. 3). If S120) fails, the process of S700 is performed based on the image data representing the target image without detecting the size of the left and right eyes and performing S300 and S500. In the following, the details of the flowchart of FIG. 2 will be described on the assumption that the target image is an image including a human face and left and right eyes of the human face appearing.

2. Left and right eye size detection process:
FIG. 3 is a flowchart showing details of S100.
In S105, the face area detection unit 21 acquires image data 14a representing the target image from a predetermined recording medium such as a memory card MC. Of course, if the printer 10 has a hard disk drive (HDD), the face area detection unit 21 may acquire the image data 14 a stored in the HDD or via the I / F unit 13. The image data 14a may be acquired from a PC, server, digital still camera, or the like. The image data 14a is bitmap data in which the color indicated by each pixel is expressed by a combination of gradation values of RGB channels (vector in the RGB color space). In the drawings to be described later, a binary image may be shown for the sake of convenience, but in reality, multi-tone color image data is the target of processing. The image data 14a may be compressed when recorded on a recording medium or the like, or the color of each pixel may be expressed in another color space. In these cases, the development of the image data 14a and the conversion of the color space are executed, and the face area detection unit 21 acquires the image data 14a of the RGB bitmap data. When the user operates the operation unit 14 with reference to a user interface (UI) screen displayed on the display unit 15, image data 14a as a target image is designated.

  In S110, the face area detection unit 21 detects a face included in the image data 14a. The face area detection unit 21 detects a face area from the image data 14a by so-called pattern matching using a plurality of templates (face template 14b). In performing pattern matching with the face template 14b, a rectangular comparison area CA is set in the image data 14a, and the position of the comparison area CA, the size, and the rotation angle are changed, and the image and each face template in the comparison area CA are changed. The similarity with the image of 14b is evaluated. Then, the comparison area CA in which the similarity satisfies a certain criterion is determined as a face area, and the position, size, and rotation angle of the comparison area CA are acquired. In the present embodiment, the comparison area CA is rotated by 30 degrees. By moving the comparison area CA over the entire image data 14a, the position, size, and rotation angle of the area for one or more faces existing in the image data 14a can be acquired. In the present embodiment, the description will be continued assuming that a single face is detected.

FIG. 4 shows a rectangle of the comparison area CA determined as the face area in S110.
In S115, the face area detection unit 21 extracts an image included in the comparison area CA from the image data 14a as face image data FD based on the position, size, and rotation angle of the comparison area CA determined as the face area. . At this time, resolution conversion is performed so that the size of the face image data FD becomes a constant size. For example, the pixels of the face image data FD are interpolated or thinned out so as to have a size of 100 × 100 pixels. If the comparison area CA is rotated corresponding to the rotation angle of the face in the image data 14a, the face image data FD is rotated so as to eliminate this rotation. However, since the rotation angle of the comparison area CA is 30 degrees, the face rotation angle can remain in the range of ± 15 degrees in the face image data FD.

  In S120, the facial organ detection unit 22 detects the left and right eyes as facial organs. The face organ detection unit 22 detects the right eye region and the left eye region from the face image data FD by pattern matching using a plurality of templates (eye templates 14c). The face organ detection unit 22 also sets a rectangular comparison area CA in the face image data FD when performing pattern matching with the eye template 14c, and changes the position, size, and rotation angle of the comparison area CA while changing the comparison area CA. The similarity between the image inside and the image of each eye template 14c is evaluated. Then, the two comparison areas CA whose similarity satisfies a certain criterion are determined as a right eye area and a left eye area according to their positions. In this embodiment, images including various eyes such as a single eyelid, a double eyelid, and a half-opened eye are prepared as various eye templates 14c so that various eyes can be detected. . In this specification, when the figure is viewed from the front, the eye shown on the right is the left eye of the person in the target image, and when the figure is viewed from the front, the eye shown on the left is the person's left eye. Right eye. Note that the eye template 14c includes a large number of left eye and right eye so that the left and right eyes can be detected.

  FIG. 5 shows a state of the face image data FD in which the comparison area CA where the left and right eyes are present is detected. In the figure, a rectangular comparison area CA1 including the left eye in the center and a rectangular comparison area CA2 including the right eye in the center are detected. The comparison area CA1 is a left eye area, and the comparison area CA2 is a right eye area. The sizes of the comparison areas CA1 and CA2 are determined according to the size of the eyes included in each. When the detection of the right eye region and the left eye region is completed, in S125, the inclination correction unit P2c1 of the both-eye size detection unit 221 calculates the coordinates of the centroids of the comparison regions CA1 and CA2, and with respect to the straight horizontal line connecting the centroids. The inclination GR is calculated. As described above, since the rotation angle correction is performed in units of 30 degrees at the face area detection stage, the inclination GR corresponding to an angle within ± 15 degrees is basically calculated here.

FIG. 6 shows a software configuration of the facial organ detection unit 22 with the binocular size detection unit 221 as the center. The both-eye size detection unit 221 includes an inclination correction unit P2c1, a sampling unit P2c2, a scalar conversion unit P2c3, a conversion function setting unit P2c4, a parameter search unit P2c5, and an evaluation value calculation unit P2c6.
In S130, the inclination correction unit P2c1 extracts images belonging to the comparison areas CA1 and CA2 from the face image data FD as the left eye image data LE and the right eye image data RE, respectively. At this time, inclination correction according to the inclination GR calculated in S125 is performed on the left eye image data LE and the right eye image data RE.

  FIG. 7 conceptually shows the state of inclination correction executed by the inclination correction unit P2c1 in S130. In the drawing, the upper side and the lower side of the rectangular comparison area CA1 are inclined according to the inclination GR, and pixels belonging to the comparison area CA1 (shown by a broken line) after the inclination are extracted. Since the upper side and the lower side of the comparison area CA1 are inclined according to the inclination GR, the positions of the extracted pixel columns are shifted. Then, by aligning the upper end / lower end positions of the extracted pixel columns, the left eye image data LE and the right eye image data RE are obtained. Thereby, the inclination of the eyes included in the left eye image data LE and the right eye image data RE can be eliminated, and the eyes can be made almost horizontal in the left eye image data LE and the right eye image data RE. In addition, since the pixel position is shifted from the original image, the smoothness of the contour may be inaccurate or the shape of the eye may be distorted, but the corresponding amount of inclination within ± 15 degrees It is not a problem because it is just an adjustment. When the left-eye image data LE and the right-eye image data RE are obtained as described above, a process of converting the left-eye image data LE and the right-eye image data RE into image data having a scalar quantity Z is executed in S135. The vertical position in the left eye image data LE and the right eye image data RE is represented by y, the horizontal position is represented by x, and the upper left corner is x = y = 0.

  FIG. 8 shows the flow of the scalar conversion process executed in S135. The left-eye image data LE and the right-eye image data RE are image data in which each pixel has RGB gradation (RGB vector). In the scalar conversion process, the left-eye image data LE and the right-eye image data RE each have a scalar amount Z. A process of converting into image data having information is executed. First, in S210, the sampling unit P2c2 samples some pixels of the face image data FD.

  FIG. 9 shows the state of sampling in S210. In the face image data FD, a line segment L1 connecting the lower left corner and the lower right corner of the comparison areas CA1 and CA2 including the left and right eyes, and a vertical line segment L2 that bisects the line segment L1 between the comparison areas CA1 and CA2. It is shown. The line segments L1 and L2 have a predetermined length, and pixels at positions corresponding to the line segments L1 and L2 are set as sampling targets in the present embodiment. The line segments L1 and L2 correspond to a position at a substantially constant height below the eyes and a position between the left and right eyes. By sampling from such a position, pixels of colors such as outside the outline of the face, eyes, eyebrows, nostrils and mouth are not sampled, and basically only the skin-colored pixels can be sampled. Further, since sampling is performed from both the line segments L1 and L2, even if pixels other than skin color such as a frame of eyeglasses or bangs are included in a part of the line segments L1 and L2, the statistics described later are used. The influence on the indicator can be suppressed. Of course, it is only necessary to narrow the skin color to some extent. For example, sampling may be performed from another region such as a cheek or forehead.

The sampling unit P2c2 acquires the RGB gradation of the pixel to be sampled, and calculates the average μ (μ _R , μ _G , μ _B ) of the RGB gradation and the variance / covariance matrix S as statistical indexes. The average μ can be obtained by calculating an arithmetic average of gradations for each RGB channel. The variance / covariance matrix S is defined by the following equation (1).


In the above equation (1), S _R , S _G , and S _B indicate dispersion for each RGB channel, and S _RG , S _GB , and S _RB are between _RG , _GB , and _RB . Covariance is shown.

FIG. 10 is a graph showing the average μ (μ _R , μ _G , μ _B ) obtained by sampling in S210 and the variances S _R , S _G , S _B. In the present embodiment, it is assumed that the distribution of the color indicated by the sampled pixel in the three-dimensional RGB color space is a normal distribution N (μ, S), and the RGB gradation distribution has an average μ (μ _R , It is illustrated that normal distributions N _R (μ _R , S _R ), N _G (μ _G , S _G ), and N _B (μ _B , S _B ) centered on μ _G , μ _B ) are illustrated. ing. The average μ (μ _R , μ _G , μ _B ) indicates the most standard skin color in the detected face, and the variances S _R , S _G , S _B indicate the degree of gradation variation of each channel indicating the skin color. Show. Usually, the variances S _R , S _G , and S _B have different sizes, but the mean μ (μ _R , μ _G , μ _B ) is added to or subtracted from the square root (standard deviation) of each variance S _R , S _G , S _B. The probability of the range surrounded by the values is 68.2%.

In this specification, “skin color” does not mean a specific color whose absolute color value is specified, but a color distributed in the vicinity of an average μ (μ _R , μ _G , μ _B ). To do. Therefore, the absolute color meaning “skin color” varies depending on the face to be processed. When the average μ and the variance matrix S are obtained, in S220, the scalar conversion unit P2c3 acquires the left eye image data LE and the right eye image data RE in which each pixel has RGB gradation. The subsequent processing is performed for each of the left-eye image data LE and the right-eye image data RE. The processing for the left-eye image data LE will be described as an example.

In S230, the scalar conversion unit P2c3 sequentially selects one pixel from the left-eye image data LE, and substitutes the RGB gradation (r, g, b) of the selected pixel into the following equation (2) to thereby calculate the Mahalanobis square. Convert to distance D _M ² .

In the above equation (2), Δr is (r−μ _R ), Δg is (g−μ _G ), and Δb is (b−μ _B ). The Mahalanobis square distance D _M ² is the average μ (μ _R , μ _G , μ _B ) indicating the standard skin color obtained by the above-described skin color sampling, and the RGB gradation (r, g, b) of the selected pixel. The index value corresponding to the square distance in the RGB color space (the shift amount of the RGB gradation values), and the magnitude thereof does not depend on the color shift direction in the RGB color space. That is, if the Mahalanobis square distance D _M ² is the same, even if the color shift directions from the average μ (μ _R , μ _G , μ _B ) are different, there is a difference of the same degree in terms of probability. In the following, it is possible to perform processing ignoring the color misregistration direction in the RGB color space.

FIG. 11 shows isolines of Mahalanobis square distance D _M ^{2 in} the RGB color space (RG plane). In the figure, the horizontal axis indicates the R gradation (r), and the vertical axis indicates the G gradation (g). The isolines are substantially elliptical due to differences in the variances S _R , S _G , and S _B of the RGB channels. Mean μ (μ _R, μ _G) Mahalanobis square distance D _M ² in the zero, the average in the RG plane μ (μ _R, μ _G) Mahalanobis square distance D _M ² as the distance from is large. For this reason, white-eye and black-eye pixels different from the skin color included in the left-eye image data LE have a larger Mahalanobis square distance D _M ² than the skin-colored pixels. As described above, the RGB gradation (r, g, b) as a vector indicating the pixel color is converted into the Mahalanobis square distance D _M ² as a scalar indicating the deviation amount of the RGB gradation value from the standard skin color. can do. When the conversion to the Mahalanobis square distance D _M ² is completed for all the pixels of the left eye image data LE, the left eye image data LE has been converted to the image data D _M ² (x, y) of the Mahalanobis square distance D _M ^2. Become.

In S240, the scalar conversion unit P2c3 sequentially selects the pixels of the left-eye image data LE, and substitutes the Mahalanobis square distance D _M ² of the selected pixels into the conversion function represented by the following equation (3) to the scalar quantity Z. Convert.


In the above equation (3), α and u are parameters that determine the conversion characteristics of the conversion function. In the present embodiment, α = 1 is fixed, and the initial value of u is 3.

FIG. 12 shows the conversion characteristics of the conversion function. In the figure, the horizontal axis indicates the Mahalanobis square distance D _M ² before conversion, and the vertical axis indicates the scalar quantity Z after conversion. Further, the distribution of Mahalanobis square distance D _M ² before conversion of certain left-eye image data LE and the distribution of scalar quantity Z after conversion are also shown. Scalar quantity Z is -1 next time Mahalanobis square distance D _M ² is 0, the Mahalanobis square distance D _M ² represented by a nonlinear function becomes 0 when parameter u. Here, in the Mahalanobis square distance D _M ² on the horizontal axis, a region centered on the parameter u = 3 is represented as a second region A2, and a region in which the value of the Mahalanobis square distance D _M ² is larger than the second region A2 is represented. It shall represent with 1st area | region A1. In the second area A2, the slope of the conversion function is steeper than in the other areas, and the amount of variation of the scalar amount Z corresponding to the unit variation of the Mahalanobis square distance D _M ² is larger than in the other regions. ing. In the first region A1 where the Mahalanobis square distance D _M ² is larger than the second region A2, the scalar amount Z after conversion has a characteristics such as going to saturate gradually 1. The second region A2 includes the Mahalanobis square distance D _M ² in the region corresponding to the skin color, is preferably set between the non Mahalanobis squared distance flesh color corresponding to D _M ² area. The position of the second area A2 is set by the parameter u, but the initial parameter u = 3 is set based on the following grounds.

FIG. 13 is a graph showing the relationship between the Mahalanobis square distance D _M ² and the probability distribution. In the figure, the horizontal axis indicates the Mahalanobis square distance D _M ² . On the other hand, the vertical axis represents the normal distribution N (μ, S) defined by the average μ (μ _R , μ _G , μ _B ) of the RGB gradation of the skin color pixels sampled in S210 and the variance / covariance matrix S. The probability distribution (lower probability) is shown. As shown in the figure, the Mahalanobis square distance D _M ² follows an n-dimensional chi-square distribution. In the present embodiment, a color image represented by RGB gradation values is a processing target, and the probability corresponding to each Mahalanobis square distance D _M ² is estimated by a three-dimensional chi-square distribution (shown by a solid line). it can. When the Mahalanobis square distance D _M ² = 3, the lower probability is about 60%. That is, about 60% of the pixel indicating the flesh color, it can be seen that the Mahalanobis square distance D _M ² is 3 or less. If the Mahalanobis square distance D _M ² exceeds 3, it can be inferred that the pixel becomes suspicious to show skin color. That is, the Mahalanobis area nearby square distance D _M ² comes beyond 3, the Mahalanobis square distance D _M ² in the region corresponding to the skin color, Mahalanobis squared distance between the D _M ² in the region corresponding to the non-skin-color It can be estimated that this is a region. For this reason, in the present embodiment, the initial value of the parameter u is set to 3, and an area in the vicinity where the Mahalanobis square distance D _M ² is 3 is set as the second area A2. In the case of a monochrome image, the Mahalanobis square distance D _M ² follows a one-dimensional chi-square distribution (shown by a broken line).

In the distribution of the Mahalanobis square distance D _M ² before conversion by the conversion function shown in FIG. 12, the skin color distribution G1 centered on the Mahalanobis square distance D _M ² = 0 and the white (white eye) included in the left-eye image data LE. ) Distribution G2 and black (black eyes, eyebrows, etc.) distribution G3. On the other hand, in the distribution of the scalar quantity Z after conversion, the skin color distribution G1 is distributed around the scalar quantity Z = −1, and the white distribution G2 and the black distribution G3 included in the left-eye image data LE are distributions. The whole is converted to a value almost saturated to 1. In addition, the bright skin color and the dark skin color compared to the standard skin color indicated by the average μ (μ _R , μ _G , μ _B ) are located near the initial value 3 of the parameter u, and the slope is It exists in the steep second area A2.

By converting the Mahalanobis square distance D _M ² of all the pixels of the left eye image data LE into the scalar quantity Z by the conversion function described above, the left eye image data LE is converted into the image data Z (x, y) of the scalar quantity Z. Can be converted. Hereinafter, the left-eye image data LE and the right-eye image data RE converted into the image data Z (x, y) of the scalar quantity Z are also expressed as a Z map. The Z map corresponds to a scalar quantity map. The Mahalanobis square distance D _M ² can also be used as an indicator of skin color, but according to the scalar amount Z converted by the above-described conversion function, it is possible to more clearly determine whether or not it is likely to be skin color. Can do.

  In S250, the conversion function setting unit P2c4 counts pixels in which the scalar amount Z exceeds the predetermined threshold Th1, and determines whether the number ratio (area ratio) of pixels exceeding the threshold Th1 exceeds the predetermined threshold Th2. judge. In this embodiment, the threshold value Th1 = 0.6 and the threshold value Th2 = 33%. In FIG. 12, the threshold value Th1 = 0.6 is indicated by a broken line, and a pixel exceeding the threshold value Th1 = 0.6 is regarded as saturated, and is determined not to be a skin color. Also, the white distribution G2 and the black distribution G3 correspond to pixels exceeding the threshold Th1 = 0.6. That is, it is possible to determine whether each pixel of the left eye image data LE is a skin color or not a skin color by threshold determination based on the threshold Th1. It can be said that the larger the ratio of the pixels exceeding the threshold Th1, the larger the area ratio in the left-eye image data LE of the pixels that do not look like skin color.

  In this embodiment, it is assumed that it is appropriate that the area ratio occupied by pixels that do not appear to be skin-colored in the left-eye image data LE exceeds the threshold value Th2 = 33%, and the area ratio occupied by pixels that do not appear to be skin-colored is the threshold value Th2 = 33% or less If it is, the conversion function setting unit P2c4 changes the parameter u of the conversion function described above in S260. Since the parameter u is an estimated value when a normal distribution N (μ, S) is assumed, it is desirable to adjust the parameter u according to the validity based on the actual area ratio as in the present embodiment. Note that the threshold Th1 and the threshold Th2 in the present embodiment are examples, and different values may be adopted. Further, an appropriate range of the area ratio may be defined not only by the threshold value Th2 but also by an upper limit value and a lower limit value.

In S260, the parameter u is changed to four times the original value, and the process returns to S240. Then, in S250, to apply the changed parameter u in the equation (3), again, by converting the Mahalanobis square distance D _M ² of the left eye image data LE scalar quantity Z, again, to obtain a Z map. In S250, the validity of the area ratio of pixels that are not likely to be skin-colored in the Z map is similarly determined, and if not valid, the parameter u is again multiplied by four. By repeatedly executing the above processing, the conversion function can be optimized until the area ratio of pixels that do not look like a skin color is appropriate.

In the present embodiment, since the parameter u is sequentially increased, the conversion characteristic by the conversion function changes as indicated by a broken line in FIG. That is, it is possible to gradually shifting the second region A2 where the slope is steeper in the Mahalanobis square distance D _M ² increases direction. Along with this, the width of the first region A1 is narrowed. In this way, an appropriate conversion function can be set for various faces with different skin color variations. In S250, if it is determined that the ratio of pixels in which the scalar quantity Z exceeds the threshold Th1 exceeds the predetermined threshold Th2, the scalar conversion process is terminated without changing the parameter u, and the RGB level of each pixel is determined. The left-eye image data LE (Z map) in which the tone is converted to the gradation value of the scalar quantity Z is output to the parameter search unit P2c5. In the present embodiment, the parameter u is changed. However, the parameter α may be changed in order to adjust the range of the first area A1 and the second area A2. In order to widen the first area A1, the parameter α should be increased, and in order to widen the second area A2, the parameter α should be decreased. Of course, the parameter u may be set larger and gradually changed to be smaller.

  FIG. 14 shows an example of a Z map in which each pixel is represented by a scalar quantity Z (x, y). In the figure, the scalar quantity Z (x, y) in the vertical section and the horizontal section of the left-eye image data LE is shown. In the vertical section, the portion corresponding to the black eye has a scalar amount Z (x, y) close to 1, and the portion corresponding to the other skin color has a value between −1 and 0. . At the boundary between the black eye and the skin color constituting the outline of the eye, the vertical gradient is large. On the other hand, in the horizontal cross section, the portions corresponding to the black eyes and the white eyes both have a scalar amount Z close to 1, and the other portions corresponding to the skin color have values close to -1. The gradient in the horizontal direction at the boundary between the white eye and the skin color constituting the outline of the eye is a large value, but the gradient is extremely small at the boundary between the black eye and the white eye. In the above description, the processing for converting the left eye image data LE into the Z map has been described as an example. However, the same processing is executed for the right eye image data RE, and the right eye image data RE is also converted into the Z map.

In S140, the parameter search unit P2c5 acquires a Z map obtained by converting the left eye image data LE. In S145, the parameter search unit P2c5 initializes the eye contour parameters L, R, T, and B in the Z map.
FIG. 15 shows the contour parameters L, R, T, and B in the Z map of the left eye. The contour parameters L, R, T, and B correspond to the corner of the left eye, the top of the eye, the upper vertex, and the lower vertex, respectively, and L (x _L , y _L ), R ( x _R , y _R ), T (x _T , y _T ), and B (x _B , y _B ). When initial values of the contour parameters L, R, T, and B can be set, a contour line O (first approximate curve) for approximating the left eye can be generated (first approximation means). In the present embodiment, the contour line O is approximated by a curve O1 in the upper left portion, a curve O2 in the upper right portion, a curve O3 in the lower left portion, and a curve O4 in the lower left portion. Each curve O1-O4 is represented by the following formula (4).

In the above equation (4), the curve O1 is represented by a convex quadratic curve that passes through the coordinates of the contour parameter L with the coordinates of the contour parameter T as the vertex. a ₁ and a ₂ are positive, and a ₃ and a ₄ are negative. The curve O2 is expressed by a quadratic curve that is convex upward and passes the coordinates of the contour parameter R with the coordinates of the contour parameter T as a vertex. On the other hand, the curve O3 is represented by a quadratic curve that is convex downward passing through the coordinates of the contour parameter L with the coordinates of the contour parameter B as the vertices. The curve O4 is represented by a quadratic curve that is convex downward with the coordinates of the contour parameter B as the vertex and passing through the coordinates of the contour parameter R. When the coordinates of the contour parameters L, R, T, and B are determined, the curves O1 to O4 are uniquely determined. Therefore, by initializing the coordinates of the contour parameters L, R, T, and B, the position and shape of the contour line O are determined. Will be initialized. Note that the coordinates of the contour parameters L, R, T and the contour parameters L, R, B, each of which is a combination of three points, correspond to the end points where the coordinates of the contour parameters L, R are located on both outer sides in the horizontal direction. The coordinates of the contour parameters T and B correspond to a common vertex. In order to arrange the contour line O as described above, the initial values of the contour parameters L, R, T, and B are at least x _L <x _T <x _R , x _L <x _B <x _R , y _T <y _It suffices if _L <y _B and y _T <y _R <y _B are satisfied. In the present embodiment, as shown in FIG. 15, the left-eye image data LE is set to be symmetrical with respect to the central vertical line and symmetrical with respect to the horizontal line slightly below the center. In addition, initial values of the coordinates of the contour parameters L, R, T, and B are set slightly outside of the eye contour standardized by the eye template 14c (illustrated by a broken line).

  As described above, when the initial values of the coordinates of the contour parameters L, R, T, and B are set in the Z map, the parameter search unit P2c5 and the evaluation value calculation unit P2c6 perform optimum contour parameters L, R, and T in S150. , B search processing is performed. When the parameter search unit P2c5 updates the contour parameters L, R, T, and B and outputs the updated contour parameters L, R, T, and B to the evaluation value calculation unit P2c6, the evaluation value calculation unit P2c6 V is calculated, and the evaluation value V is returned to the parameter search unit P2c5. Then, the parameter search unit P2c5 searches for the coordinates of the contour parameters L, R, T, and B that maximize the evaluation value V.

FIG. 16 schematically illustrates the concept of the evaluation value V. In FIG. 16, a microline element having a length dl on the outline O and a normal unit vector p of the microline element are shown. Since the curves O1 to O4 can be specified by the above equation (4), the normal unit vector p for an arbitrary minute line element can be obtained. The normal unit vector p is set so that the inner direction is positive. That is, the y component of the normal unit vector p of the minute line element on the curves O1 and O2 is positive in the downward direction, and the y component of the normal unit vector p of the minute line element on the curves O3 and O4 is upward. Is positive. Further, the x component of the normal unit vector p of the minute line element on the curves O1 and O3 is positive in the right direction, and conversely, the x component of the normal unit vector p of the minute line element on the curves O2 and O4 is in the left direction. Is positive. The gradient vector g is expressed by the following equation (5).

In the above equation (5), the gradient vector g is given by a horizontal gradient and a vertical gradient, and becomes larger in a region where the variation of the scalar quantity Z (x, y) indicating the skin color is more severe. The evaluation value calculation unit P2c6 uses the normal unit vector p and the gradient vector g described above, and calculates an evaluation value V for evaluating the closeness of the contour line O to the contour by the following equation (6).

  In the above equation (6), the evaluation value V is obtained by performing line integration (total) along the contour O of the inner product (small evaluation value) of the normal unit vector p and the gradient vector g regarding the minute line element. However, the integral value relating to the curves O1 and O2 in the upper part of the contour O is weighted twice as much as the integral value relating to the curves O3 and O4 in the lower part of the contour O. The inner product of the normal unit vector p and the gradient vector g becomes larger as the normal unit vector p and the gradient vector g are in the same direction and the gradient vector g is larger. Accordingly, each microline element constituting the contour O is orthogonal to the gradient direction of the scalar quantity Z (x, y), and the evaluation value V increases as the gradient increases.

Since the gradient of the scalar quantity Z (x, y) can be considered to be a degree that the skin color-likeness fluctuates, when the evaluation value V is large, the contour line O passes through a region where the skin-color-likeness fluctuation is large. Can be evaluated. That is, when the evaluation value V is large, it can be considered that the contour line O passes through the contour of the eye where the variation in skin color is large. In particular, the scalar amount Z (x, y) is converted so as to fluctuate severely in the second region A2 suspected of being a skin color, so that the gradient becomes extremely large in the vicinity of the contour of the eye that is not a skin color. On the other hand, as shown in FIG. 12, the white distribution G2 and the black distribution G3 are both saturated to a value close to 1, and the scalar quantity Z (x, y) of the boundary between the white eye and the black eye is also obtained. The gradient will be small. That is, even if the white distribution G2 and the black distribution G3 show different values at the Mahalanobis square distance D _M ² , the Mahalanobis of the white distribution G2 and the black distribution G3 is converted by converting to a scalar quantity Z by a conversion function. The difference in the square distance D _M ² can be converted into a minute one, and the gradient of the scalar quantity Z can be prevented from occurring between them. Therefore, it is possible to prevent the evaluation value V from increasing at the boundary between the white eye and the black eye, and to clearly distinguish the boundary between the white eye and the black eye and the outline of the eye. In order to facilitate conceptual understanding, the evaluation value V, the gradient vector g, and the normal vector n have been described to be calculated on a continuous image plane. In this case, an equivalent operation is performed. In the search processing (search means, contour detection means) described below, the coordinates of contour parameters L, R, T, and B that increase the evaluation value V are searched.

  FIG. 17 schematically illustrates a search procedure in the search process. In the figure, the movement patterns of the contour parameters L, R, T, and B are shown, and the movement pattern is composed of the first to fourth phases. In the first phase, the contour parameters L, R are changed to four coordinates (b, c, d, e) shifted by two pixels in the diagonal four directions from the coordinates (a) of the current contour parameters L, R, T, B. , T, B are moved. In the second phase, the contour parameters L, R, T, B are shifted from the coordinates (a) of the current contour parameters L, R, T, B by two pixels in the vertical and horizontal directions (b, c, d, e). Move T and B. In the third phase, the contour parameters L, R are changed to four coordinates (b, c, d, e) shifted by one pixel in the four diagonal directions from the coordinates (a) of the current contour parameters L, R, T, B. , T, B are moved. In the fourth phase, the contour parameters L, R, Move T and B. The search process is terminated when the fourth phase is completed.

FIG. 18 shows a detailed search procedure in each phase. First, the contour parameters L, R are centered on the coordinates (a) of the current contour parameters L, R, T, B determined in the immediately preceding phase (initial values in the first phase) according to the movement pattern shown in FIG. , T, B are moved. The contour parameters L, R, T, and B are not moved simultaneously, but are moved in the order of L → R → T → B. First current contour parameter L, and outputs R, T, and B to the evaluation value calculation unit P2c6, calculates an evaluation value V _a. Next, the contour parameters R, T, and B are fixed, and the contour parameter L is sequentially moved to four coordinates (b, c, d, e) around the current coordinate (a). At this time, every time the contour parameter L moves, the contour parameters L, R, T, and B are output to the evaluation value calculation unit P2c6, and the evaluation values V _a , V _b , V _c , V _d , and V _e are calculated. The order of movement to the four coordinates (b, c, d, e) may be any. When the evaluation values V _a , V _b , V _c , V _d , and V _e for the contour parameter L of the four coordinates (a, b, c, d, e) are obtained as described above, the evaluation values V _a , Based on V _b , V _c , V _d , and V _e , the coordinate (h) of the contour parameter L that maximizes the evaluation value V is predicted.

First, a quadratic curve passing through the evaluation values V _a , V _b , and V _c regarding the line segment b-ac passing through the coordinates (a) of the current contour parameter L is calculated, and the quadratic curve is maximized. Calculate the coordinates. It should be noted that coordinates that maximize the quadratic curve can be calculated only when the quadratic curve is convex upward and the vertex is between the line segments b-ac. When the coordinate (f) that maximizes the quadratic curve cannot be calculated, the larger coordinate (b) or coordinate (c) of the end evaluation values V _b and V _c is set as the coordinate (f). Then, a straight line l1 parallel to the line segment d-a-e and passing through the coordinate (f) is generated. Next, _a quadratic curve that passes through the coordinates (a) of the current contour parameter L and passes through the evaluation values V _a , V _d , and V _e regarding the line segment d-a-e orthogonal to the line segment b-a-c. And the coordinates (g) for maximizing the quadratic curve are calculated in the same procedure. Then, a straight line l2 parallel to the line segment b-a-c and passing through the coordinate (g) is generated.

When the straight lines l1 and l2 can be generated as described above, the coordinates of these intersections are calculated as the coordinates (h) of the contour parameter L. When this coordinate (h) can be calculated, the contour parameter L is moved to the coordinate (h), and the evaluation value V _h at that time is calculated. When the six evaluation values V _a , V _b , V _c , V _d , V _e , and V _h can be calculated as described above, the coordinate corresponding to the largest one of these is obtained as the optimum contour in the phase. Determined as the coordinates of the parameter L. When the optimum coordinates for the contour parameter L are determined, the same processing is performed for the contour parameter R to determine the optimum coordinates. Further, the same processing is sequentially performed on the contour parameters T and B to determine the optimum coordinates, the phase is terminated, and the process proceeds to the next phase. When the fourth phase is completed, the contour parameters L, R, T, and B are finally determined.

  As described above, in the first and second phases of the initial search, the contour parameters L, R, T, and B are moved over a wide range. , Close to the top and bottom vertices. Further, in the third and fourth phases in the latter half of the search, the contour parameters L, R, T, and B are locally moved within a narrow range, so that the contour parameters are converged so as to converge to the corner of the eye, the top of the eye, the upper vertex, and the lower vertex The coordinates of L, R, T, and B can be finely adjusted. The search may be terminated when the constant evaluation value V is reached, and the contour parameters L, R, T, and B at that time may be output.

The contour parameters L, R, T, and B for the left eye finally searched in this way are stored in a predetermined area of the internal memory 12, and the contour detection of the right eye is subsequently executed.
In S155 to S165, the parameter search unit P2c5 and the evaluation value calculation unit P2c6, based on the Z map obtained by converting the right eye image data RE, similarly to the processing described in S140 to S150, the right eye contour parameters L, R, The search for T and B is performed, and the finally searched right eye contour parameters L, R, T, and B are stored in a predetermined area of the internal memory 12.

In S <b> 170, the both-eye size detection unit 221 detects the left-eye size using the left-eye contour parameters L, R, T, and B stored in the internal memory 12, and the right-eye contour parameter L, stored in the internal memory 12. The size of the right eye is detected using R, T, and B. In the case of the eye size, various information such as eye height (eye length in the face height direction), eye width (eye length in the lateral direction of the face), eye area, and the like can be considered. In this embodiment, the eye height is detected as the eye size. This is because the left and right eyes of a human face are basically not significantly different in width, but the height is likely to be different between left and right. The both-eye size detection unit 221 has a maximum value and a minimum value among the y-coordinate values y _L , y _R , y _T , y _B in the Z map of the left eye contour parameters L, R, T, B stored in the internal memory 12. The difference from the value (in many cases, the difference between y _T and y _B ) is calculated, and the calculated difference is acquired as the size LS of the left eye. Similarly, the both-eye size detection unit 221 calculates the difference between the maximum value and the minimum value among the y-coordinate values of the right-eye contour parameters L, R, T, and B stored in the internal memory 12 and calculates the difference. The difference is acquired as the right eye size RS.

  However, the both-eye size detection unit 221 may acquire information representing the size of the left and right eyes more easily without using the contour parameter. As described above, the size of the region of the left eye region (comparison region CA1) and right eye region (comparison region CA2) detected by the facial organ detection unit 22 in S120 is determined according to the size of the eye contained therein. Has been. Therefore, the binocular size detection unit 221 acquires, for example, the vertical length (number of pixels) of the comparison area CA1 in the face image data FD as the left eye size LS, and the vertical length of the comparison area CA2 in the face image data FD. May be acquired as the size RS of the right eye. When detecting the lengths of the sides of the comparison areas CA1 and CA2 as the eye size, the processing of S125 to S165 (FIG. 3) is not necessary. As described above, in the present embodiment, the binocular size detection unit 221 detects information that directly or indirectly represents the size of the left and right eyes as the left eye size LS and the right eye size RS. Hereinafter, the size LS of the left eye and the size RS of the right eye are simply expressed as LS, RS.

3. Correction amount determination process:
If LS and RS are detected, then the correction amount determination unit 23 determines the correction amount based on LS and RS.
FIG. 19 is a flowchart illustrating details of S300 (FIG. 2) executed by the correction amount determination unit 23. In S305, the correction amount determination unit 23 determines the target eye for determining the correction amount by performing a size determination between LS and RS. In the present embodiment, it is determined which of the left and right eyes is smaller, and the correction amount β (enlargement ratio) for the smaller eye is determined. Then, as will be described later, image deformation (S500) is performed to enlarge the size of the smaller eye based on the correction amount β.

If it is determined that RS = LS in the size determination, the size of the left and right eyes is balanced. Therefore, the printer 10 performs the process of S700 based on the image data 14a representing the target image without determining the correction amount β described later and the process of S500. The following description will be made assuming that it is determined that RS <LS, that is, a case where it is determined that the right eye is smaller than the left eye.
In S310, the correction amount determination unit 23 generates a function for determining the correction amount for the eye (right eye) that is the correction amount determination target in the above-described size determination. First, the correction amount determination unit 23 determines the minimum value and the maximum value of the correction amount β. Here, the minimum value of the correction amount β is 1, and the maximum value of the correction amount β is LS / RS. Next, the correction amount determination unit 23 generates a function that outputs minimum value = 1 and maximum value = LS / RS.

FIG. 20 shows an example of the function F1 generated by the correction amount determination unit 23. In the figure, an upwardly convex quadratic curve is shown where the input value (horizontal axis) = RS / LS and the output value (vertical axis) = correction amount β. The correction amount determination unit 23 generates such a quadratic curve as the function F1. The function F1 is a function that outputs the minimum value = 1 when the input value = 0, and outputs the maximum value = LS / RS when the input value = 1. Note that the curve shape itself of the function F1 is predetermined, and the scale interval on the vertical axis of the function F1 is changed according to the maximum value of the correction amount β.
In S315, the correction amount determination unit 23 determines the correction amount β by inputting RS / LS to the generated function F1. The correction amount β thus determined is a correction amount for enlarging the size of the right eye in the present embodiment.

  By using the function F1, the smaller eye size is made closer to the larger eye size as the ratio of the smaller eye size to the larger eye size among the left and right eyes increases. The correction amount β to be determined can be determined. That is, when the ratio is low to some extent, the determined correction amount β is also a value that is somewhat away from the maximum value (larger eye size / smaller eye size), so excessive deformation of the small eye It is prevented that the image is broken by (enlargement). On the other hand, when the ratio is high to some extent, the determined correction amount β is also a value close to the maximum value, and the sizes of both eyes are substantially the same after deformation. If the difference in size between the two eyes is originally small, the naturalness of the image is not impaired even if the smaller eye is enlarged to the same size as the larger eye.

However, the method of determining the correction amount β for enlarging the size of the smaller eye is not limited to the method described above. For example, the correction amount determination unit 23 may determine the correction amount β based on the average value Ave = (LS + RS) / 2 of LS and RS. Specifically, the correction amount determination unit 23 sets the correction amount β = Ave / RS so that the size of the smaller eye (right eye) is substantially enlarged to the average value Ave by image deformation described later. If the correction amount β = Ave / RS is determined, the difference between the left and right eye sizes remains even after deformation, but excessive deformation (enlargement) of the smaller eye is prevented, and the observer does not feel uncomfortable. The balance of the size of both eyes can be adjusted to the extent. Or, more simply, the correction amount determination unit 23 corrects the correction amount β = so that the size of the smaller eye (right eye) is enlarged to the size of the larger eye (left eye) by image deformation described later. It may be LS / RS.
Needless to say, if LS <RS, the relationship between RS and LS in determining the correction amount β described above is reversed.

4). Transform and print processing:
After the correction amount β is determined, the deformation area setting unit 24, the deformation area dividing unit 25, and the deformation process executing unit 26 perform an image deformation process based on the correction amount β. The deformation area setting unit 24, the deformation area dividing unit 25, and the deformation process execution unit 26 may be collectively referred to as a deformation unit.
FIG. 21 is a flowchart showing details of S500 (FIG. 2) executed by the deformation unit. In S505, the deformation area setting unit 24 sets the deformation area TA. The deformation area TA is an area that includes at least a part of the face on the image data 14a and can be an object of image deformation. The deformation area TA can be set based on, for example, the comparison area C (comparison area CA shown in FIG. 4) detected as a face area in S110 (FIG. 3). Hereinafter, the comparison area C detected as a face area in S110 is referred to as a face area FA.

  FIG. 22 shows an example of the face area FA and the deformation area TA set based on the face area FA. In the figure, the reference line SL of the face area FA is shown. The reference line SL is a line that passes through the center of gravity of the face area FA and is a line that faces the height direction of the face of the face area FA. Note that the face area FA shown in FIG. 22 is slightly different in position and angle on the image data 14a compared to the face area (comparison area C) shown in FIG. This is because, before the deformation area TA is set, the deformation area setting unit 24 displays the rectangle of the face area detected in S110, and the upper and lower sides of the face area in the width direction of the face (for example, a line connecting the centers of both eyes). This is because the main organs in the face are moved along the reference line SL so as to surely enter the rectangle. Such adjustment of the angle and position of the face area may or may not be performed.

As shown in FIG. 22, the deformation area TA is set as an area obtained by extending (or shortening) the face area FA in a direction parallel to the reference line SL and in a direction perpendicular to the reference line SL. Specifically, if the size in the height direction of the face area FA is Hf and the size in the width direction is Wfi, the face area FA is extended by k1 · Hf upward and by k2 · Hf downward, A region extended by k3 · Wfi to the left and right is set as the deformation region TA. k1, k2, and k3 are predetermined coefficients. As shown in FIG. 22, the deformation area TA is set as an area that generally includes an image from the chin to the forehead with respect to the height direction of the face and includes images of the left and right cheeks with respect to the width direction of the face. The In the present embodiment, the above-described coefficients k1, k2, and k3 are set in advance based on the relationship with the size of the face area FA so that the deformation area TA is an area that includes an image in the above-described range. .
In S510, the deformation area dividing unit 25 divides the deformation area TA into a plurality of small areas.

  FIG. 23 is an explanatory diagram showing an example of a method of dividing the deformation area TA into small areas. The deformation area dividing unit 25 arranges a plurality of division points D in the deformation area TA, and divides the deformation area TA into a plurality of small areas using a straight line connecting the division points D. The manner of arrangement of the division points D (the number and position of the division points D) is defined by the division point arrangement pattern table 41 (FIG. 1). The deformation area dividing unit 25 arranges the dividing points D with reference to the dividing point arrangement pattern table 41. As shown in FIG. 23, as an example, the dividing points D are arranged at the intersections of the horizontal dividing line Lh and the vertical dividing line Lv and at the intersections of the horizontal dividing line Lh and the vertical dividing line Lv and the outer frame of the deformation area TA. Is done. The horizontal dividing line Lh and the vertical dividing line Lv are reference lines for arranging the dividing point D in the deformation area TA. In the arrangement of the dividing points D shown in FIG. 23, two horizontal dividing lines Lh perpendicular to the reference line SL and four vertical dividing lines Lv parallel to the reference line SL are set. The two horizontal dividing lines Lh are called Lh1 and Lh2 in order from the bottom of the deformation area TA. The four vertical dividing lines Lv are referred to as Lv1, Lv2, Lv3, and Lv4 in order from the left of the deformation area TA.

  The horizontal dividing line Lh1 is arranged immediately below the eye image in the deformation area TA, and the horizontal dividing line Lh2 is arranged in the vicinity immediately above the eye image. The vertical dividing lines Lv1 and Lv4 are arranged outside the left and right eye corner images, and the vertical dividing lines Lv2 and Lv3 are arranged inside the left and right eye images. The horizontal dividing line Lh and the vertical dividing line Lv are arranged in the deformation area TA set in advance so that the positional relationship between the horizontal dividing line Lh and the vertical dividing line Lv and the image becomes the above-described positional relationship as a result. It is executed according to the correspondence with the size.

  According to the arrangement of the horizontal dividing line Lh and the vertical dividing line Lv described above, the intersection of the horizontal dividing line Lh and the vertical dividing line Lv, and the intersection of the horizontal dividing line Lh and the vertical dividing line Lv and the outer frame of the deformation area TA In addition, the dividing point D is arranged. As shown in FIG. 23, division points D located on the horizontal division line Lhi (i = 1 or 2) are referred to as D0i, D1i, D2i, D3i, D4i, and D5i in order from the left. For example, the dividing points D located on the horizontal dividing line Lh1 are called D01, D11, D21, D31, D41, D51. Similarly, the dividing points D located on the vertical dividing line Lvj (j = 1, 2, 3, 4) are called Dj0, Dj1, Dj2, Dj3 in order from the bottom. For example, the division points D located on the vertical division line Lv1 are called D10, D11, D12, and D13. As shown in FIG. 23, the arrangement of the dividing points D is symmetrical with respect to the reference line SL.

The deformation area dividing unit 25 divides the deformation area TA into a plurality of small areas by a straight line connecting the arranged division points D (that is, the horizontal division line Lh and the vertical division line Lv). Since the arrangement of the dividing points D is determined by the numbers and positions of the horizontal dividing lines Lh and the vertical dividing lines Lv, the dividing point arrangement pattern table 41 defines the numbers and positions of the horizontal dividing lines Lh and the vertical dividing lines Lv. It is also possible to paraphrase.
In S515, the deformation process execution unit 26 performs an image deformation process on the deformation area TA. The deformation process by the deformation process execution unit 26 is performed by moving the position of the division point D arranged in the deformation area TA in S510 to deform the small area.

  FIG. 24 is an explanatory diagram showing an example of the position movement of the dividing point D. The deformation processing execution unit 26 determines the dividing point D surrounding the eye (right eye in the present embodiment) for which the correction amount β is determined by the correction amount determination unit 23 in the V direction (reference) based on the determined correction amount β. In a direction parallel to the line SL). Note that “V” used in the description of FIG. 24 means the direction of the deformation area TA in the image data 14a, and is different from the evaluation value V. The division points D surrounding the right eye are D11, D12, D22, and D21. The small area (small area shown with hatching) having the vertexes at the division points D11, D12, D22, and D21 corresponds to an example of an area including the right eye in the claims. In addition, the small area | region which uses the division | segmentation points D31, D32, D42, and D41 as a vertex corresponds to an example of the area | region containing the left eye said to the claim.

  If the distance (number of pixels) in the V direction of the small area having the vertexes at the dividing points D11, D12, D22, and D21 is, for example, V1, the deformation processing execution unit 26 determines that the distance in the V direction of the small area is β · All or part of the dividing points D11, D12, D22, and D21 are moved so as to be enlarged to V1. When all of the dividing points D11, D12, D22, and D21 are moved, the dividing points D11 and D21 on the horizontal dividing line Lh1 positioned below the eyes are set to the minus side (below the face) in the V direction. The dividing points D12 and D22 on the horizontal dividing line Lh2 located above the eyes are moved to the plus side (upper face) in the V direction. However, in the present embodiment, as illustrated in FIG. 24, the transformation processing execution unit 26 does not move the dividing points D11 and D21 located on the horizontal dividing line Lh1, and the dividing point D12 located on the horizontal dividing line Lh2. , D22 is moved to the plus side in the V direction. In the present embodiment, the division point D is not moved in the H direction (direction orthogonal to the reference line SL).

The deformation processing execution unit 26 does not move the position of the dividing point D located on the outer frame of the deformation area TA so that the boundary between the inner and outer images of the deformation area TA does not become unnatural. In FIG. 24, the division point D before the movement is indicated by a white circle, and the division point D after the movement or the division point D without the movement of the position is indicated by a black circle. Each divided point D after the movement is represented with “′”.
For the small area constituting the deformation area TA, the deformation processing execution unit 26 is configured such that the image of the small area before the position movement of the dividing point D is the image of the small area newly defined by the position movement of the dividing point D. In this way, an image deformation process is performed. Specifically, the image of the small area having the vertices at the division points D11, D12, D22, and D21 is transformed (enlarged) into the image of the small area having the vertices at the division points D11, D′ 12, D′ 22, and D21. Is done. Along with this, the image of the small area having the vertexes at the dividing points D01, D02, D12, and D11 is transformed (enlarged) into the image of the small area having the vertexes at the dividing points D01, D02, D′ 12, and D11. Then, the image of the small area having the vertexes at the dividing points D21, D22, D32, and D31 is transformed (enlarged) into the image of the small area having the vertexes at the dividing points D21, D'22, D32, and D31.

  FIG. 25 is an explanatory diagram showing the concept of the image deformation processing method by the deformation processing execution unit 26. In FIG. 25, the dividing point D is indicated by a black circle. In FIG. 25, for simplification of description, regarding the four small regions, the state before the position movement of the dividing point D is shown on the left side, and the state after the position moving of the dividing point D is shown on the right side. In the example of FIG. 25, the center dividing point Da is moved to the position of the dividing point Da ', and the positions of the other dividing points D are not moved. Thereby, for example, an image of a rectangular small area (hereinafter also referred to as “pre-deformation noticeable small area BSA”) having the vertices at the division points Da, Db, Dc, Dd before the movement of the division point D is obtained from the division point Da ′. , Db, Dc, and Dd are transformed into an image of a rectangular small area (hereinafter also referred to as “the noticed small area ASA after deformation”).

  In the present embodiment, a rectangular small region is divided into four triangular regions using the center of gravity CG of the small region, and image deformation processing is performed in units of triangular regions. In the example of FIG. 25, the pre-deformation attention small area BSA is divided into four triangular areas having the centroid CG of the pre-deformation attention small area BSA as one vertex. Similarly, the post-deformation attention small area ASA is divided into four triangular areas having the centroid CG ′ of the post-deformation attention small area ASA as one vertex. Then, image deformation processing is performed for each corresponding triangular area in each state before and after the movement of the dividing point Da. For example, an image of a triangular area having vertices at the division points Da and Dd and the center of gravity CG in the attention small area BSA before deformation has a vertex at the division points Da ′ and Dd and the center of gravity CG ′ in the attention small area ASA after deformation. It is transformed into an image of a triangular area.

  FIG. 26 is an explanatory diagram showing the concept of an image deformation processing method in a triangular area. In the example of FIG. 26, the image of the triangular area stu with the points s, t, u as vertices is transformed into the image of the triangular area s′t′u ′ with the points s ′, t ′, u ′ as vertices. . Note that “s” and “u” used in the description of FIG. 26 are used in a different meaning from the same reference numerals already described in this specification. For the deformation of the image, the position of a certain pixel in the image of the triangular area s't'u 'after the deformation corresponds to the position in the image of the triangular area stu before the deformation, and the calculated position This is performed by using the pixel value in the image before deformation in step S4 as the pixel value of the image after deformation. For example, in FIG. 26, the position of the pixel of interest p ′ in the image of the triangular area s′t′u ′ after deformation corresponds to the position p in the image of the triangular area stu before deformation. The position p is calculated as follows. First, coefficients m1 and m2 for expressing the position of the target pixel p ′ by the sum of the vector s′t ′ and the vector s′u ′ as shown in the following equation (7) are calculated.

Next, by using the calculated coefficients m1 and m2, the position p is obtained by calculating the sum of the vector st and the vector su in the triangular area stu before deformation according to the following equation (8).


When the position p in the triangular area stu before deformation coincides with the pixel center position of the image before deformation, the pixel value of the pixel is set as the pixel value of the image after deformation. On the other hand, when the position p in the triangular area stu before deformation is shifted from the pixel center position of the image before deformation, an interpolation operation such as bicubic using the pixel values of the pixels around the position p. Thus, the pixel value at the position p is calculated, and the calculated pixel value is set as the pixel value of the image after deformation.

  Image deformation from the image of the triangular area stu to the image of the triangular area s't'u 'by calculating the pixel value for each pixel in the image of the triangular area s't'u' after the deformation as described above Processing can be performed. The deformation process execution unit 26 performs a deformation process by defining a triangular area as described above for the small area constituting the deformation area TA illustrated in FIG. 24, and performs an image deformation process in the deformation area TA.

  As described above, in the present embodiment, the deformation processing execution unit 26 does not move the dividing points D11 and D21 positioned on the horizontal dividing line Lh1, but corrects the dividing points D12 and D22 positioned on the horizontal dividing line Lh2. Move to the plus side in the V direction according to β. Therefore, an image surrounded by the horizontal dividing line Lh1, the horizontal dividing line Lh2, and the vertical dividing line Lv3, particularly, an image surrounded by the horizontal dividing line Lh1, the horizontal dividing line Lh2, the vertical dividing line Lv1, and the vertical dividing line Lv2, Enlarged above the face in the V direction. As a result, the region including the smaller eye (right eye) of the left and right eyes is enlarged, and after the enlargement, the difference in the size of the left and right eyes is reduced (or the difference in the size of the left and right eyes is reduced). No longer).

  However, the region to be deformed by the deformation processing execution unit 26 does not necessarily need to be a region surrounded by the dividing points D set in the deformation region TA. For example, the deformation processing execution unit 26 uses the correction amount determination unit 23 to correct the correction amount among the left eye region (comparison region CA1) and right eye region (comparison region CA2) detected by the facial organ detection unit 22 in S120 (FIG. 3). A region on the side corresponding to the eye for which β is determined may be a deformation target. In this case, the deformation processing execution unit 26 performs deformation for enlarging (multiplying by β) the image in the rectangle of the region to be deformed (for example, the right eye region) in the eye height direction based on the correction amount β. do it.

  In S700 (FIG. 2), the print processing unit 32 controls the printer engine 16 to print the target image. That is, the print processing unit 32 adds the resolution conversion process, the color conversion process, and the half conversion to the image data 14a after the deformation process (the image data 14a acquired in S105 (FIG. 3 when the deformation process is not executed)). Print data is generated by performing necessary processes such as tone processing. The generated print data is supplied from the print processing unit 32 to the printer engine 16, and the printer engine 16 executes printing based on the print data. Thereby, the printing of the target image after the deformation process is completed.

  Thus, according to the present embodiment, the image deformation unit 20 detects the left and right eye sizes LS and RS of the face included in the target image, and based on the detected LS and RS, the left and right eyes A correction amount β for enlarging the smaller eye is determined. Then, based on the determined correction amount β, a deformation process for enlarging the region including the smaller eye in the target image along the V direction, that is, the approximate height direction of the face is performed. As a result, even if there is a difference in the size of the left and right eyes of the face in the target image, an image that gives a good impression to the observer, with the difference in the size of the left and right eyes automatically corrected Can be obtained.

  The left and right eyes of humans have basically little difference in the width (the length of the eyes in the lateral direction of the face), while the height (the length of the eyes in the height direction of the face) is Differences are likely to occur due to differences in eyelid shape and eye movement. Therefore, in the present embodiment, as described above, the region including the smaller eye is enlarged along the approximate height direction of the face (the approximate height direction of the eyes), so that the left and right that are actually generated in the target image. Correct the difference in eye size. Furthermore, when examining the difference in the size of the left and right eyes, many of the differences are due to the difference in the position of the top of the left and right eyes (the bottom of the eyelid), while the position of the bottom of the left and right eyes. It can be said that there is no big difference. Therefore, in the present embodiment, as described above, the dividing points D11 and D21 on the horizontal dividing line Lh1 located below the eyes are not moved, and the dividing points D12 and D22 on the horizontal dividing line Lh2 located above the eyes are moved. As a result, the pixels located on the upper side of the face with respect to the horizontal dividing line Lh1 are moved toward the upper side of the face by moving to the plus side in the V direction. Therefore, the corrected face image is obtained without any sense of incongruity because the target image is corrected so that the eye represented smaller than the other eye is widened and the lower end positions of the left and right eyes are substantially maintained. It is done.

  In the above description, the case where the smaller eye is enlarged based on the correction amount β for enlarging the smaller eye among the left and right eyes has been described. However, in the present invention, a correction amount (reduction rate) for reducing the larger eye is determined based on LS and RS, and the larger eye is reduced based on the determined correction amount. Is possible. Alternatively, a correction amount for reducing the larger eye and a correction amount for enlarging the smaller eye are determined based on LS and RS, and the correction amount for reducing the larger eye is large. It is also possible to perform a modification that enlarges the smaller eye based on the correction amount for reducing the smaller eye and enlarging the smaller eye.

5. Other Embodiments The image deformation unit 20 includes a face direction estimation unit 27 (FIG. 1). The face direction estimation unit 27 is a function that estimates the face direction (front direction, upward direction, downward direction, right direction, left direction, etc.) in the target image. The face of the target image is upward, meaning that the face tends to face up, the downward direction means that the face is prone, and the right direction is the left profile for the person in the target image. Means the state in which the right profile for the person of the target image is mainly visible in the image. In the process of executing S100 and S300 (FIG. 2), the image deforming unit 20 causes the face direction estimating unit 27 to estimate the face direction in the target image, and the estimated degree of the face direction (degree of face swing). ), The correction amount β may be changed.

  FIG. 27 exemplifies the comparison area CA detected as a face area from the image data 14a in S110 (FIG. 3). FIG. 27 also illustrates the positions of the comparison area CA1 as the left eye area and the comparison area CA2 as the right eye area detected by the facial organ detection unit 22 in S120. In FIG. 27, illustration of the face itself in the image data 14a is omitted. Further, FIG. 27 also illustrates the position of a rectangular area (mouth area CA3) including the mouth image detected by the facial organ detection unit 22. In S120, the image transformation unit 20 causes the facial organ detection unit 22 to detect the mouth in addition to the left and right eyes. That is, the face organ detection unit 22 further sets a rectangular comparison area CA in the comparison area CA (face image data FD) as a face area, and changes the position, size, and rotation angle of the set comparison area CA. However, the mouth area CA3 can be detected by performing pattern matching for evaluating the similarity between the image in the set comparison area CA and the image of each template (mouth template).

  When the comparison area CA1, the comparison area CA2, and the mouth area CA3 are detected, the face orientation estimation unit 27 executes a face orientation estimation process. Although the timing at which the estimation process is executed varies, as an example, the face direction estimation unit 27 performs the estimation process at a timing between S120 and S125. Note that if the face organ detection unit 22 cannot detect any one of the left eye region, the right eye region, and the mouth region CA3, the face direction estimation unit 27 may not perform the estimation process. Here, the length of the line segment CL connecting the center point Ce (l) of the comparison area CA1 and the center point Ce (r) of the comparison area CA2 is referred to as a reference width Wr, and the center point Cm and the line segment of the mouth area CA3. The distance from CL is called the reference height Hr. As is clear from FIG. 27, the reference width Wr is an index correlated with the width of the face image, and the reference height Hr is an index correlated with the height of the face image.

  The face direction estimation unit 27 calculates the reference height Hr and the reference width Wr, calculates the ratio (Hr / Wr) between the reference height Hr and the reference width Wr as the determination index value DI, and determines the determination index value DI. Based on the above, the face orientation is estimated. Specifically, when the determination index value DI is equal to or greater than the threshold value DT1 and less than the threshold value DT2, the face direction estimation unit 27 estimates that the face image direction is the front direction. The threshold values DT1 and DT2 are both positive values and have a relationship of DT1 <DT2. When the determination index value DI is less than the threshold value DT1, the face orientation estimation unit 27 estimates that the orientation of the face image is upward or downward. When the determination index value DI is equal to or greater than the threshold value DT2, the face orientation estimation unit 27 estimates that the orientation of the face image is rightward or leftward.

  When the orientation of the face image is rightward or leftward, the reference height Hr is hardly changed, but the reference width Wr is considered to be small compared to the case of facing the front. Therefore, the determination index value DI (= Hr / Wr) is greater when the orientation of the face image is rightward or leftward than when it is frontward. On the other hand, when the orientation of the face image is upward or downward, the reference width Wr is hardly changed, but the reference height Hr is considered to be small compared to the case of facing the front. Therefore, the determination index value DI (= Hr / Wr) is smaller when the orientation of the face image is upward or downward than when the face image is frontward. Threshold values DT1 and DT2 are statistically determined from determination index values DI of a plurality of face sample images. Predetermined threshold values DT 1 and DT 2 are stored in a predetermined area in the internal memory 12.

  In the present embodiment, when the face is estimated to be rightward or leftward, the estimation result is taken into account when determining the correction amount β. Therefore, the face orientation estimating unit 27 records the calculated determination index value DI in the internal memory 12 when it is estimated that the orientation of the face image is rightward or leftward. Therefore, the determination index value DI recorded in the internal memory 12 is at least a value equal to or greater than the threshold value DT2. It can be said that the determination index value DI recorded in the internal memory 12 indicates the rightward or leftward degree (lateral degree) of the face in the target image. The correction amount determination unit 23 determines whether or not the determination index value DI is recorded in the internal memory 12 in the execution process of S300. When the determination index value DI is recorded in the internal memory 12, the correction amount β determined in S315 (FIG. 19) is changed according to the magnitude of the determination index value DI. Hereinafter, the corrected correction amount is referred to as a correction amount β ′.

  FIG. 28 illustrates a function F2 for changing the correction amount β according to the determination index value DI. The function F2 is a downwardly convex quadratic curve with input value (horizontal axis) = determination index value DI and output value (vertical axis) = correction amount β ′. The function F2 outputs the maximum value (correction amount β) when the input value = DT2, and outputs a value closer to 1 as the input value increases. Accordingly, the larger the determination index value DI is (the higher the degree of lateral orientation of the face in the target image), the smaller the correction amount β is set to a smaller value with 1 being the lower limit. The correction amount determination unit 23 transfers the correction amount β ′ thus output based on the function F2 to the deformation unit. The deformation unit performs the above-described image deformation based on the correction amount β ′ (S500).

  If the face in the target image is facing right or left, the back eye (the right eye of the face if the face of the target image is facing right, the left eye of the face if the face of the target image is facing left) is It is natural that it is displayed smaller than the eyes. In this way, if the small and natural back-side eye is enlarged based on the correction amount β determined as described above, it becomes excessively deformed, and on the contrary, the image may become unnatural. Therefore, in the present embodiment, when the face direction estimation unit 27 estimates that the degree of the horizontal direction of the face is high, the correction amount determination unit 23 sets the correction amount β for enlarging the smaller eye as described above. By reducing (changing to the correction amount β ′), excessive deformation of the image is prevented. When the correction amount β ′ after the correction by the correction amount determination unit 23 is 1, the image is not substantially deformed. In this case, the printer 10 skips S500 and proceeds to S700.

As an example of processing based on the estimation result by the face direction estimation unit 27, the image deformation unit 20 may not perform image deformation if the determination index value DI calculated by the face direction estimation unit 27 is equal to or greater than a certain value. Good. That is, the determination index value DI calculated by the face direction estimation unit 27 in the next step of S120 is equal to or greater than the threshold value DT2 or equal to or greater than a predetermined value (threshold value DT3) greater than the threshold value DT2. If it is, the image deformation unit 20 assumes that the correction amount β as the enlargement ratio (or reduction ratio) is changed to 1, and does not perform the processes of S125 to S170, S300, and S500, and proceeds to S700. You may go forward.
In the present embodiment, the processing by the printer 10 has been described. However, the above-described processing by the image deformation unit 20 (processing of S100 to S500) can be performed by various hardware other than the printer, such as a PC or a digital still camera. It is.

1 is a block diagram illustrating a schematic configuration of a printer as an image processing apparatus. 4 is a flowchart illustrating processing executed by a printer. It is a flowchart which shows the detail of a both-eye size detection process. It is a figure which shows the mode of face detection. It is a figure which shows the face image data after a face organ detection. It is a block diagram which shows the software structure of a both-eye size detection part. It is a figure which shows the mode of inclination correction. It is a flowchart which shows the flow of a scalar conversion process. It is a figure which shows the mode of sampling. It is a histogram obtained by sampling. It is a graph which shows the isoline of Mahalanobis square distance. It is a graph which shows the conversion characteristic by a conversion function. It is a graph which shows the relationship between Mahalanobis square distance and probability distribution. It is a figure which shows an example of Z map. It is a figure which shows the outline parameter in Z map. It is a figure explaining an evaluation value. It is a schematic diagram which shows the search procedure in a search process. It is a schematic diagram which shows the detailed procedure of a search process. It is a flowchart which shows the detail of correction amount determination processing. It is a figure which shows an example of the function for correction amount determination. It is a flowchart which shows the detail of an image deformation process. It is a figure which shows an example of the deformation | transformation area | region set on image data. It is a figure which shows an example of a mode that the deformation | transformation area | region was divided | segmented into the small area | region. It is a figure which shows an example of the position movement of a dividing point. It is explanatory drawing which shows the concept of the deformation | transformation process of an image. It is explanatory drawing which shows the concept of the deformation | transformation process of the image in a triangular area | region. It is a figure which shows a left eye area | region, a right eye area | region, a mouth area | region, etc. FIG. It is a figure which shows the function for changing correction amount according to a determination parameter | index value.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Printer, 11 ... CPU, 12 ... Internal memory, 14a ... Image data, 14b ... Face template, 14c ... Eye template, 16 ... Printer engine, 17 ... Card I / F, 20 ... Image deformation part, 21 ... Face area Detection unit, 22 ... Face organ detection unit, 23 ... Correction amount determination unit, 24 ... Deformation region setting unit, 25 ... Deformation region dividing unit, 26 ... Deformation processing execution unit, 27 ... Face orientation estimation unit, 32 ... Print processing unit , 41... Division point arrangement pattern table, 172... Card slot, 221..., Eye size detection unit, P2c1... Tilt correction unit, P2c2 ... sampling unit, P2c3 ... scalar conversion unit, P2c4 ... conversion function setting unit, P2c5 , P2c6 ... evaluation value calculation unit

Claims (10)

A detection unit that detects information indicating the size of the left and right eyes of the face in the target image;
A correction amount determination unit that determines a correction amount for the size of at least one of the left and right eyes based on the detected information;
An image processing apparatus comprising: a deformation unit that performs image deformation processing on at least one of the region including the left eye and the region including the right eye based on the determined correction amount.   The correction amount determining unit determines a correction amount for enlarging the size of the smaller one of the left and right eyes, and the deforming unit is an area including the smaller eye based on the correction amount for enlarging. The image processing apparatus according to claim 1, wherein a deformation process for enlarging the image is performed.   The correction amount determination unit increases the size of the smaller eye as the ratio of the size of the smaller eye to the size of the larger eye among the left and right eyes is higher as the correction amount to be enlarged. The image processing apparatus according to claim 2, wherein a correction amount that approaches the size of the larger eye is determined.   The correction amount determining unit calculates an average value of the sizes of the left and right eyes, and determines a correction amount that substantially enlarges the size of the smaller eye to the average value. 2. The image processing apparatus according to 2.   The image processing apparatus according to claim 2, wherein the correction amount determination unit determines a correction amount that substantially enlarges the size of the smaller eye to the size of the larger eye.   The image processing according to claim 1, wherein the deformation unit deforms an image of a region targeted for deformation processing along a substantially height direction of eyes included in the region. apparatus.   The deformation unit moves, based on the correction amount, a pixel that is located above the face from a line that divides the area and is located below the eye among pixels of the area that is the target of the deformation process. The image processing apparatus according to claim 1, wherein the deformation processing is performed. A face orientation estimating unit for estimating the orientation of the face in the target image,
The image processing apparatus according to claim 1, wherein the correction amount determination unit changes the correction amount in accordance with the face direction estimated by the estimation unit. A detection step of detecting information representing the size of the left and right eyes of the face in the target image;
A correction amount determining step for determining a correction amount for the size of at least one of the left and right eyes based on the detected information;
An image processing method comprising: a deformation step of performing an image deformation process on at least one of the region including the left eye and the region including the right eye based on the determined correction amount. A detection function for detecting information representing the size of the left and right eyes of the face in the target image;
A correction amount determination function for determining a correction amount for the size of at least one of the left and right eyes based on the detected information;
An image processing program for causing a computer to execute a deformation function for performing an image deformation process on at least one of an area including a left eye and an area including a right eye based on the determined correction amount.