JP4888412B2 - Outline detection method, outline detection apparatus, and outline detection program - Google Patents

Description

  The present invention relates to a contour detection method, a contour detection device, and a contour detection program, and more particularly to a contour detection method, a contour detection device, and a contour detection program for detecting the contour of a facial organ.

As a method for detecting the contours of the eyes of a person included in image data, a method for correcting initial parameters representing the contours based on the rotation angle of the face has been proposed (see Patent Document 1). In such a configuration, since the initial parameter representing the contour is corrected based on the rotation angle of the face, it is possible to detect the contour while suppressing the influence of the rotation angle.
JP 2005-339288 A

  However, the position and shape of the eye contour varies from person to person, and there is a problem that it is not possible to set appropriate initial parameters by correcting only the rotation angle. Furthermore, even for the eyes of the same person, the contours of the eyes differ depending on the expression at that time, and there is a problem that it is difficult to set appropriate initial parameters. If the initial parameter is too far from the actual contour, the processing load and processing time required for contour extraction will increase.

  The present invention has been made in view of the above problems, and provides a contour detection method, a contour detection apparatus, and a contour detection program with high accuracy and low processing load.

  In order to solve the above-mentioned problem, first, a first approximation curve to be approximated to the contour of either the left or right eye is initially set, and evaluation for evaluating the closeness of the first approximation curve to the contour of the one eye A value is calculated, and the first approximate curve that improves the closeness based on the evaluation value is searched for. As a result, the contour of either the left or right eye can be detected by the first approximate curve. Next, a second approximate curve to be approximated to the other eye is initialized based on the searched position and shape of the first approximate curve. That is, in the same person, it can be estimated that the contour of one eye resembles the contour of the other eye, so that the first eye approximates the other eye based on the already-searched first approximate curve. The second approximate curve can be appropriately initialized. When the second approximate curve is initially set, similarly, an evaluation value for evaluating the closeness of the second approximate curve to the contour of the other eye is calculated, and the closeness is improved based on the evaluation value. The second approximate curve is searched. Note that the contour of the left eye may be approximated by the first approximate curve, and conversely, the contour of the right eye may be approximated by the first approximate curve.

  In addition, when initializing a second approximate curve to be approximated to the other eye based on the searched first approximate curve, the second approximate curve is formed so as to be symmetrical with the first approximate curve after the search. May be set. Since the eye shape of the same person can be estimated to be nearly symmetrical, the shape of the second approximate curve can be appropriately initialized. In addition, the shape of the second approximate curve may be initially set so as to be a predetermined amount larger than the first approximate curve after the search. In this way, the search can be performed in the direction of reducing the second approximate curve.

  Further, as specific examples of the first approximate curve and the second approximate curve, two points located on both outer sides in the horizontal direction among the three points arranged at different positions of the image data are respectively set as end points, The remaining one point may be a common inflection point and may be represented by a pair of quadratic curves connected to each other at the inflection point. By doing so, the contour of the eye can be accurately expressed by the first approximate curve and the second approximate curve. Alternatively, the position and size of a rectangular area including the left and right eyes may be detected in advance, and the first approximate curve and the second approximate curve may be initially set in each rectangular area. By searching the first approximate curve and the second approximate curve in the rectangular area detected in advance, the search efficiency is higher than when searching the first approximate curve and the second approximate curve in a wide range of the image data. Can be improved.

  Furthermore, since the contour detection result by the first approximate curve is reflected in the initial setting of the second approximate curve, it is desirable that the contour detection by the first approximate curve is accurate. First, the eye with the more accurate contour detection may be predicted, and the predicted eye may be approximated by the first approximate curve. In this way, the initial setting of the second approximate curve can be accurately performed based on the accurately detected first approximate curve.

  Furthermore, the technical idea of the present invention can be implemented not only by a specific contour detection method but also by a contour detection apparatus. That is, the present invention can be specified as a contour detection apparatus having means corresponding to each step performed by the contour detection method described above. Of course, when the above-described contour detection apparatus reads a program to realize each of the above-described means, the technical features of the present invention can be applied to a program for executing a function corresponding to each of the means and various recording media on which the program is recorded. It goes without saying that the idea can be embodied. In addition, it cannot be overemphasized that the outline detection apparatus of this invention can be disperse | distributed not only by a single apparatus but by several apparatuses. For example, each unit included in the contour detection apparatus can be distributed in both the printer driver executed on the personal computer and the printer. Further, each means of the contour detection device of the present invention can be included in a printing apparatus such as a printer.

Hereinafter, embodiments of the present invention will be described in the following order.
1. Contour detection device configuration:
2. Outline detection process flow:
2-1. Left eye contour detection:
2-2. Right eye contour detection:
3. Variations:
3-1. Modification 1:
3-2. Modification 2:
3-3. Modification 3:
3-4. Modification 4:
3-5. Modification 5:

1. Configuration of Contour Detection Device FIG. 1 shows a configuration of a computer that specifically realizes a contour detection device according to an embodiment of the present invention. In FIG. 1, a computer 10 includes a CPU 11, a RAM 12, a ROM 13, a hard disk drive (HDD) 14, a general purpose interface (GIF) 15, a video interface (VIF) 16, an input interface (IIF) 17, and a bus 18. The bus 18 implements data communication between the elements 11 to 17 constituting the computer 10, and communication is controlled by a chip set (not shown). The HDD 14 stores program data 14a for executing various programs including an operating system (OS), and the CPU 11 executes calculations according to the program data 14a while expanding the program data 14a in the RAM 12. .

  The HDD 14 stores image data 14e input by a digital still camera or scanner, and a number of face templates 14b, eye templates 14c, and mouth templates 14d used in pattern matching described later. The GIF 15 provides an interface conforming to the USB standard, for example, and connects an external printer 20 to the computer 10. The VIF 16 connects the computer 10 to an external display 40 and provides an interface for displaying an image on the display 40. The IIF 17 connects the computer 10 to an external keyboard 50a and a mouse 50b, and provides an interface for the computer 10 to acquire input signals from the keyboard 50a and the mouse 50b.

  FIG. 2 shows a software configuration of a program executed in the computer 10. In the figure, an operating system (OS) P1, a contour detection application P2, and a printer driver P3 are executed. The OS P1 provides an interface between the programs, and the printer driver P3 executes a process for controlling the printer 20. The contour detection application P2 includes a face detection unit P2a, a face organ detection unit P2b, and a contour detection unit P2c. Further, the contour detection unit P2c includes an inclination correction unit P2c1, a sampling unit P2c2, a scalar conversion unit P2c3, a conversion function setting unit P2c4, a parameter search unit P2c5, and an evaluation value calculation unit P2c6. Details of processing executed by each of the modules P2a to P2c constituting the contour detection application P2 will be described together with a flow of contour detection processing described later.

2. Flow of contour detection processing FIG. 3 shows a flow of contour detection processing. In step S <b> 100, the face detection unit P <b> 2 a acquires from the HDD 14 image data 14 e targeted for contour detection. In this embodiment, the image data 14e to be subjected to contour detection is bitmap data in which the color indicated by each pixel is expressed by a combination of gradation values of RGB channels (vector in the RGB color space). In the drawings to be described later, a binary image may be shown for the sake of convenience, but in reality, multi-tone color image data is the target of processing. The image data 14e may be compressed when stored in the HDD 14, or the color of each pixel may be expressed in another color space. In these cases, the development of the image data 14e and the conversion of the color space are executed, and the face detection unit P2a acquires the image data 14e of RGB bitmap data. Note that the image data 14e is not limited to that stored in the HDD 14, and may be stored in, for example, a digital still camera or a memory card. By operating the keyboard 50a and the mouse 50b with reference to the UI screen displayed on the display 40, the image data 14e to be subjected to the contour detection process is designated. In step S110, the face detection unit P2a detects a face included in the image data 14e.

  FIG. 4 schematically shows the face detection performed in step S110. In the figure, a plurality of face templates 14b acquired from the HDD 14 by the face detection unit P2a are shown. In the present embodiment, a face is detected from the image data 14e by performing pattern matching between the image data 14e and each face template 14b. Each face template 14b is image data of a certain size, and each face is included. The size and position (center) of the face included in each face template 14b are standardized so as to be substantially constant. In addition, various face templates 14b include various faces so that various faces can be detected. In performing pattern matching with the face template 14b, a rectangular comparison area CA is generated in the image data 14e, and the position of the comparison area CA, the size, and the rotation angle are changed. The similarity with the image of the face template 14b is evaluated.

  When the similarity satisfies a certain standard, it is determined that a face exists, and the position, size, and rotation angle of the comparison area CA at that time are acquired. In the present embodiment, the comparison area CA is rotated by 30 degrees. The face rotated in the image data 14e is matched when the comparison area CA is rotated by the corresponding angle. By moving the comparison area CA over the entire image data 14e, the position, size, and rotation angle of the comparison area CA where one or more faces existing at unspecified positions in the image data 14e can be acquired. . In the present embodiment, the following description will be made assuming that a single face is detected.

  In step S120, the face detection unit P2a converts the image of the area included in the comparison area CA from the image data 14e to the face image data FD based on the position, size, and rotation angle of the comparison area CA matching the face template 14b. Extract. At this time, resolution conversion is performed so that the size of the face image data FD becomes a constant size. In the present embodiment, the pixels of the face image data FD are interpolated or thinned out so as to have a size of 100 × 100 pixels. When the comparison area CA is rotated corresponding to the rotation angle of the face in the image data 14e, the face image data FD is rotated so as to eliminate this rotation. However, since the rotation angle of the comparison area CA is 30 degrees, the face rotation angle can remain in the range of ± 15 degrees in the face image data FD. Further, since the size and position of the face included in each face template 14b are standardized, the position and size of the face in the face image data FD matching this are almost constant. When the face image data FD is obtained as described above, the face organ detection unit P2b detects the eyes and mouth as the face organs in step S130.

  FIG. 5 schematically shows how a facial organ is detected in step S130. The face organ detection unit P2b acquires a large number of eye templates 14c and mouth templates 14d from the HDD 14, and performs pattern matching between these and the face image data FD. Again, a rectangular comparison area CA is generated in the face image data FD, and the similarity between the images inside the comparison area CA and each eye template 14c and each mouth template 14d is changed while changing the position of the comparison area CA. Compare. However, since the position and size of the face in the face image data FD are almost constant and the rotation angle of the face is corrected in units of 30 degrees, there is a large deviation in the size and rotation angle of the eyes and mouth included in the face. You can think of it not. Therefore, here, pattern matching can be performed without changing the size or rotation angle of the comparison area CA. In the present embodiment, the size of the eye template 14c is standardized to a size of 25 × 25 pixels, and the size of the comparison area CA when detecting the eyes is also fixed to 25 × 25 pixels. In addition, since the position (center) and size of the eyes included in each eye template 14c are standardized, it can be considered that the size and position of the eyes in the comparison area CA matching each eye template 14c is also substantially constant. . In the case of a face photographed from the front, normally, the position of the comparison area CA where the left and right eyes and mouth exist is detected. In the present specification and drawings, left and right (LR) mean left and right when the drawing is viewed from the front. Therefore, it is opposite to the right and left in the real face. Note that the eye template 14c includes a large number of left eye and right eye so that the left and right eyes can be detected.

  FIG. 6 shows the face image data FD in which the comparison area CA where the left and right eyes and mouth exist is detected. In the figure, rectangular comparison areas CA1 and CA2 including the left and right eyes in the center and a comparison area CA3 including the mouth are detected. When the detection of each facial organ is completed as described above, in step S140, the inclination correction unit P2c1 of the contour detection unit 2c calculates the coordinates of the center of gravity of the comparison areas CA1 and CA2 including the left and right eyes in the center. The inclination S of the straight line connecting the centroids with respect to the horizontal line is calculated. As described above, since the rotation angle is corrected in units of 30 degrees at the face detection stage, the inclination S corresponding to an angle within ± 15 degrees is basically calculated here. In step S150, the inclination correction unit P2c1 extracts areas belonging to the comparison areas CA1 and CA2 from the face image data FD as the left eye image data LE and the right eye image data RE, respectively. At this time, the inclination correction according to the inclination S calculated in step S140 is performed on the left eye image data LE and the right eye image data RE.

  FIG. 7 conceptually shows the state of inclination correction executed by the inclination correction unit P2c1 in step S150. In the figure, the upper side and the lower side of the rectangular comparison area CA are inclined according to the inclination S, and pixels belonging to the comparison area CA (shown by broken lines) after the inclination are extracted. Since the upper side and the lower side of the comparison area CA are inclined according to the inclination S, the positions of the extracted pixel columns are shifted. Then, by aligning the upper end / lower end positions of the extracted pixel columns, the left eye image data LE and the right eye image data RE are obtained. Thereby, the inclination of the eyes included in the left eye image data LE and the right eye image data RE can be eliminated, and the eyes can be made almost horizontal in the left eye image data LE and the right eye image data RE. In addition, since the pixel position is shifted from the original image, the smoothness of the contour may be inaccurate or the shape of the eye may be distorted, but the corresponding amount of inclination is within ± 15 degrees. It is not a problem because it is just an adjustment. When the left-eye image data LE and the right-eye image data RE are obtained as described above, a process of converting the left-eye image data LE and the right-eye image data RE into image data having a scalar quantity Z is executed in step S200. Note that the vertical position in the left eye image data LE and the right eye image data RE is represented as y, the horizontal position is represented as x, and the upper left corner is x = y = 0.

  FIG. 8 shows the flow of scalar conversion processing (scalar amount map acquisition means) executed in step S200. The left-eye image data LE and the right-eye image data RE are image data in which each pixel has RGB gradation (RGB vector). In the scalar conversion process, the left-eye image data LE and the right-eye image data RE each have a scalar amount Z. A process of converting into image data having information is executed. First, in step S210, the sampling unit P2c2 samples some pixels of the face image data FD.

  FIG. 9 shows the state of sampling in step S210. In the face image data FD, a line segment L1 connecting the lower left corner and the lower right corner of the comparison area CA including the left and right eyes, and a vertical line segment L2 that bisects the line segment L1 between the left and right comparison areas CA are shown. Has been. The length of the line segment L2 is the same as the width of the comparison area CA. The line segments L1 and L2 have a predetermined width, and pixels at positions corresponding to the line segments L1 and L2 are set as sampling targets in the present embodiment. The line segments L1 and L2 correspond to a position at a substantially constant height below the eyes and a position between the left and right eyes. By sampling from such a position, pixels of colors such as outside the outline of the face, eyes, eyebrows, nostrils and mouth are not sampled, and basically only the skin-colored pixels can be sampled. Further, since sampling is performed from both the line segments L1 and L2, even if pixels other than skin color such as a frame of eyeglasses or bangs are included in a part of the line segments L1 and L2, the statistics described later are used. The influence on the indicator can be suppressed. Of course, it is only necessary to narrow the skin color to some extent. For example, sampling may be performed from another region such as a cheek or forehead.

前記の（１）式において、Ｓ_R，Ｓ_G，Ｓ_BはＲＧＢ各チャネルについての分散を示しており、Ｓ_RG，Ｓ_GB，Ｓ_RBはＲ−Ｇ，Ｇ−Ｂ，Ｒ−Ｂ間の共分散を示している。 The sampling unit P2c2 acquires the RGB gradation of the pixel to be sampled, and calculates the average μ (μ _R , μ _G , μ _B ) of the RGB gradation and the variance / covariance matrix S as a statistical index (average calculation) means). The average μ can be obtained by calculating an arithmetic average of gradations for each RGB channel. The variance / covariance matrix S is defined by the following equation (1).

In the above equation (1), S _R , S _G , and S _B indicate dispersion for each RGB channel, and S _RG , S _GB , and S _RB are between _RG , _GB , and _RB . Covariance is shown.

FIG. 10 is a graph showing the average μ (μ _R , μ _G , μ _B ) obtained by sampling in step S210 and the variances S _R , S _G , S _B. In the present invention, it is assumed that the distribution of the color indicated by the sampled pixels in the three-dimensional RGB color space is a normal distribution N (μ, S), and the RGB gradation distributions are average μ (μ _R , μ _{The state} assumed to be a normal distribution N _R (μ _R , S _R ), N _G (μ _G , S _G ), N _B (μ _B , S _B ) centered on _G , μ _B ) is illustrated. Yes. The average μ (μ _R , μ _G , μ _B ) indicates the most standard skin color in the detected face, and the variances S _R , S _G , S _B indicate the degree of gradation variation of each channel indicating the skin color. Show. Usually, the variances S _R , S _G , and S _B have different sizes, but the mean μ (μ _R , μ _G , μ _B ) is added to or subtracted from the square root (standard deviation) of each variance S _R , S _G , S _B. The probability of the range surrounded by the values is 68.2%.

In this specification, “skin color” does not mean a specific color whose absolute color value is specified, but a color distributed in the vicinity of an average μ (μ _R , μ _G , μ _B ). To do. Therefore, the absolute color meaning “skin color” varies depending on the face to be processed. When the average μ and the dispersion matrix S are obtained, in step S220, the scalar conversion unit P2c3 acquires left-eye image data LE and right-eye image data RE in which each pixel has RGB gradation. The subsequent processing is performed for each of the left-eye image data LE and the right-eye image data RE. The processing for the left-eye image data LE will be described as an example.

In step S230, the scalar conversion unit P2c3 sequentially selects one pixel from the left-eye image data LE, and substitutes the RGB gradation (r, g, b) of the selected pixel into the following equation (2), thereby detecting Mahalanobis. Convert to square distance D _M ² .

In the above equation (2), Δr is (r−μ _R ), Δg is (g−μ _G ), and Δb is (b−μ _B ). The Mahalanobis square distance D _M ² is the average μ (μ _R , μ _G , μ _B ) indicating the standard skin color obtained by the above-described skin color sampling, and the RGB gradation (r, g, b) of the selected pixel. The index value corresponding to the square distance in the RGB color space (the shift amount of the RGB gradation values), and the magnitude thereof does not depend on the color shift direction in the RGB color space. That is, if the Mahalanobis square distance D _M ² is the same, even if the color shift directions from the average μ (μ _R , μ _G , μ _B ) are different, there is a difference of the same degree in terms of probability. In the following, it is possible to perform processing ignoring the color misregistration direction in the RGB color space.

FIG. 11 shows isolines of Mahalanobis square distance D _M ^{2 in} the RGB color space (RG plane). In the figure, the horizontal axis indicates the R gradation (r), and the vertical axis indicates the G gradation (g). The isolines are substantially elliptical due to differences in the variances S _R , S _G , and S _B of the RGB channels. Mean μ (μ _R, μ _G) Mahalanobis square distance D _M ² in the zero, the average in the RG plane μ (μ _R, μ _G) Mahalanobis square distance D _M ² as the distance from is large. For this reason, white-eye and black-eye pixels different from the skin color included in the left-eye image data LE have a larger Mahalanobis square distance D _M ² than the skin-colored pixels. As described above, the RGB gradation (r, g, b) as a vector indicating the pixel color is converted into the Mahalanobis square distance D _M ² as a scalar indicating the deviation amount of the RGB gradation value from the standard skin color. can do. When the conversion to the Mahalanobis square distance D _M ² is completed for all the pixels of the left eye image data LE, the left eye image data LE has been converted to the image data D _M ² (x, y) of the Mahalanobis square distance D _M ^2. Become.

前記の（３）式においてα、ｕは前記変換関数の変換特性を決定付けるパラメータである。本実施形態では、α＝１に固定し、ｕの初期値を３とする。 In step S240, the scalar conversion unit P2c3 sequentially selects the pixels of the left-eye image data LE, and substitutes the Mahalanobis square distance D _M ² of the selected pixels into the conversion function expressed by the following equation (3), thereby obtaining the scalar quantity Z. It will be converted to.

In the above equation (3), α and u are parameters that determine the conversion characteristics of the conversion function. In the present embodiment, α = 1 is fixed, and the initial value of u is 3.

FIG. 12 shows the conversion characteristics of the conversion function. In the figure, the horizontal axis indicates the Mahalanobis square distance D _M ² before conversion, and the vertical axis indicates the scalar quantity Z after conversion. Further, the distribution of Mahalanobis square distance D _M ² before conversion of certain left-eye image data LE and the distribution of scalar quantity Z after conversion are also shown. Scalar quantity Z is -1 next time Mahalanobis square distance D _M ² is 0, the Mahalanobis square distance D _M ² represented by a nonlinear function becomes 0 when parameter u. Here, in the Mahalanobis square distance D _M ² on the horizontal axis, a region centered on the parameter u = 3 is represented as a second region A2, and a region in which the value of the Mahalanobis square distance D _M ² is larger than the second region A2 is represented. It shall represent with 1st area | region A1. In the second area A2, the slope of the conversion function is steeper than in the other areas, and the amount of variation of the scalar amount Z corresponding to the unit variation of the Mahalanobis square distance D _M ² is larger than in the other regions. ing. In the first region A1 where the Mahalanobis square distance D _M ² is larger than the second region A2, the scalar amount Z after conversion has a characteristics such as going to saturate gradually 1. The second region A2 includes the Mahalanobis square distance D _M ² in the region corresponding to the skin color, is preferably set between the non Mahalanobis squared distance flesh color corresponding to D _M ² area. The position of the second area A2 is set by the parameter u, but the initial parameter u = 3 is set based on the following grounds.

FIG. 13 is a graph showing the relationship between the Mahalanobis square distance D _M ² and the probability distribution. In the figure, the horizontal axis indicates the Mahalanobis square distance D _M ² . On the other hand, the vertical axis represents the normal distribution N (μ, S defined by the mean μ (μ _R , μ _G , μ _B ) of the RGB gradation of the skin color pixels sampled in step S210 and the variance / covariance matrix S. ) Shows a probability distribution (lower probability). As shown in the figure, the Mahalanobis square distance D _M ² follows an n-dimensional chi-square distribution. In the present embodiment, a color image represented by RGB gradation values is a processing target, and the probability corresponding to each Mahalanobis square distance D _M ² is estimated by a three-dimensional chi-square distribution (shown by a solid line). it can. When the Mahalanobis square distance D _M ² = 3, the lower probability is about 60%. That is, about 60% of the pixel indicating the flesh color, it can be seen that the Mahalanobis square distance D _M ² is 3 or less. If the Mahalanobis square distance D _M ² exceeds 3, it can be inferred that the pixel becomes suspicious to show skin color. That is, the Mahalanobis area nearby square distance D _M ² comes beyond 3, the Mahalanobis square distance D _M ² in the region corresponding to the skin color, Mahalanobis squared distance between the D _M ² in the region corresponding to the non-skin-color It can be estimated that this is a region. For this reason, in the present embodiment, the initial value of the parameter u is set to 3, and an area in the vicinity where the Mahalanobis square distance D _M ² is 3 is set as the second area A2. In the case of a monochrome image, the Mahalanobis square distance D _M ² follows a one-dimensional chi-square distribution (shown by a broken line).

In the distribution of the Mahalanobis square distance D _M ² before conversion by the conversion function shown in FIG. 12, the skin color distribution G1 centered on the Mahalanobis square distance D _M ² = 0 and the white (white eye) included in the left-eye image data LE. ) Distribution G2 and black (black eyes, eyebrows, etc.) distribution G3. On the other hand, in the distribution of the scalar quantity Z after conversion, the skin color distribution G1 is distributed around the scalar quantity Z = −1, and the white distribution G2 and the black distribution G3 included in the left-eye image data LE are distributions. The whole is converted to a value almost saturated to 1. In addition, the bright skin color and the dark skin color compared to the standard skin color indicated by the average μ (μ _R , μ _G , μ _B ) are located near the initial value 3 of the parameter u, and the slope is It exists in the steep second area A2.

By converting the Mahalanobis square distance D _M ² of all the pixels of the left eye image data LE into the scalar quantity Z by the conversion function described above, the left eye image data LE is converted into the image data Z (x, y) of the scalar quantity Z. Can be converted. Hereinafter, the left-eye image data LE and the right-eye image data RE converted into the image data Z (x, y) of the scalar quantity Z are also expressed as a Z map. The Z map corresponds to a scalar quantity map. The Mahalanobis square distance D _M ² can also be used as an indicator of skin color, but according to the scalar amount Z converted by the above-described conversion function, it is possible to more clearly determine whether or not it is likely to be skin color. Can do.

  In step S250, the conversion function setting unit P2c4 counts pixels in which the scalar amount Z exceeds the predetermined threshold Th1, and whether or not the number ratio (area ratio) of the pixels exceeding the threshold Th1 exceeds the predetermined threshold Th2. Determine. In this embodiment, the threshold value Th1 = 0.6 and the threshold value Th2 = 33%. In FIG. 12, the threshold value Th1 = 0.6 is indicated by a broken line, and a pixel exceeding the threshold value Th1 = 0.6 is regarded as saturated, and is determined not to be a skin color. Also, the white distribution G2 and the black distribution G3 correspond to pixels exceeding the threshold Th1 = 0.6. That is, it is possible to determine whether each pixel of the left eye image data LE is a skin color or not a skin color by threshold determination based on the threshold Th1. It can be said that the larger the ratio of the pixels exceeding the threshold Th1, the larger the area ratio in the left-eye image data LE of the pixels that do not look like skin color.

  Here, the left eye image data LE represents an image of the left eye and its surrounding area, and the area occupied by the left eye in the left eye image data LE is the same as that of the eye template 14c matching the left eye image data LE. As described above, since the eye size is standardized in the eye template 14c, the eye size in the left eye image data LE can be considered to be the same as the eye size standardized in the eye template 14c. Therefore, in the Z map, it can be considered that the area ratio occupied by pixels that do not look like a skin color also has a certain tendency. In the present embodiment, it is assumed that it is appropriate that the area ratio occupied by pixels that do not appear to be skin-colored exceeds the threshold Th2 = 33%, and the area ratio that is occupied by pixels that do not appear to be skin-colored is the threshold Th2 = 33% or less. The conversion function setting unit P2c4 changes the parameter u of the conversion function described above in step S260. Since the parameter u is an estimated value when a normal distribution N (μ, S) is assumed, it is desirable to adjust the parameter u according to the validity based on the actual area ratio as in the present embodiment. Note that the threshold Th1 and the threshold Th2 in the present embodiment are examples, and different values may be adopted. Further, an appropriate range of the area ratio may be defined not only by the threshold value Th2 but also by an upper limit value and a lower limit value.

In step S260, the parameter u is changed to four times the original value, and the process returns to step S240. Then, in step S250, to apply the changed parameter u in the equation (3), again, by converting the Mahalanobis square distance D _M ² of the left eye image data LE scalar quantity Z, again, to obtain a Z map. In step S250, the validity of the area ratio of pixels that are not likely to be skin-colored in the Z map is similarly determined, and if not valid, the parameter u is again multiplied by four. By repeatedly executing the above processing, the conversion function can be optimized until the area ratio of pixels that do not look like a skin color is appropriate.

In the present embodiment, since the parameter u is sequentially increased, the conversion characteristic by the conversion function changes as indicated by a broken line in FIG. That is, it is possible to gradually shifting the second region A2 where the slope is steeper in the Mahalanobis square distance D _M ² increases direction. Along with this, the width of the first region A1 is narrowed. By doing so, it is possible to set an appropriate conversion function even for a face with a skin color variation. If it is determined in step S250 that the ratio of the pixels whose scalar amount Z exceeds the threshold value Th1 exceeds the predetermined threshold value Th2, the scalar conversion process is terminated without changing the parameter u, and the RGB of each pixel is determined. The left eye image data LE (Z map) in which the gradation is converted into the gradation value of the scalar quantity Z is output to the parameter search unit P2c5. In the present embodiment, the parameter u is changed. However, the parameter α may be changed in order to adjust the range of the first area A1 and the second area A2. In order to widen the first area A1, the parameter α should be increased, and in order to widen the second area A2, the parameter α should be decreased. Of course, the parameter u may be set larger and gradually changed to be smaller.

  FIG. 14 shows an example of a Z map in which each pixel is represented by a scalar quantity Z (x, y). In the figure, the scalar quantity Z (x, y) in the vertical section and the horizontal section of the left-eye image data LE is shown. In the vertical section, the portion corresponding to the black eye has a scalar amount Z (x, y) close to 1, and the portion corresponding to the other skin color has a value between −1 and 0. . At the boundary between the black eye and the skin color constituting the outline of the eye, the vertical gradient is large. On the other hand, in the horizontal cross section, the portions corresponding to the black eyes and the white eyes both have a scalar amount Z close to 1, and the other portions corresponding to the skin color have values close to -1. The gradient in the horizontal direction at the boundary between the white eye and the skin color constituting the outline of the eye is a large value, but the gradient is extremely small at the boundary between the black eye and the white eye. In the above description, the processing for converting the left eye image data LE into the Z map has been described as an example. However, the same processing is executed for the right eye image data RE, and the right eye image data RE is also converted into the Z map.

2-1. Up to now, the left eye image data LE and the right eye image data RE are processed in common, but from the next step S160 (FIG. 3), a specific process is performed on the left eye image data LE. . In step S160, the parameter search unit P2c5 acquires a Z map obtained by converting the left eye image data LE. In step S170, the parameter search unit P2c5 initializes the eye contour parameters L, R, T, and B in the Z map.

FIG. 15 shows the contour parameters L, R, T, and B in the Z map of the left eye. The contour parameters L, R, T, and B correspond to the corner of the left eye, the top of the eye, the upper vertex, and the lower vertex, respectively, and L (x _L , y _L ), R ( x _R , y _R ), T (x _T , y _T ), and B (x _B , y _B ). When initial values of the contour parameters L, R, T, and B can be set, a contour line O (first approximate curve) for approximating the left eye can be generated (first approximation means). In the present embodiment, the contour line O is approximated by a curve O1 in the upper left portion, a curve O2 in the upper right portion, a curve O3 in the lower left portion, and a curve O4 in the lower left portion. Each curve O1-O4 is represented by the following formula (4).

In the above equation (4), the curve O1 is expressed by an upwardly convex quadratic curve passing through the coordinates of the contour parameter L with the coordinates of the contour parameter T as the inflection point. a ₁ and a ₂ are positive, and a ₃ and a ₄ are negative. The curve O2 is expressed by a quadratic curve that is convex upward and passes the coordinates of the contour parameter R with the coordinates of the contour parameter T as the inflection point. On the other hand, the curve O3 is represented by a quadratic convex curve that passes through the coordinates of the contour parameter L with the coordinates of the contour parameter B as inflection points. The curve O4 is represented by a downwardly convex quadratic curve passing through the coordinates of the contour parameter R with the coordinates of the contour parameter B as inflection points. When the coordinates of the contour parameters L, R, T, and B are determined, the curves O1 to O4 are uniquely determined. Therefore, by initializing the coordinates of the contour parameters L, R, T, and B, the position and shape of the contour line O Will be initialized. Note that the coordinates of the contour parameters L, R, T and the contour parameters L, R, B are equivalent to the end points where the coordinates of the contour parameters L, R are located on both outer sides in the horizontal direction. The coordinates of the contour parameters T and B correspond to a common inflection point. In order to arrange the contour line O as described above, the initial values of the contour parameters L, R, T, and B are at least x _L <x _T <x _R , x _L <x _B <x _R , y _T <y _It suffices if _L <y _B and y _T <y _R <y _B are satisfied. In the present embodiment, as shown in FIG. 15, the left-eye image data LE is set to be symmetrical with respect to the central vertical line and symmetrical with respect to the horizontal line slightly below the center. In addition, initial values of the coordinates of the contour parameters L, R, T, and B are set slightly outside the contour of the eye standardized by the eye template 14c (illustrated by a broken line).

  As described above, when the initial values of the coordinates of the contour parameters L, R, T, and B are set in the Z map, in step S180, the parameter search unit P2c5 and the evaluation value calculation unit P2c6 perform optimum contour parameters L, R, T and B search processing is performed. When the parameter search unit P2c5 updates the contour parameters L, R, T, and B and outputs the updated contour parameters L, R, T, and B to the evaluation value calculation unit P2c6, the evaluation value calculation unit P2c6 V is calculated, and the evaluation value V is returned to the parameter search unit P2c5. Then, the parameter search unit P2c5 searches for the coordinates of the contour parameters L, R, T, and B that maximize the evaluation value V.

FIG. 16 schematically illustrates the concept of the evaluation value V. In FIG. 16, a microline element having a length dl on the outline O and a normal unit vector p of the microline element are shown. Since the curves O1 to O4 can be specified by the above equation (4), the normal unit vector p for an arbitrary minute line element can be obtained. The normal unit vector p is set so that the inner direction is positive. That is, the y component of the normal unit vector p of the minute line element on the curves O1 and O2 is positive in the downward direction, and the y component of the normal unit vector p of the minute line element on the curves O3 and O4 is upward. Is positive. Further, the x component of the normal unit vector p of the minute line element on the curves O1 and O3 is positive in the right direction, and conversely, the x component of the normal unit vector p of the minute line element on the curves O2 and O4 is in the left direction. Is positive. The gradient vector g is expressed by the following equation (5).

In the above equation (5), the gradient vector g is given by a horizontal gradient and a vertical gradient, and becomes larger in a region where the variation of the scalar quantity Z (x, y) indicating the skin color is more severe. The evaluation value calculation unit P2c6 uses the normal unit vector p and the gradient vector g described above, and calculates an evaluation value V for evaluating the closeness of the contour line O to the contour by the following equation (6).

  In the above equation (6), the evaluation value V is obtained by performing line integration (total) along the contour O of the inner product (small evaluation value) of the normal unit vector p and the gradient vector g regarding the minute line element. However, the integral value relating to the curves O1 and O2 in the upper part of the contour line O is weighted twice as much as the integral value relating to the curves O3 and O4 in the lower part of the contour line O. The inner product of the normal unit vector p and the gradient vector g becomes larger as the normal unit vector p and the gradient vector g are in the same direction and the gradient vector g is larger. Accordingly, each microline element constituting the contour O is orthogonal to the gradient direction of the scalar quantity Z (x, y), and the evaluation value V increases as the gradient increases.

Since the gradient of the scalar quantity Z (x, y) can be considered to be a degree that the skin color-likeness fluctuates, when the evaluation value V is large, the contour line O passes through a region where the skin-color-likeness fluctuation is large. Can be evaluated. That is, when the evaluation value V is large, it can be considered that the contour line O passes through the contour of the eye where the variation in skin color is large. In particular, the scalar amount Z (x, y) is converted so as to fluctuate severely in the second region A2 suspected of being a skin color, so that the gradient becomes extremely large in the vicinity of the contour of the eye that is not a skin color. On the other hand, as shown in FIG. 12, the white distribution G2 and the black distribution G3 are both saturated to a value close to 1, and the scalar quantity Z (x, y) of the boundary between the white eye and the black eye is also obtained. The gradient will be small. That is, even if the white distribution G2 and the black distribution G3 show different values at the Mahalanobis square distance D _M ² , the Mahalanobis of the white distribution G2 and the black distribution G3 is converted by converting to a scalar quantity Z by a conversion function. The difference in the square distance D _M ² can be converted into a minute one, and the gradient of the scalar quantity Z can be prevented from occurring between them. Therefore, it is possible to prevent the evaluation value V from increasing at the boundary between the white eye and the black eye, and to clearly distinguish the boundary between the white eye and the black eye and the outline of the eye. In order to facilitate conceptual understanding, the evaluation value V, the gradient vector g, and the normal vector n have been described to be calculated on a continuous image plane. In this case, an equivalent operation is performed. In the search processing (search means, contour detection means) described below, the coordinates of contour parameters L, R, T, and B that increase the evaluation value V are searched.

  FIG. 17 schematically illustrates a search procedure in the search process. In the figure, the movement patterns of the contour parameters L, R, T, and B are shown, and the movement pattern is composed of the first to fourth phases. In the first phase, the contour parameters L, R are changed to four coordinates (b, c, d, e) shifted by two pixels in the diagonal four directions from the coordinates (a) of the current contour parameters L, R, T, B. , T, B are moved. In the second phase, the contour parameters L, R, T, B are shifted from the coordinates (a) of the current contour parameters L, R, T, B by two pixels in the vertical and horizontal directions (b, c, d, e). Move T and B. In the third phase, the contour parameters L, R are changed to four coordinates (b, c, d, e) shifted by one pixel in the four diagonal directions from the coordinates (a) of the current contour parameters L, R, T, B. , T, B are moved. In the fourth phase, the outline parameters L, R, T, B are shifted to the four coordinates (b, c, d, e) which are shifted by one pixel in the vertical and horizontal directions from the coordinates (a) of the current outline parameters L, R, T, B. Move T and B. The search process is terminated when the fourth phase is completed.

FIG. 18 shows a detailed search procedure in each phase. First, the contour parameters L, R are centered on the coordinates (a) of the current contour parameters L, R, T, B determined in the immediately preceding phase (initial values in the first phase) according to the movement pattern shown in FIG. , T, B are moved. The contour parameters L, R, T, and B are not moved simultaneously, but are moved in the order of L → R → T → B. First current contour parameter L, and outputs R, T, and B to the evaluation value calculation unit P2c6, calculates an evaluation value V _a. Next, the contour parameters R, T, and B are fixed, and the contour parameter L is sequentially moved to four coordinates (b, c, d, e) around the current coordinate (a). At this time, every time the contour parameter L moves, the contour parameters L, R, T, and B are output to the evaluation value calculation unit P2c6, and the evaluation values V _a , V _b , V _c , V _d , and V _e are calculated. The order of movement to the four coordinates (b, c, d, e) may be any. When the evaluation values V _a , V _b , V _c , V _d , and V _e for the contour parameter L of the four coordinates (a, b, c, d, e) are obtained as described above, the evaluation values V _a , Based on V _b , V _c , V _d , and V _e , the coordinate (h) of the contour parameter L that maximizes the evaluation value V is predicted.

First, a quadratic curve passing through the evaluation values V _a , V _b , and V _c regarding the line segment b-ac passing through the coordinates (a) of the current contour parameter L is calculated, and the quadratic curve is maximized. Calculate the coordinates. Only when the quadratic curve is convex upward and the inflection point is between the line segments b-a-c, the coordinates that maximize the quadratic curve can be calculated. When the coordinate (f) that maximizes the quadratic curve cannot be calculated, the larger coordinate (b) or coordinate (c) of the end evaluation values V _b and V _c is set as the coordinate (f). Then, a straight line l1 parallel to the line segment b-a-c and passing through the coordinate (f) is generated. Next, _a quadratic curve that passes through the coordinates (a) of the current contour parameter L and passes through the evaluation values V _a , V _d , and V _e regarding the line segment d-a-e orthogonal to the line segment b-a-c. And the coordinates (g) for maximizing the quadratic curve are calculated in the same procedure. Then, a straight line l2 parallel to the line segment dae and passing through the coordinates (g) is generated.

When the straight lines l1 and l2 can be generated as described above, the coordinates of these intersections are calculated as the coordinates (h) of the contour parameter L. When this coordinate (h) can be calculated, the contour parameter L is moved to the coordinate (h), and the evaluation value V _h at that time is calculated. When the six evaluation values V _a , V _b , V _c , V _d , V _e , and V _h can be calculated as described above, the coordinate corresponding to the largest one of these is obtained as the optimum contour in the phase. Determined as the coordinates of the parameter L. When the optimum coordinates for the contour parameter L are determined, the same processing is performed for the contour parameter R to determine the optimum coordinates. Further, the same processing is sequentially performed on the contour parameters T and B to determine the optimum coordinates, the phase is terminated, and the process proceeds to the next phase. When the fourth phase is completed, the contour parameters L, R, T, and B are finally determined.

  As described above, in the first and second phases of the initial search, the contour parameters L, R, T, and B are moved over a wide range. , Close to the top and bottom vertices. Further, in the third and fourth phases in the latter half of the search, the contour parameters L, R, T, and B are locally moved within a narrow range, so that the contour parameters are converged so as to converge to the corner of the eye, the top of the eye, the upper vertex, and the lower vertex. The coordinates of L, R, T, and B can be finely adjusted. However, the search distance in the first to fourth phases should be set according to the size of the left eye image data LE, and it is desirable to increase the search distance as the left eye image data LE increases. The search distance may be gradually shortened, and a search of four phases or more may be performed. In addition, since the initial values of the contour parameters L, R, T, and B are set slightly below the center, it is possible to prevent erroneous detection of the contours of the black eyebrows and the skin color. Also, an appropriate contour line O is set according to the coordinates of the initial contour parameters L, R, T, and B, and the coordinates of the contour parameters L, R, T, and B are moved little by little in each phase. Therefore, the shape of the contour line O does not become extremely abnormal. For example, by moving only the contour parameter L in a concentrated manner, it is possible to prevent the contour parameter L from converging on the boundary between black eyes and white eyes. Moreover, since the evaluation value V evaluates the orthogonality with respect to the gradient vector g with respect to the entire contour line O, it is possible to prevent one point of the contour parameters L, R, T, and B from converging to an abnormal point.

Furthermore, since the initial values of the contour parameters L, R, T, and B are set slightly outside the eye contour standardized by the eye template 14c, the contour parameters L, R, T, and B move inward and converge to optimum coordinates. Therefore, it is possible to prevent the positional relationship between the contour parameters L, R, T, and B from being reversed during the search. Since the contour parameters L, R, T, and B move independently of each other, even if the contour of the left eye is vertically asymmetrical or laterally asymmetrical, fitting can be performed with high accuracy using the four quadratic curves O1 to O4. Further, by moving the contour parameters L, R, T, and B in the order of L → R → T → B, the position is shifted ahead of the contour parameters T and B that tend to fluctuate greatly depending on the state of the eyelid. The stable contour parameters L and R can be searched, and the overall search accuracy and search efficiency can be improved. In the search process, the positional relationship between the contour parameters L, R, T, and B does not satisfy x _L <x _T <x _R , x _L <x _B <x _R , or the aspect ratio of the eyes ( If the ratio of the distance between the LR and the TB is clearly abnormal, the search may be terminated in error. Depending on the facial expression of the eye, y _B <y _R and y _B <y _L may be considered. In this case, the signs of a ₃ and a ₄ are inverted in the middle of the search. The search may be terminated when the constant evaluation value V is reached, and the contour parameters L, R, T, and B at that time may be output.

As shown in FIG. 14, in the region corresponding to the eye outline, the scalar quantity Z (x, y) always increases toward the inside of the outline. That is, the gradient vector g always faces inward in the minute line element on the contour line. On the other hand, since the sign of the normal vector n is always defined so that the inner direction of the curves O1 to O4 is positive, the relationship between the gradient direction of the scalar quantity Z (x, y) and the curves O1 to O4 is given. Can be kept constant. That is, it is possible to prevent the coordinates of the contour parameter T from converging on the lower outline of the eye and the coordinates of the contour parameter B from being converged to the upper outline of the eye. Further, since the lower contour line tends to be ambiguous compared to the upper contour line, the emphasis is placed on the upper contour line with respect to the lower contour line in the evaluation value V, thereby making the ambiguous lower contour line. It can prevent that the whole detection accuracy falls by the influence of a line. Further, the scalar quantity Z (x, y) is calculated based on the average μ (μ _R , μ _G , μ _B ) of the skin color pixels sampled from the vicinity of the eyes, so The contour can be detected without depending on the skin color (in the sense).

2-2. Right Eye Contour Detection When the left eye contour can be detected as described above, the finally searched contour parameters L, R, T, and B are stored in the RAM 12, and the right eye contour detection is subsequently executed. In step S190 (FIG. 3), the parameter search unit P2c5 acquires a Z map obtained by converting the right eye image data RE. In step S192, the parameter search unit P2c5 initializes the eye contour parameters L, R, T, and B in the right eye Z map. Here, the coordinates of the contour parameters L, R, T, B indicating the contour of the left eye are read from the RAM 12, and the contour parameters L, R for approximating the right eye based on the coordinates of the contour parameters L, R, T, B are read out. , T, B are initialized.

FIG. 19A shows contour parameters L, R, T, and B for the left eye searched in step S180. By specifying the coordinates of the contour parameters L, R, T, and B in the left-eye Z map, the contour line O as the first approximate curve having high closeness to the contour of the left eye can be uniquely specified. In FIG. 19B, the contour parameters L, R, T, and B that are initially set in the right-eye Z map in step S192 and the second approximate curve that is uniquely specified by the contour parameters L, R, T, and B are shown. The outline O is shown. As shown in the figure, the right eye contour parameters L, R, T, and B are initially set so that the right eye contour O is symmetrical with respect to the left eye contour O with respect to the central vertical line. . Note that the coordinates of the contour parameters L, R, T, and B of the left eye after the search are L (x _L , y _L ), R (x _R , y _R ), T (x _T , y _T ), and B (x _B , Y _B ), the coordinates of the default right eye contour parameters L, R, T, B are L (25−x _R , y _R ), R (25−x _L , y _L ), T ( 25-x _T , y _T ), B (25-x _B , y _B ). That is, the coordinate of x is replaced with a value (25−x) obtained by horizontally inverting, and the positional relationship between the left and right contour parameters L and R is exchanged.

  When the right eye contour parameters L, R, T, and B can be initialized as described above, the (second) search process is executed in step S194. Here, since the same processing as the processing for the left eye performed in step S180 is performed, description thereof is omitted. In the search process in step S194, the search can be started from an outline O that is symmetrical to the outline of the left eye searched in step S180. Since it can be estimated that the left and right eyes of the same person have a nearly symmetrical contour shape, the search can be started from a shape close to reality. It can also be considered that the state of the left and right eyes accompanying the change in facial expression is similar. That is, the left and right eyelids are simultaneously opened and closed unless otherwise noted. Also from this point, it can be said that the search can be started from a shape close to reality. Since the search can be started from a shape that is close to reality, the distance that the contour parameters L, R, T, and B of the right eye should move in the search can be shortened. For example, the search process is started from the third phase. The contour parameters L, R, T, and B can be moved to appropriate positions. Therefore, it is possible to reduce the processing load in contour detection.

  When the search for the contour parameters L, R, T, and B for the right eye is completed as described above, the contour line O defined by the final contour parameters L, R, T, and B for the left and right eyes is moved to the left and right in step S196. Output as the outline of the eyes. The contour line O detected as described above can be used for facial expression determination and focus determination, for example. Alternatively, image processing according to the shape of the contour line O may be executed, and printing may be executed by the printer driver P3 and the printer 20. Further, the image data 14e may be managed by associating and storing the contour line O with the image data 14e.

3. Modification 3-1. Modification 1
FIG. 20 schematically shows the flow of contour detection processing according to the modification. In this modification, monochrome image data 14e in which each pixel is represented by a luminance gradation is a target for contour detection. In this modified example, pixels having flesh color luminance are sampled as in the above-described embodiment, and the normal distribution N (μ, S) is obtained. Then, the Mahalanobis square distance D _M ² is calculated for the luminance gradation of each pixel of the left-eye image data LE, thereby converting it to the Mahalanobis square distance D _M ² . In this modification, it is only necessary to consider a one-dimensional normal distribution N (μ, S), and therefore the Mahalanobis square distance D _M ² is a variance S of the absolute value of the deviation between the luminance gradation of each pixel and the average luminance gradation μ. The value divided by is the Mahalanobis square distance D _M ² . Further, by substituting the deviation D of each pixel into the conversion function expressed by the above equation (3), it is converted into image data (Z map) of scalar Z (x, y). The parameter u at this time is 1. When the Z map is obtained as described above, the evaluation value V can be calculated as in the previous embodiment, and the same search can be performed. In this modification, a one-dimensional normal distribution N (μ, S) relating only to the luminance gradation is assumed, and the parameter u = 1 corresponds to a percent point of 68.2% as shown by a broken line in FIG.

すなわち、輪郭線Ｏの全体について均等な重み付けを行って、評価値Ｖを算出することとしている。このようにすることにより、上輪郭と下輪郭を同等に評価することができる。さらに、評価値Ｖを下記の（８）式によって定義するようにしてもよい。

前記の（８）式においては、輪郭線Ｏに関する積分をするにあたり、水平方向の位置ｘに応じた重みｗ（ｘ）が積算される。 3-2. Modification 2
In this modification, the evaluation value V is defined by the following equation (7).

That is, the evaluation value V is calculated by equally weighting the entire contour line O. By doing in this way, an upper outline and a lower outline can be evaluated equally. Furthermore, the evaluation value V may be defined by the following equation (8).

In the equation (8), the weight w (x) corresponding to the position x in the horizontal direction is integrated when integrating the contour line O.

  FIG. 21 shows an example of the weight w (x). In the figure, the weight w (x) is larger than the other parts at the center in the horizontal direction. By doing in this way, the approximation accuracy of the outline O can be improved about the center part of a horizontal direction among the outlines of eyes. When a face is photographed from the front, the horizontal center portion of the eye contour is clearly photographed, so that the positions of the top and bottom vertices of the eyes can be detected reliably.

前記の（９）式において、Ｏ_uは上側輪郭線（曲線Ｏ１，Ｏ２）を示し、Ｏ_lは下側輪郭線（曲線Ｏ３，Ｏ４）を示し、ｋは任意の定数を示し、Ｑは多数の目テンプレート１４ｃに含まれる目の平均面積を示している。前記の（９）式の第２項は、上側輪郭線Ｏ_uと下側輪郭線Ｏ_lによって囲まれた面積が、目テンプレート１４ｃに含まれる目の平均面積に等しいか否かを評価する項となっている。予め目テンプレート１４ｃとのパターンマッチングを行っているため、上側輪郭線Ｏ_uと下側輪郭線Ｏ_lによって囲まれた面積は、目テンプレート１４ｃに含まれる目の平均面積に近くなるのが妥当である。前記の（９）式の第２項によれば、上側輪郭線Ｏ_uと下側輪郭線Ｏ_lによって囲まれた面積の妥当性を評価することができ、輪郭検出の精度を向上させることができる。 3-3. Modification 3
In this modification, the evaluation value V is defined by the following equation (9).

In the above equation (9), O _u represents the upper contour line (curves O1, O2), O _l represents the lower contour line (curves O3, O4), k represents an arbitrary constant, and Q is a large number. The average area of the eyes included in the eye template 14c is shown. The second term of the expression (9) evaluates whether or not the area surrounded by the upper contour line O _u and the lower contour line O _l is equal to the average area of the eyes included in the eye template 14c. It has become. Since the performing pattern matching with a pre eye template 14c, the area surrounded by the upper contour line O _u and the lower contour line O _l is reasonable that the closer to the average area of the eyes included in the eye template 14c is there. According to the second term of the equation (9), the validity of the area surrounded by the upper contour line O _u and the lower contour line _Ol can be evaluated, and the accuracy of contour detection can be improved. it can.

3-3. Modification 3
In the above description, the contour detection of the eyes is exemplified, but the contours of other facial organs can be detected by the same method. For example, when detecting the outline of the mouth, the skin amount distribution can be investigated by the same method, and the scalar amount Z can be calculated. In this case, the scalar amount Z corresponding to the red pixel indicating the lips and the white pixel indicating the teeth is converged to 1, and the conversion to the scalar amount Z in which the inclination near the boundary between the skin color and the non-skin color is steep is performed. Will be. Accordingly, the contour of the mouth can be accurately detected without erroneously detecting the boundary between the lips and the teeth as the contour. Further, the contour of the mouth can be accurately approximated by the four quadratic curves O1 to O4.

3-4. Modification 4
FIG. 22 shows contour parameters L, R, T, and B that are initially set in the right-eye Z map in step S192 (FIG. 3), and contours that are uniquely specified by the contour parameters L, R, T, and B. A modification of line O is shown. As shown in the figure, the right eye contour parameters L, R, T, and B are initialized so as to be slightly larger than a contour line (shown by a broken line) that is symmetrical with respect to the left eye contour line O. Yes. The coordinates of the contour parameters L, R, T, and B of the left eye after the search are expressed as L (x _L , y _L ), R (x _R , y _R ), T (x _T , y _T ), B (x _B , y _B ), the coordinates of the contour parameters L, R, T, and B of the right eye that are initially set in this modification are L (25−x _R −2, y _R ), R (25−x _L +2, y). _L ), T (25−x _T , y _T +2), and B (25−x _B , y _B −2). By doing in this way, the initial setting of the contour parameters L, R, T, and B can be set so that the search is started from a position outside by two pixels with respect to the contour line that is symmetric. Since it is conceivable that the positions of the left and right eyes detected by the detection of the facial organ in step S130 may vary, the contour line O of the right eye may be set larger so that the variation can be absorbed. If the contour line O of the right eye is set larger, the contour parameters L, R, T, and B can be converged to optimum coordinates while moving inward, and the contour parameters L, R, It is possible to prevent the positional relationship between R, T, and B from being reversed.

3-5. Modification 5
In the embodiment described above, the contour detection of the left eye is performed first, but the contour of the eye with higher reliability is searched first, and based on the search result, the contour parameters L, It is desirable to initially set R, T, and B. For example, when the face is not facing the front, one eye is positioned closer to the digital still camera, and thus a large image is taken. In this case, it is possible to detect a contour with high reliability by detecting the contour of an eye that has been photographed larger than when detecting the contour of an eye that has been photographed smaller. Therefore, in the face organ detection in step S130, the comparison areas CA1 and CA2 (in this modification, the sizes of the comparison areas CA1 and CA2 are not fixed, but pattern matching is performed while changing) are increased. It is desirable to detect the contour of the eye first. Furthermore, in the face detection in step S110, the face orientation may also be detected, and the contour of the eye closer to the front in the face orientation may be detected first.

  FIG. 23 schematically shows how the eye with higher reliability is determined when the contour of the eye with higher reliability is searched first. In the figure, an example of face image data FD in a state where the detection of the facial organ (step S130) is completed is shown. In the example of the figure, the face is directed leftward, and the right eye comparison area CA2 is detected larger than the left eye comparison area CA1. In this case, it can be considered that the larger right eye comparison area CA2 has a sharper eye contour. Therefore, it is possible to detect the contour of the right eye with high reliability first, and to initialize the contour parameters L, R, T, and B of the left eye based on the result.

  In addition, although what performed the outline detection of both right-and-left eyes was illustrated above, it cannot be overemphasized that the outline detection method of this invention is applicable to detecting the outline of any one eye. The contour detection method of the present invention is not limited to the one executed on the computer, and may be executed by an image device such as a printer, a digital still camera, or a scanner. If the contour detection method of the present invention is performed by a printer, image processing corresponding to the contour can be executed during printing. If the contour detection method of the present invention is performed with a digital still camera, various shooting controls can be performed according to the contour.

It is a block diagram which shows the hardware constitutions of an outline detection apparatus. It is a block diagram which shows the software structure of an outline detection apparatus. It is a flowchart which shows the flow of an outline detection process. It is a figure which shows the mode of face detection. It is a figure which shows the mode of face organ detection. It is a figure which shows the face image data after a face organ detection. It is a figure which shows the mode of inclination correction. It is a flowchart which shows the flow of a scalar conversion process. It is a figure which shows the mode of sampling. It is a histogram obtained by sampling. It is a graph which shows the isoline of Mahalanobis square distance. It is a graph which shows the conversion characteristic by a conversion function. It is a graph which shows the relationship between Mahalanobis square distance and probability distribution. It is a figure which shows an example of Z map. It is a figure which shows the outline parameter in Z map. It is a figure explaining an evaluation value. It is a schematic diagram which shows the search procedure in a search process. It is a schematic diagram which shows the detailed procedure of a search process. It is a figure which shows the outline parameter in Z map of a right eye. It is a schematic diagram which shows the outline detection process concerning a modification. It is a graph which shows an example of the weight concerning a modification. It is a figure which shows the modification of the outline parameter in Z map of a right eye. It is a figure explaining a mode that the outline detection order of a right-and-left eye is determined.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Computer, 11 ... CPU, 12 ... RAM, 13 ... ROM, 14 ... HDD, 14a ... Program data, 14b ... Face template, 14c ... Eye template, 14d ... Mouth template, 14e ... Image data, 15 ... GIF, 16 ... VIF, 17 ... IIF, 18 ... bus, 20 ... printer, 40 ... display, 50a ... keyboard, 50b ... mouse, P1 ... OS, P2 ... contour detection application, P2a ... face detection unit, P2b ... face organ detection unit, P2c: contour detection unit, P2c1: inclination correction unit, P2c2: sampling unit, P2c3: scalar conversion unit, P2c4: conversion function setting unit, P2c5: parameter search unit, P2c6: evaluation value calculation unit, P3: printer driver.

Claims (8)

A contour detection method for detecting the contours of left and right eyes of a face included in image data,
Initializing a first approximate curve that approximates the contour of either the left or right eye,
Calculating an evaluation value for evaluating the closeness of the first approximate curve to the contour of the one eye, and searching for the first approximate curve for improving the closeness based on the evaluation value;
Initializing a second approximate curve to be approximated to the other eye based on the searched first approximate curve;
An evaluation value for evaluating the closeness of the second approximate curve to the contour of the other eye is calculated, and the second approximate curve for improving the closeness is searched based on the evaluation value. Contour detection method.   The contour detection method according to claim 1, wherein the second approximate curve is initially set so as to have a symmetrical shape with the first approximate curve after the search.   2. The contour detection method according to claim 1, wherein the second approximate curve is initially set to be a predetermined amount larger than the first approximate curve after the search.   In the first approximate curve and / or the second approximate curve, two points located at both outer sides in the horizontal direction among the three points arranged at different positions of the image data are respectively terminated, and the remaining one point is shared. The contour detection method according to claim 1, wherein the contour detection method is expressed by a pair of quadratic curves connected to each other at the inflection point.   2. The position and size of a rectangular area including the left and right eyes are detected in advance, and the first approximate curve and the second approximate curve are initially set in each rectangular area. The contour detection method according to claim 4.   6. The method according to claim 1, further comprising: predicting, in advance, a right and left eye whose contour can be accurately detected, and approximating the predicted contour of the eye with the first approximate curve. The contour detection method according to one item. A contour detection device for detecting a contour of a facial organ included in image data,
First approximating means for initially setting a first approximating curve that approximates the contour of one of the left and right eyes;
First search means for calculating an evaluation value for evaluating the closeness of the first approximate curve to the contour of the one eye and searching for the first approximate curve for improving the closeness based on the evaluation value; ,
Second approximating means for initially setting a second approximate curve to be approximated to the other eye based on the position or shape of the searched first approximate curve;
Second search means for calculating an evaluation value for evaluating the closeness of the second approximate curve to the contour of the other eye and searching for the second approximate curve for improving the closeness based on the evaluation value; A contour detection apparatus comprising: A computer-readable contour detection program for causing a computer to execute a function of detecting a contour of a facial organ included in image data,
A first approximation function that initially sets a first approximation curve that approximates the contour of either the left or right eye;
A first search function for calculating an evaluation value for evaluating the closeness of the first approximate curve to the contour of the one eye, and searching for the first approximate curve for improving the closeness based on the evaluation value; ,
A second approximation function for initially setting a second approximation curve to be approximated to the other eye based on the searched first approximation curve;
A second search function for calculating an evaluation value for evaluating the closeness of the second approximate curve to the contour of the other eye and searching for the second approximate curve for improving the closeness based on the evaluation value; A computer-readable contour detection program for causing a computer to execute.

Images