JP2010282339A - Image processor for correcting position of pupil in eye, image processing method, image processing program and printer - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a satisfactory direction of a line of sight in a figure image. <P>SOLUTION: An image processor is provided with: a face region detection part for detecting a face region from an image under consideration; an eye region detection part for detecting an eye region from a face image; an acquisition part for acquiring an eye model for reproducing an eye image while fluctuating the position of the pupil in the eye, based on a featured value; a featured value detection part for detecting the featured value when the eye image to be reproduced by the eye model and the eye region approximate as an approximate featured value; and a correction part for correcting the approximate featured value so that the ideal eye image in which the position of the pupil becomes ideal is reproduced by the eye model; and also for correcting the eye region by the ideal eye image in which the model is reproduced based on the corrected approximate featured value. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, an image processing program, and a printing apparatus that correct the position of a pupil in an eye.

  There are cases in which a person's eyes are not gazing at the camera in a portrait (in other words, not a so-called camera line of sight). Such a human photograph has a problem that it does not look good even if it is printed. Note that an imaging apparatus that starts an imaging operation based on the line-of-sight direction is disclosed. (See Patent Document 1).

JP 2007-180869 A

However, there has been a problem that the direction of the line of sight of a person in a human photograph that has already been taken cannot be corrected to improve the appearance.
The present invention has been made to solve the above-described problems, and an object of the present invention is to provide an image processing device, an image processing method, an image processing program, and a printing device that improve the line-of-sight direction in a human image. To do.

  In order to solve at least a part of the problems, the present invention adopts the following aspects. That is, the face area detection unit detects a face area from the attention image, and the eye area detection unit detects an eye area from the face image. Further, the setting unit acquires an eye model that reproduces the eye image while changing the position of the pupil in the eye based on the feature amount. The feature quantity detection unit detects a feature quantity when the eye image reproduced by the eye model approximates the eye area as an approximate feature quantity. Further, the correction unit corrects the approximate feature amount so that the eye model reproduces an ideal eye image in which the pupil position is ideal, and the model reproduces based on the corrected approximate feature amount. The eye area is corrected by the ideal eye image. Since the ideal eye image is corrected so that the position of the pupil is ideal, the direction of the line of sight depending on the position of the pupil can be corrected.

  As a preferred example of the eye model, the acquisition unit linearly combines the eigenvectors of each principal component by performing principal component analysis on the color values of each pixel of a plurality of sample images including eye images with different line-of-sight directions. The eye model that reproduces the eye image may be acquired by the above. According to the principal component analysis of the color values of each pixel of the plurality of sample images, the variation of the color value distribution according to the pupil position can be statistically analyzed, and the feature amount strongly correlated only with the pupil position Can be obtained. Therefore, it is possible to correct only the position of the pupil while maintaining other characteristics such as the color of the pupil.

  Further, it is desirable to detect a swing angle of a face included in the face area and correct the eye area only when the swing angle is within a predetermined range. By doing so, it is possible to prevent correction to an unnatural line of sight. For example, it is possible to prevent only the line of sight from being a camera line of sight even though the face is extremely swaying to the side.

  As an example of a guideline for correcting the line-of-sight direction, there is a correction in which the line-of-sight direction is perpendicular to the image plane of the target image. By doing in this way, it can correct | amend so that a gaze direction may face the front with respect to an image. As another example of the guideline for correcting the gaze direction, there is a correction in which the gaze direction is the front direction of the face included in the face area. For example, it can be corrected to look at the front in a scene where the user has not looked at the camera but has looked at the camera. Another example of the guideline for correcting the line-of-sight direction is correction that causes the line-of-sight direction to face the center of the image of interest. By doing in this way, in many cases, it can correct | amend so that the lens of the camera located in the center of the said attention image may be gazes.

  Note that the present invention can be realized in various modes, for example, an image processing method and apparatus, a feature position detection method and apparatus, a facial expression determination method and apparatus, and the functions of these methods or apparatuses. For example, a recording medium storing the image processing program, a data signal including the computer program and embodied in a carrier wave, and the like.

1 is an explanatory diagram schematically illustrating a configuration of a printer 100 as an image processing apparatus according to an embodiment of the present invention. It is a flowchart which shows the flow of the AAM setting process in an Example. It is explanatory drawing which shows an example of sample image SI. It is explanatory drawing which shows an example of the setting method of the feature point CP in the sample image SI. It is explanatory drawing which shows an example of the coordinate of the feature point CP set to sample image SI. It is explanatory drawing which shows an example of average shape s0. It is explanatory drawing which illustrated the relationship between shape vector s _i and shape parameter p _i, and face shape s. It is explanatory drawing which shows an example of the method of the warp W of the sample image SI. It is explanatory drawing which shows an example of average face image _A0 (x). It is explanatory drawing which shows an example of sample image SI for an eye texture model. It is a graph which shows the relationship between a texture parameter and an eye image. It is a flowchart which shows the flow of the face feature detection process in an Example. It is explanatory drawing which shows an example of the detection result of the face area FA in the attention image OI. It is a flowchart which shows the flow of the initial position setting process of the feature point CP in an Example. It is explanatory drawing which shows an example of the temporary setting position of the feature point CP by changing the value of a global parameter. It is explanatory drawing which shows an example of the average shape image I (W (x; p)). It is a flowchart which shows the flow of the feature point CP setting position correction process in an Example. It is explanatory drawing which shows an example of the result of CP setting position correction processing. It is a flowchart which shows the flow of an eye feature detection process (texture detection process). It is explanatory drawing which shows a mode that an eye texture image is extracted from an average shape image. It is a flowchart which shows the flow of a gaze determination / correction | amendment process. It is a figure which shows an angle table. It is a schematic diagram explaining a swing angle θ and a relative line-of-sight angle γ. It is a figure which shows a mode that the position of a pupil is correct | amended. It is a flowchart which shows the flow of a gaze determination / correction | amendment process concerning a modification. It is a schematic diagram explaining the ideal visual axis in a modification. It is a schematic diagram explaining the ideal visual axis in a modification.

  Hereinafter, a printer which is an aspect of an image processing apparatus according to the present invention will be described based on examples with reference to the drawings.

A. Example:
A1. Configuration of image processing device:
FIG. 1 is an explanatory diagram schematically showing a configuration of a printer 100 as an image processing apparatus in the present embodiment of the present invention. The printer 100 of this embodiment is an ink jet color printer that supports so-called direct printing, in which an image is printed based on image data acquired from a memory card MC or the like. The printer 100 includes a CPU 110 that controls each unit of the printer 100, an internal memory 120 configured by a ROM and a RAM, an operation unit 140 configured by buttons and a touch panel, a display unit 150 configured by a liquid crystal display, and printing. A mechanism 160 and a card interface (card I / F) 170 are provided. The printer 100 may further include an interface for performing data communication with other devices (for example, a digital still camera or a personal computer). Each component of the printer 100 is connected via a bus so that bidirectional communication is possible.

  The printing mechanism 160 performs printing based on the print data. The card interface 170 is an interface for exchanging data with the memory card MC inserted into the card slot 172. In this embodiment, an image file including image data is stored in the memory card MC.

  The internal memory 120 stores an image processing unit 200, a display processing unit 310, and a print processing unit 320. The image processing unit 200 is a computer program and is executed by the CPU 110 under a predetermined operating system to perform face feature detection processing, eye feature detection processing, and line-of-sight determination / correction processing. The face feature detection process is a process of detecting the position of a predetermined feature part (for example, the corner of the eye, the nose head, or the face line) in the face image. The face feature detection process will be described in detail later. The display processing unit 310 and the print processing unit 320 are also executed by the CPU 110 to realize the respective functions.

  The image processing unit 200 includes, as program modules, a setting unit 210, a feature position detection unit 220, a face area detection unit 230, a texture detection unit 240, a line-of-sight determination unit 250, and a line-of-sight correction unit 260. Yes. The feature position detection unit 220 and the texture detection unit 240 include generation units 222 and 242, calculation units 224 and 244, and correction units 226 and 246, respectively. The functions of these units will be described in detail in the description of face feature detection processing and line-of-sight determination / correction processing described later.

  The display processing unit 310 is a display driver that controls the display unit 150 to display processing menus, messages, images, and the like on the display unit 150. The print processing unit 320 is a computer program for generating print data from image data, controlling the printing mechanism 160, and printing an image based on the print data. The CPU 110 reads out and executes these programs (the image processing unit 200, the display processing unit 310, and the print processing unit 320) from the internal memory 120, thereby realizing the functions of these units.

  The internal memory 120 stores AAM information AMI. The AAM information AMI is information set in advance by an AAM setting process described later, and is acquired and used in a face feature detection process and an eye feature detection process described later. The contents of the AAM information AMI will be described in detail in the description of AAM setting processing described later.

A2. AAM setting process:
FIG. 2 is a flowchart showing the flow of AAM setting processing in this embodiment. The AAM setting process is a process of setting a face shape model, a face texture model, and an eye texture model used for modeling an image called an AAM (Active Appearance Model). In the present embodiment, the AAM setting process is performed by an operator using a computer. The computer has a CPU, a RAM, a ROM, an HDD, a display, an input device, and the like, which are connected by a bus. The CPU reads out a program recorded in the HDD and executes arithmetic processing according to the program, thereby causing the computer to execute each process described later. The eye texture model corresponds to an “eye model” in the claims.

  First, the operator prepares a plurality of images including a person's face as a sample image SI (step S110). FIG. 3 is an explanatory diagram showing an example of the sample image SI. As shown in FIG. 3, the sample image SI includes face images that are different from each other with respect to various attributes such as personality, race, gender, facial expression, and orientation (front, upward, downward, rightward, leftward, etc.). Prepared. According to these sample images SI, any face image can be accurately modeled by AAM, and accurate face feature detection processing (described later) for any face image can be executed. The sample image SI is also called a learning image. Further, in this specification, the left-right direction indicates the left-right direction based on the subject person.

  Hereinafter, processing performed on the sample images SI belonging to the sample group G1 will be described as an example. A feature point CP is set to the face image included in each sample image SI belonging to the sample group G1 (step S120). FIG. 4 is an explanatory diagram showing an example of a method for setting the feature point CP in the sample image SI. The feature point CP is a point indicating the position of a predetermined feature part in the face image. In this embodiment, as predetermined feature parts, a predetermined position on the eyebrows in a person's face (for example, an end point or a four-divided point, the same applies hereinafter), a predetermined position on the eye contour, and a predetermined position on the nose and nose contours 68 parts such as a predetermined position on the contour of the upper and lower lips and a predetermined position on the contour of the face (face line) are set. That is, in the present embodiment, a predetermined position in the facial organs (eyebrows, eyes, nose, mouth) and facial contour that are commonly included in the human face are set as the characteristic parts. As shown in FIG. 4, the feature points CP are set (arranged) at positions representing 68 feature parts designated by the operator in each sample image SI. Since each feature point CP set in this way corresponds to each feature part, the arrangement of the feature points CP in the face image can be expressed as specifying the shape of the face.

  The position of the feature point CP in the sample image SI is specified by coordinates. FIG. 5 is an explanatory diagram showing an example of the coordinates of the feature point CP set in the sample image SI. In FIG. 5, SI (g) (g = 1, 2, 3...) Indicates each sample image SI, and CP (k) (k = 0, 1,..., 67) indicates each feature. Point CP is shown. CP (k) -X represents the X coordinate of the feature point CP (k), and CP (k) -Y represents the Y coordinate of the feature point CP (k). The coordinates of the feature point CP include predetermined reference points (for example, in the sample image SI normalized with respect to the size of the face, the inclination of the face (inclination in the image plane), and the position of the face in the X direction and the Y direction) The coordinates with the origin at the lower left point of the image are used. Further, in the present embodiment, a case where a plurality of human face images are included in one sample image SI is allowed (for example, two sample face images are included in the sample image SI (2)). Each person in one sample image SI is specified by a person ID. Each sample image SI, the coordinates of each feature point CP, and the person ID are input to the computer.

  Subsequently, the operator sets the AAM face shape model (step S130). Specifically, the computer performs principal component analysis on a coordinate vector (see FIG. 5) composed of the coordinates (X coordinate and Y coordinate) of 68 feature points CP in each sample image SI, and the computer further uses the feature points. The face shape s specified by the position of the CP is modeled by the following equation (1). The face shape model is also referred to as a feature point CP arrangement model.

In the formula (1), s ₀ is an average shape. FIG. 6 is an explanatory diagram showing an example of the average shape s ₀ . As shown in FIGS. 6A and 6B, the average shape s ₀ is a model representing the average face shape specified by the average position (average coordinate) for each feature point CP of the sample image SI. is there. In the present embodiment, in the average shape s ₀ , a region surrounded by a straight line connecting feature points CP (face lines, eyebrows, feature points CP corresponding to the eyebrows, see FIG. 4) located on the outer periphery (FIG. 6 ( b) is indicated by hatching) and is referred to as “average shape region BSA”. As shown in FIG. 6A, the computer sets a plurality of triangular areas TA having the feature points CP as vertices so as to divide the average shape area BSA into a mesh shape with respect to the average shape s ₀ .

In the equation (1) representing the face shape model, s _i is a shape vector, and p _i is a shape parameter representing the weight of the shape vector s _i . The shape vector s _i is a vector representing the characteristics of the face shape s and is an eigenvector corresponding to the i-th principal component obtained by principal component analysis. As shown in the equation (1), in the face shape model in this embodiment, the face shape s representing the arrangement of the feature points CP is the sum of the average shape s ₀ and the linear combination of n shape vectors s _i. Modeled. As shown in the equation (1a), it is possible to reproduce the face shape s in any image by appropriately setting the shape parameter p _i in the face shape model. Further, as shown in the equation (1b), by appropriately setting the shape parameter p _i in the face shape model, it is possible to reversely convert from any shape s to the average shape s ₀ .

FIG. 7 is an explanatory diagram illustrating the relationship between the shape vector s _{i, the} shape parameter p _i, and the face shape s. As shown in FIG. 7 (a), in order to identify the face shape s, the computer selects the number n (the number) selected from the eigenvector corresponding to the principal component having the larger contribution rate in the descending order of the cumulative contribution rate. eigenvectors of that.) and 10 pieces in the embodiment, used as the shape vectors s _i. Each of the shape vectors s _i corresponds to the movement direction / movement amount of each feature point CP, as indicated by the arrows in FIG. In this embodiment, most first shape vector s ₁ corresponding to the first principal component having a large contribution rate is a vector that is approximately correlated with the left and right appearance of a face, by the magnitude of the shape parameter p _1, 7 As shown in (b), the horizontal face direction of the face shape s changes. Second shape vector s ₂ corresponding to the large second principal component of the contribution rate to the second is a vector that is approximately correlated with the vertical appearance of a face, by the magnitude of the shape parameter p _2, FIG. 7 (c) As shown in FIG. 4, the vertical face direction of the face shape s changes. The third shape vector s ₃ corresponding to the third principal component having the third largest contribution ratio is a vector that is substantially correlated with the aspect ratio of the face shape, and the fourth principal component having the fourth largest contribution ratio. The fourth shape vector s ₄ corresponding to is a vector that is substantially correlated with the degree of mouth opening. As described above, the value of the shape parameter represents the feature of the face image such as facial expression and face orientation.

The average shape s ₀ and the shape vector s _i set by the computer in the shape model setting step (step S130) are stored in the internal memory 120 of the printer 100 as AAM information AMI (FIG. 1). Subsequently, the computer sets an AAM face texture model (step S140). Specifically, first, for each sample image SI, image conversion (hereinafter referred to as “warp”) is performed so that the setting position of the feature point CP in the sample image SI is equal to the setting position of the feature point CP in the average shape s ₀ . Also called “W”).

FIG. 8 is an explanatory diagram showing an example of a method of warping W of the sample image SI. In each sample image SI, similarly to the average shape s ₀ , the computer sets a plurality of triangular regions TA that divide the region surrounded by the feature points CP located on the outer periphery into a mesh shape. The warp W is a set of affine transformations for each of the plurality of triangular areas TA. That is, in the warp W, an image of a certain triangular area TA in the sample image SI is affine transformed into an image of the corresponding triangular area TA in the average shape s ₀ . The computer generates a sample image SI (hereinafter referred to as “sample image SIw”) in which the setting position of the feature point CP is equal to the setting position of the feature point CP in the average shape s ₀ by the warp W.

  Each sample image SIw has a rectangular frame containing an average shape area BSA (shown with hatching in FIG. 8) as an outer periphery, and an area other than the average shape area BSA (hereinafter also referred to as “mask area MA”). Generated as a masked image. An image area in which the average shape area BSA and the mask area MA are combined is referred to as a reference area BA. In addition, the computer normalizes each sample image SIw as an image having a size of, for example, 56 pixels × 56 pixels.

Next, the computer performs a principal component analysis on a color value vector composed of color values (RGB) in each pixel group x of each sample image SIw, and performs facial texture (also referred to as “appearance”) A (x) Is modeled by the following equation (2). The pixel group x is a set of pixels located in the average shape area BSA.

In the formula (2), A ₀ (x) is an average face image. FIG. 9 is an explanatory diagram showing an example of the average face image A ₀ (x). The average face image A ₀ (x) is an image representing the average face of the sample image SIw (see FIG. 8) after the warp W. That is, the average face image A ₀ (x) is an image calculated by taking the average of the pixel values (color values) of the pixel group x in the average shape area BSA of the sample image SIw. Therefore, the average face image A ₀ (x) is a model representing the average face texture (appearance) in the average face shape. Note that the average face image A ₀ (x) is composed of the average shape area BSA and the mask area MA, similarly to the sample image SIw, and is calculated as an image having a size of 56 pixels × 56 pixels, for example.

In the equation (2) representing the face texture model, a _k (x) is a texture vector, and λ _k is a texture parameter representing the weight of the texture vector a _k (x). The texture vector a _k (x) is a vector representing the characteristics of the facial texture A (x), and is specifically an eigenvector corresponding to the k-th principal component obtained by principal component analysis. That is, the eigenvectors of the number m (four in this embodiment) set based on the cumulative contribution rate are adopted as the texture vector a _k (x) in order from the eigenvector corresponding to the principal component having the larger contribution rate. Is done. In the present embodiment, the first texture vector A ₁ (x) corresponding to the first principal component having the largest contribution rate is a vector that is substantially correlated with a change in face color (also regarded as a gender difference).

As shown in the equation (2), in the face texture model in the present embodiment, the face texture A (x) representing the appearance of the face includes the average face image A ₀ (x) and m texture vectors a _k ( x) is modeled as the sum of the linear combination. As shown in the equation (2a), it is possible to reproduce the facial texture A (x) in any image by appropriately setting the texture parameter λ _k as a weighting coefficient in the face texture model. Conversely, as shown in the equation (2b), by appropriately setting the texture parameter λ _k in the face texture model, the facial texture A (x) in every image is changed to the average face image A ₀ (x). Inverse transformation can be performed.

The average face image A ₀ (x) and the texture vector a _k (x) set in the texture model setting step (step S140 in FIG. 2) are stored in the internal memory 120 of the printer 100 as AAM information AMI (FIG. 1). Stored. The computer records the AAM information AMI on any recording medium, and when the printer 100 is manufactured, the internal memory 120 that records the AAM information AMI read from the recording medium is incorporated in the printer 100 or rarely incorporated in the printer 100. The AAM information AMI read from the recording medium is recorded in the internal memory 120.

By the processing so far, a face shape model for modeling the face shape and a face texture model for modeling the face texture are set. By combining the set face shape model and the face texture model, that is, conversion from the average shape s ₀ to the shape s for the synthesized texture A (x) (inverse conversion of the warp W shown in FIG. 8) By performing the above, it is possible to reproduce the shape and texture of any face image. When the face shape model and the face texture model are set, an eye texture model as an eye model is set by the same method.

  The eye texture model is also set according to the AAM setting process (FIG. 2). However, since the eyes can be considered to be substantially flat, the setting of the shape model for the eyes (step S130) is not executed. In setting the eye texture model, the operator prepares a plurality of images including human eye images as sample images SI (step S110). FIG. 10 is an explanatory diagram showing an example of the sample image SI. As shown in FIG. 4, the sample image SI includes the position of the pupil in the eye (center, left direction, right direction, up direction, down direction), the color of the pupil, and red eyes (when the flash light is reflected by the retina). It is prepared to include eye images that are different from each other with respect to various attributes. These attributes are associated with each sample image SI. In order to improve the robustness of the eye texture model, a sample image SI when wearing eyeglasses may be included. According to these sample images SI, all eye images can be accurately modeled by AAM, and accurate eye feature detection processing (described later) for all eye images can be executed. Since the right eye and the left eye can be modeled symmetrically with each other, only the right eye image is modeled in this embodiment.

Since it is only necessary to model the pupil color including the position of the pupil and the case of the red eye, it is desirable that the variation in the eye shape (eye contour) is small in the sample image SI. The shape and size of the eyes can be detected by a face shape model that sets feature points CP for the eye contour. The sample image SI for the eye image may be prepared by extracting only the image region corresponding to the eye from the sample image SIw for the face image. Since the sample image SIw is obtained by modifying each sample image SI so that the setting position of the feature point CP is equal to the setting position of the feature point CP in the average shape s ₀ , the position, shape, and size of the eye image Is constant. Accordingly, by extracting an image region having a certain position, shape and size from the sample image SIw, it is possible to obtain a sample image SI for an eye image whose shape and size are averaged. In this embodiment, it is assumed that the right eye is modeled, and the left eye sample image SI is inverted in advance.

When the sample image SI for the eye image can be prepared, the eye texture model is set by performing the AAM setting process similar to the above-described face texture model. Similar to the face texture model, the texture of the eye image can be reproduced by the equation (2). In the eye texture model, A (x) in the equation (2) indicates an arbitrary eye image, A ₀ (x) indicates an average eye image, and a _k (x) and λ _k are face textures. The texture vector and texture parameters are shown as well as the model. In the present embodiment, the first texture vector a ₁ corresponding to the first principal component having the largest contribution rate is a vector that is substantially correlated with the horizontal position of the pupil, and by increasing or decreasing the texture parameter λ ₁ , The pupil's horizontal position in the eye changes. The second texture vector λ ₂ corresponding to the second principal component having the second largest contribution rate is a vector that is substantially correlated with whether or not it is red-eye. By increasing or decreasing the texture parameter λ ₂ , the redness-likeness is increased. Change.

FIG. 11A shows the correlation between the value of the texture parameter λ ₁ and the (horizontal) position of the pupil, and FIG. 11B shows the correlation between the value of the texture parameter λ ₁ and λ ₂ and whether or not it is red-eye. Show. In FIG. 11A, the horizontal axis indicates the value (−1 to ₁ ) of the texture parameter λ ₁ , and the vertical axis indicates the horizontal position of the pupil in the eye. As shown in the figure, the larger the value of the texture parameter λ ₁ , the closer the pupil position is to the inside (nose side: left side for the right eye), and when the texture parameter λ ₁ value is 0, the pupil position is It is the center that is neither inward nor outward (the position of the pupil when the line of sight is in the front direction). The smaller the value of the texture parameter λ ₁ is, the closer the pupil is to the outside (close to the contour: to the right for the right eye).

In FIG. 11B, the horizontal axis and the vertical axis indicate the values (−1 to ₁ ) of the texture parameters λ ₁ and λ ₂ . In FIG. 11B, the texture parameters λ ₁ and λ ₂ are plotted, and those that are red-eye are indicated by ◯, and those that are not red-eye are indicated by ●. As shown in FIG. 11B, the larger the texture parameter λ ₂ is, the more red eyes are distributed. Therefore, whether or not it is red-eye has a positive correlation with the texture parameter λ ₂ . On the other hand, with respect to the texture parameter λ ₁ , red-eye and non-red-eye are distributed almost evenly and have no correlation. Ish larger the eye the greater the value of the texture parameter lambda _2, the value of the texture parameter lambda ₂ is not Rashiku smaller red (the normal). Note that the redness is mainly the intensity of the red eye color (mainly the intensity of redness or goldness) of the pupil part at the center of the pupil. When the texture parameter λ ₂ is increased from −1, the pupil part Changes from the original color of the pupil to brown (λ ₂ ≈0) → red to gold. In addition, the texture parameter λ ₂ statistically reflects the texture characteristics of the red-eye sample image SI. Therefore, by changing the texture parameter λ ₂ , characteristics other than the simple color of the pupil part can be obtained. Change. For example, it is predicted that the gradation change of the color of the pupil portion that changes due to the red eye also changes.

In the present embodiment, when the texture parameter λ ₂ is 0 by equalizing the number of red-eye sample images SI and non-red-eye sample images SI, it cannot be said that the pupil color is red-eye or normal. It will show a dark brown color. As for the color of the iris part outside the pupil of the pupil, there is little color variation depending on whether it is red-eye or not, and the variation of the component corresponds to the third and fourth principal components with the third largest contribution ratio The third and fourth texture vectors a _{3 and} a ₄ are substantially correlated. In this embodiment, the fourth principal component (m = 4) is used, but the fifth and subsequent principal components are components corresponding to the size of the pupil, the vertical position of the pupil, and the like. Note that the hue directions of the color values that vary depending on the _{third and} fourth texture vectors a _{3 and} a ₄ are different from each other. For example, the color value of the third texture vector a ₃ varies along the R axis, and the color value of the fourth texture vector a ₄ varies along the B axis.

The average eye image A ₀ (x) and the texture vector a _k (x) set in the texture model setting step (step S140 in FIG. 2) for the eye texture model are the printer 100 as AAM information AMI (FIG. 1). Stored in the internal memory 120. The computer records the AAM information AMI on any recording medium, and when the printer 100 is manufactured, the internal memory 120 that records the AAM information AMI read from the recording medium is incorporated in the printer 100 or rarely incorporated in the printer 100. The AAM information AMI read from the recording medium is recorded in the internal memory 120. In executing the facial feature detection process and the eye feature detection process, the CPU that acquires the AAM information AMI as the eye model from the internal memory 120 in the printer 100 corresponds to the “acquisition unit” of the present invention.

A3. Face feature detection processing:
FIG. 12 is a flowchart illustrating a flow of face feature detection processing in the embodiment. The face feature detection process in the present embodiment is a feature point CP setting position correction process (step S250) that uses AAM (face model (face shape model)) to determine the arrangement of the feature points CP in the face image included in the target image. ) To detect the position of the characteristic part in the face image. As described above, in the present embodiment, in the AAM setting process (FIG. 2), a total of 68 predetermined positions in the human facial organs (eyebrows, eyes, nose, mouth) and facial contour are set as the characteristic parts. (See FIG. 4). Therefore, in the face feature detection process of the present embodiment, the arrangement of 68 feature points CP indicating predetermined positions in the human face organ and face contour is determined.

When the arrangement of the feature points CP in the face image is determined by the face feature detection process, the value of the shape parameter p _i for the face image is specified. Accordingly, the processing result of the face feature detection process can be used for face orientation determination for detecting a face image in a specific direction (for example, rightward or downward), face deformation for deforming the face shape, face shading correction, and the like. .

  First, the image processing unit 200 (FIG. 1) acquires image data representing an attention image that is a target of face feature detection processing (step S210). In the printer 100 of the present embodiment, when the memory card MC is inserted into the card slot 172, thumbnail images of image files stored in the memory card MC are displayed on the display unit 150. One or more images to be processed are selected by the user via the operation unit 140. The image processing unit 200 acquires an image file including image data corresponding to one or more selected images from the memory card MC and stores it in a predetermined area of the internal memory 120. Note that the acquired image data is referred to as attention image data, and an image represented by the attention image data is referred to as attention image OI.

  The face area detection unit 230 (FIG. 1) detects an image area including at least a part of the face image included in the target image OI as the face area FA (step S230). The detection of the face area FA can be performed using a known face detection method. Known face detection methods include, for example, pattern matching methods, skin color region extraction methods, learning using sample images (for example, learning using neural networks, learning using boosting, and support vector machines). For example, a method using learning data set by learning).

  FIG. 13 is an explanatory diagram illustrating an example of the detection result of the face area FA in the target image OI. FIG. 13 shows the face area FA detected in the target image OI. In the present embodiment, a face detection method is used in which a rectangular area including the vertical direction of the face from the forehead to the chin and the horizontal direction from the outside of both ears is detected as the face area FA.

  The setting unit 210 (FIG. 1) sets the initial position of the feature point CP in the target image OI (Step S240). FIG. 14 is a flowchart showing a flow of initial position setting processing of the feature point CP in the present embodiment. In the present embodiment, the setting unit 210 changes the values of global parameters representing the size, inclination, and position (vertical position and horizontal position) of the face image with respect to the face area FA to change the feature point CP. The temporary setting position on the attention image OI is set (step S310).

FIG. 15 is an explanatory diagram showing an example of a temporary setting position of the feature point CP by changing the value of the global parameter. FIGS. 15A and 15B show a mesh formed by connecting feature points CP and feature points CP in the target image OI. As shown in the center of FIGS. 15 (a) and 15 (b), the setting unit 210 temporarily sets the feature points CP (hereinafter referred to as the feature points CP) such that the average shape s ₀ is formed in the center of the face area FA. (Also referred to as “reference temporary setting position”).

  The setting unit 210 also sets a plurality of temporary setting positions obtained by changing various global parameter values with respect to the reference temporary setting position. Changing the global parameters (size, inclination, vertical position and horizontal position) means that the mesh formed by the feature point CP in the target image OI is enlarged / reduced, the inclination is changed, and the parallel movement is performed. Equivalent to. Accordingly, as shown in FIG. 15A, the setting unit 210 forms a temporary setting position (below the reference temporary setting position diagram and a temporary setting position that forms a mesh obtained by enlarging or reducing the mesh at the reference temporary setting position at a predetermined magnification. Or a temporary setting position (shown on the right and left in the drawing of the reference temporary setting position) that forms a mesh whose inclination is changed clockwise or counterclockwise by a predetermined angle. Further, the setting unit 210 forms a temporary setting position (upper left of the figure of the reference temporary setting position, which forms a mesh obtained by performing a combination of enlargement / reduction and change in inclination with respect to the mesh of the reference temporary setting position. Also set in the lower left, upper right, and lower right).

  Further, as shown in FIG. 15B, the setting unit 210 forms a temporary setting position (a reference temporary setting position diagram) that forms a mesh that is translated up or down by a predetermined amount from the mesh at the reference temporary setting position. And a temporary setting position (shown on the left and right in the drawing of the reference temporary setting position) that forms a mesh moved in parallel to the left or right. Further, the setting unit 210 forms a temporary setting position (upper left and lower left in the figure of the reference temporary setting position) that forms a mesh obtained by performing a combination of vertical and horizontal parallel movements on the mesh of the reference temporary setting position. , Shown in the upper right and lower right).

  The setting unit 210 also has temporary setting positions at which the parallel movement shown in FIG. 15B is performed on each of the eight temporary setting positions other than the reference temporary setting position shown in FIG. Set. Therefore, in the present embodiment, the four global parameters (size, inclination, vertical position, horizontal position) are set in 80 ways (= 3 × 3 × 3) that are set as known three-level values. A total of 81 temporary setting positions of (3-1) temporary setting positions and reference temporary setting positions are set.

The generation unit 222 (FIG. 1) generates an average shape image I (W (x; p)) corresponding to each set temporary setting position (step S320). FIG. 16 is an explanatory diagram showing an example of the average shape image I (W (x; p)). The average shape image I (W (x; p)) is calculated by conversion such that the arrangement of the feature points CP in the input image is equal to the arrangement of the feature points CP in the average shape s ₀ .

The transformation for calculating the average shape image I (W (x; p)) is a warp W that is a set of affine transformations for each triangular area TA, similarly to the transformation for calculating the sample image SIw (see FIG. 8). Is done. Specifically, the average shape area BSA (area surrounded by the feature points CP located on the outer periphery) is specified by the feature points CP (see FIG. 15) arranged in the attention image OI, and the average shape area in the attention image OI is specified. An average shape image I (W (x; p)) is calculated by performing affine transformation for each triangular area TA on the BSA. In this embodiment, the average shape image I (W (x; p) ) is composed of an average face image A ₀ (x) and an average shape area BSA and a mask area MA, average face image A ₀ (x) And is calculated as an image of the same size.

As described above, the pixel group x is a set of pixels located in the average shape region BSA in the average shape s ₀ . The pixel group in the image (average shape area BSA of the target image OI) before execution of the warp W corresponding to the pixel group x in the image after execution of the warp W (face image having the average shape s ₀ ) is expressed as W (x; p). To express. Since the average shape image is an image composed of the color values in each of the pixel groups W (x; p) in the average shape region BSA of the target image OI, it is expressed as I (W (x; p)). FIG. 16 shows nine average shape images I (W (x; p)) corresponding to the nine temporarily set positions shown in FIG.

The calculation unit 224 (FIG. 1) calculates a difference image Ie between the average shape image I (W (x; p)) corresponding to each temporarily set position and the average face image A ₀ (x) (step S330). . Since 81 types of temporary setting positions of the feature points CP are set, the calculation unit 224 (FIG. 1) calculates 81 difference images Ie.

The setting unit 210 calculates a norm from the pixel values of each difference image Ie, and sets a temporary installation position corresponding to the difference image Ie having the smallest norm value (hereinafter also referred to as “norm minimum temporary setting position”) as the target image OI. Is set as the initial position of the feature point CP at (step S340). The pixel value for calculating the norm may be a color value or a luminance value. Thus, the feature point CP initial position setting process is completed. Note that the smaller the norm value, the smaller the difference between the average shape image I (W (x; p)) and the average face image A ₀ (x) (both approximate).

  When the feature point CP initial position setting process is completed, the feature position detector 220 (FIG. 1) corrects the set position of the feature point CP in the target image OI (step S250). FIG. 17 is a flowchart showing the flow of the feature point CP setting position correction process in the present embodiment.

  The generation unit 222 (FIG. 1) calculates an average shape image I (W (x; p)) from the attention image OI (step S410). The calculation method of the average shape image I (W (x; p)) is the same as that in step S320 in the feature point CP initial position setting process.

The feature position detection unit 220 calculates a difference image Ie between the average shape image I (W (x; p)) and the average face image A ₀ (x) (step S420). The feature position detection unit 220 determines whether or not the feature point CP setting position correction process has converged based on the difference image Ie (step S430). The feature position detection unit 220 calculates the norm of the difference image Ie, determines that it has converged if the norm value is smaller than a preset threshold value, and still converges if the norm value is greater than or equal to the threshold value. Judge that there is no. The feature position detection unit 220 determines that the calculated norm value of the difference image Ie has converged when the value is smaller than the value calculated in the previous step S430, and still determines that the value is equal to or greater than the previous value. It is good also as what determines with not having converged. Alternatively, the feature position detection unit 220 may perform the convergence determination by combining the determination based on the threshold and the determination based on comparison with the previous value. For example, the feature position detection unit 220 determines that the calculated norm value has converged only when the calculated norm value is smaller than the threshold value and smaller than the previous value, and otherwise determines that it has not yet converged. It may be a thing.

If it is determined in step S430 that the convergence is not yet completed, the correction unit 226 (FIG. 1) calculates the parameter update amount ΔP (step S450). The parameter update amount ΔP includes four global parameters (total size, inclination, X direction position, Y direction position), and n shape parameters p _i (i = 1 to n) as feature amounts. .) Means the change amount of the value. Immediately after setting the feature point CP to the initial position, the global parameter is set to the value determined in the feature point CP initial position setting process (FIG. 14). Further, difference between the initial position of the characteristic point CP and sets the position of the characteristic points CP of the average shape s ₀ in this case, the overall size, the tilt, because it is limited to the difference in position, shape parameters in the shape model p The value of _i and the texture parameter λ _k in the texture model are all zero.

  The parameter update amount ΔP is calculated by the following equation (3). That is, the parameter update amount ΔP is a product of the update matrix R and the difference image Ie.

The update matrix R in the equation (3) is a matrix of M rows and N columns set in advance by learning in order to calculate the parameter update amount ΔP based on the difference image Ie.
It is stored in the internal memory 120 as AAM information AMI (FIG. 1). In this embodiment, the number of rows M in the update matrix R is equal to the sum ((4 + n)) of the number of global parameters (4) and the number of shape parameters p _i (n), and the number of columns N is , average face image a ₀ average number of pixels in the shape area BSA of (x) - is equal to (56 pixels × 56 pixels the number of pixels in the mask area MA). The update matrix R is calculated by the following formulas (4) and (5).

The correction unit 226 (FIG. 1) updates the parameters (four global parameters and n shape parameters p _i ) based on the calculated parameter update amount ΔP (step S460). Thereby, the setting position of the feature point CP in the target image OI is corrected. The correcting unit 226 corrects so that the norm of the difference image Ie is reduced. After the parameter update, the average shape image I (W (x; p)) is calculated again from the target image OI in which the installation position of the feature point CP is corrected (step S410), and the difference image Ie is calculated (step S420), a convergence determination based on the difference image Ie (step S430) is performed. The average shape image I (W (x; p)) changes according to the update (change) of the shape parameter p _i and corresponds to a “face region” in the claims. If it is determined that the convergence has not occurred in the convergence determination again, the parameter update amount ΔP based on the difference image Ie is calculated (step S450), and the setting position correction of the feature point CP by the parameter update (step S460). ) Is performed.

When the processing from step S410 to step S460 in FIG. 17 is repeatedly performed, the position of the feature point CP corresponding to each feature part in the target image OI and the position of the actual feature part approach as a whole and converge at a certain time. In the determination (step S430), it is determined that convergence has occurred. If it is determined in the convergence determination that convergence has been achieved, the face feature detection process is completed (step S470). The setting position of the feature point CP specified by the global parameter and the shape parameter p _i set at this time is specified as the setting position of the feature point CP in the final target image OI.

FIG. 18 is an explanatory diagram illustrating an example of a result of the feature point CP setting position correction process. FIG. 18 shows the setting positions of the feature points CP finally specified in the target image OI. Since the position of the feature point CP specifies the position of the facial feature part (predetermined position in the facial organs (eyebrows, eyes, nose, mouth) and facial contour) included in the attention image OI. It is possible to detect the shape and position of the human face organ and the face contour shape in the image OI. The position of the feature point CP finally specified in the target image OI, the global parameter, and the shape parameter p _i are stored in the internal memory 120.

A4. Eye feature detection processing:
FIG. 19 is a flowchart showing the flow of eye feature detection processing (texture detection processing). The eye feature detection process includes a process in which the texture detection unit 240 corrects the texture parameter λ _k using the eye texture model. In the face feature detection process, the shape and position of the eyes are specified. In the eye feature detection process, the texture of the eyes is specified. The texture detection unit 240 that executes the texture detection process corresponds to a “feature amount detection unit” in the claims.

The generation unit 242 (FIG. 1) calculates an average shape image I (W (x; p)) from the attention image OI (step S510). The calculation method of the average shape image I (W (x; p)) is the same as that in step S320 in the feature point CP initial position setting process and step S410 in the feature point CP setting position correction process. position correction processing completion time using the values of the global parameters and the shape parameters p _i of the average shape image I (W (x; p) ) is calculated. The average shape image I (W (x; p)) calculated here is the same as the average shape image I (W (x; p)) when the convergence determination is made in step S430. Next, the generating unit 242 (FIG. 1) detects the right eye region from the average shape image I (W (x; p)), and extracts the image of the eye region as the average shape eye image AI (x). (Step S515). The generating unit 242 that detects the eye region of the right eye corresponds to an “eye region detecting unit” in the claims.

FIG. 20 shows how the average shape eye image AI (x) is extracted from the average shape image I (W (x; p)). As described above, the eye position and shape in the average shape image I (W (x; p)) are the same as the eye position and shape in the average face image A ₀ (x). Global parameters and shape parameters p _i as averaging the facial image of the target image OI is from being detected by the face feature detection process. In this embodiment, coordinate information specifying the average eye areas Q1 and Q2 indicating the left and right eye areas in the average face image A ₀ (x) is registered in the internal memory 120, and the coordinate information is read out to obtain the average shape. An average shape eye image AI (x) is extracted by extracting pixels corresponding to the average eye region Q1 from the image I (W (x; p)). Although the overall position and shape of the eyes are averaged, the positions and colors of the pupils in the average shape eye image AI (x) are not averaged, and the individuality of the eyes included in the target image OI remains. Will be doing.

Next, an eye texture image J (T (x; λ)) in which the color value of each pixel constituting the average shape eye image AI (x) is changed by an eye texture model (the following equation (6)) is calculated. (Step S517).

The right side of the equation (6) is obtained by substituting the average shape eye image AI (x) for the texture A (x) on the right side of the equation (2b), and the texture parameter λ _k is appropriately set. For example, as indicated by the left side of Equation (2b), the right side is equal to the average eye image A ₀ (x). Here, an image obtained by the right side of the above equation (6) is represented as an eye texture image J (T (x; λ)). In the first stage, the value of the texture parameter λ _k is set to an appropriate value (for example, all 0).

Next, the calculation unit 244 calculates a difference image Ie between the eye texture image J (T (x; λ)) and the average eye image A ₀ (x) (step S520). The texture detection unit 240 determines whether the texture detection process has converged based on the difference image Ie (step S530). That is, it is determined whether or not the eye texture image J (T (x; λ)) and the average eye image A ₀ (x) are sufficiently approximated. The texture detection unit 240 calculates the norm of the difference image Ie, determines that it has converged if the norm value is smaller than a preset threshold value, and has not yet converged if the norm value is greater than or equal to the threshold value. Is determined. Note that the texture detection unit 240 determines that the calculated norm value of the difference image Ie has converged if it is smaller than the value calculated in the previous step S530, and still converges if it is greater than or equal to the previous value. It is good also as what determines with not doing. The convergence determination can be performed by the same method as in step S430 in the feature point CP setting position correction process.

If it is determined in step S530 that the convergence has not yet been completed, the correction unit 246 (FIG. 1) calculates the parameter update amount Δλ (step S550). The parameter update amount ΔΛ means the amount of change in the value of m (m = 4) texture parameters λ _k (k = 1 to an integer of m) that are feature amounts. The parameter update amount ΔΛ is calculated by the following equation (7). That is, the parameter update amount ΔΛ is a product of the update matrix U and the difference image Ie.

The update matrix U in Equation (7) is a matrix of M rows and N columns set in advance by learning in order to calculate the parameter update amount ΔΛ based on the difference image Ie.
It is stored in the internal memory 120 as AAM information AMI (FIG. 1). In this embodiment, the number M of rows of the update matrix U is equal to the texture parameter λ _k (m), and the number of columns N is equal to the number of pixels in the average eye areas Q1 and Q2. The update matrix U is calculated by the following formulas (8) and (9).

The correction unit 246 (FIG. 1) updates the texture parameter λ _k based on the calculated parameter update amount ΔΛ (step S560). Thereby, the texture A (x) of the eye is corrected. That is, the correction unit 246 corrects so that the norm of the difference image Ie is reduced. After the parameter update, the eye texture image J (T (x; λ)) is calculated again based on the corrected texture parameter λ _k (step S517), the difference image Ie is calculated (step S520), and the difference image. A convergence determination (step S530) based on Ie is performed. If it is determined that the convergence has not been made in the convergence determination again, the parameter update amount ΔΛ based on the difference image Ie is calculated (step S550), and the texture is corrected by updating the parameter (step S560). .

When the processing from steps S520 to S560 in FIG. 19 is repeatedly performed, the color value of each pixel of the texture A (x) of the eye when the corrected texture parameter λ _k is applied to the equation (2a) is obtained. Then, it approaches the color value of the actual eye image as a whole, and it is determined at a certain point in time that the convergence is determined in the convergence determination (step S530). When it is determined in the convergence determination that the right eye has been converged, the right eye texture detection process is completed, and the eye texture A (x) specified by the value of the texture parameter λ _k set at this time is the final right eye appearance. Identified as Next, it is determined whether or not the texture detection process for both eyes has been completed (step S570).

If not completed, the generation unit 242 extracts pixels corresponding to the average eye area Q2 from the average shape image I (W (x; p)), and performs left-right inversion, and then the eye texture image J (T (X; λ)) is extracted (step S580). Then, the process returns to step S520, and the texture detection process (steps S520 to S560) for the left eye is executed. Note that the eye texture model for the right eye can be used by inverting the eye texture image J (T (x; λ)). If it is determined in the convergence determination (step S530) that the left eye has been converged, the left eye texture detection process is completed, and the eye texture A (x) specified by the value of the texture parameter λ _k set at this time is finally obtained. Identified as a left eye appearance. When the texture parameters λ _k of the right eye and the left eye are obtained as described above, the eye feature detection process is completed, and then the gaze determination / correction process is executed. The finally obtained right-eye and left-eye texture parameters λ _k correspond to “approximate feature amounts” in the claims.

A5. Eye gaze determination / correction processing:
FIG. 21 is a flowchart showing the flow of the line-of-sight determination / correction process. First, the line-of-sight determination unit 250 acquires a texture parameter λ ₁ that correlates with the position of the pupil of the right eye and a shape parameter p ₁ that correlates with face swing (step S610). The line-of-sight determination unit 250 reads the angle table T (FIG. 1) stored in the internal memory 120 (step S615).

FIG. 22 shows the angle table T. In the angle table T, a correspondence relationship between the value of the shape parameter p ₁ and the face swing angle θ and a correspondence relationship between the value of the texture parameter λ ₁ and the relative line-of-sight angle γ are defined. With reference to such an angle table T, the line-of-sight determination unit 250 can acquire the face swing angle θ and the eye relative line-of-sight angle γ existing in the target image OI.

FIG. 23 is a schematic diagram illustrating the swing angle θ and the relative line-of-sight angle γ. In the figure, the state in which the target image OI is photographed by the digital still camera DSC is viewed from the head of the person. An image plane IP of the target image OI is formed in a direction orthogonal to the central optical axis CA of the digital still camera DSC. The swing angle θ is an angle formed by the face front axis FA with respect to the central optical axis CA (size is 0 to 90 degrees), and is defined such that right swing is positive and left swing is negative. In addition, since the face detection in which the face shake becomes large is not normally performed, the maximum absolute value of the swing angle θ is a value smaller than 90 degrees. On the other hand, the relative line-of-sight angle γ is an angle formed by the line-of-sight axis VA with respect to the face front axis FA, and is defined such that right-handed swing is positive and left-handed swing is negative. Since the texture parameter λ ₁ is detected based on the average shape image I (W (x; p)) averaged to face the front using the values of the global parameter and the shape parameter p _i , This corresponds to the relative viewing angle γ with respect to the axis FA. Note that it can be considered that the directions of the right eye axis VA and the left eye axis VA coincide. Therefore, in this embodiment, the relative line-of-sight angle γ for the left eye is not specified.

When the swing angle θ and the relative line-of-sight angle γ are obtained, the line-of-sight determination unit 250 adds the swing angle θ and the relative line-of-sight angle γ to obtain an absolute line-of-sight angle that means the angle of the line of sight axis VA with respect to the central optical axis CA. φ is calculated (step S620). Next, the line-of-sight determination unit 250 determines whether or not the absolute line-of-sight angle φ is larger than a predetermined threshold (10 degrees in the present embodiment) (step S630), and when the absolute line-of-sight angle φ is larger than the threshold. The line-of-sight correction unit 260 calculates a correction angle η that corrects the relative line-of-sight angle γ so that the absolute line-of-sight angle φ is 0 (step S635). Specifically, a correction angle η (= 0−θ−γ) may be calculated such that a value obtained by adding the swing angle θ, the relative line-of-sight angle γ, and the correction angle η is zero. Next, the line-of-sight correction unit 260 calculates the sum of the original relative line-of-sight angle γ and the correction angle η as an ideal line-of-sight angle ε (= γ + η) (step S638), and refers to the angle table T to determine the relative line-of-sight angle. The texture parameter λ ₁ corresponding to the value of γ and the ideal viewing angle ε is acquired (step S640).

When the texture parameter λ ₁ having the ideal line-of-sight angle ε can be acquired, the line-of-sight correction unit 260 acquires only the texture parameter λ _{1 from} the texture parameter λ _k for the right eye finally obtained by the eye feature detection process. The texture parameter λ ₁ is corrected (step S645). Then, the line-of-sight correction unit 260 calculates the texture A (x) of the right eye by substituting the texture parameter λ _k into the eye texture model (formula (2b)) without changing the values of the other texture parameters λ _k. (Step S650). In the calculated texture A (x) of the right eye, the position of the pupil is corrected according to the corrected texture parameter λ ₁ .

  FIG. 24 shows how the eye pupil position is corrected in this embodiment (steps S645 to S690). The texture A (x) of the right eye calculated in step S650 corresponds to an “ideal eye image” in the claims. Next, the line-of-sight correction unit 260 replaces the area corresponding to the average eye area Q1 in the average shape image I (W (x; p)) with the calculated texture A (x) of the right eye (S655). Through the above processing, the visual axis VA of the right eye of the average shape image I (W (x; p)) is corrected so as to be orthogonal to the image plane IP.

Subsequently, the line-of-sight correction unit 260 performs the same process on the left eye. The line-of-sight correction unit 260 obtains the correction angle −η for the left eye by inverting the sign of the correction angle η for the right eye (step S660). This is because the directions of the right-eye viewing axis VA and the left-eye viewing axis VA need to coincide with each other, and when the right eye is corrected by the correction angle η, the left eye needs to be corrected by the correction angle −η. For example, when the pupil position is corrected inward (nose) by an amount corresponding to the correction angle η for the right eye, at the same time, the left eye is outward (close to the face contour) by an amount corresponding to the correction angle η. It is necessary to correct the pupil position. Next, the line-of-sight correction unit 260 calculates the sum of the relative line-of-sight angle γ and the correction angle −η as an ideal line-of-sight angle ε (= γ−η) (step S663), and refers to the angle table T to determine the relative line-of-sight angle. The texture parameter λ ₁ corresponding to the value of γ corresponding to the ideal viewing angle ε is acquired (step S665). When the texture parameter λ ₁ can be acquired in this manner, the line-of-sight correction unit 260 determines the texture parameter λ ₁ correlated with the pupil position among the texture parameters λ _k for the left eye finally obtained by the eye feature detection process. The acquired texture parameter λ ₁ is corrected (step S670).

Then, the line-of-sight correction unit 260 calculates the texture A (x) of the left eye by substituting the texture parameter λ _k into the eye texture model (formula (2b)) without changing the values of the other texture parameters λ _k. (Step S675). Since the texture A (x) of the right eye is reproduced in the eye texture model, in step S670, after the texture A (x) is calculated, the image is reversed left and right. The area corresponding to the average eye area Q2 in the average shape image I (W (x; p)) is replaced with the calculated texture A (x) of the left eye (step S680). With the above processing, the position of the pupil of the left eye of the average shape image I (W (x; p)) can be made to follow the line-of-sight correction for the right eye.

Next, the average shape image I (W (x; p)) is restored to the face image in the original face area FA (step S685). Specifically, the original face shape is reproduced by applying the global parameter and the shape parameter p _i obtained by the face feature detection process to the face shape model (formula (1a)). Further, each pixel of the average shape image I (W (x; p)) is warped based on the position of the feature point CP in the restored face shape s to restore the original face image. Basically, the original face image is restored. However, since the pupil position is corrected for the eyes, the correction is reflected. The relative positional relationship of the feature points CP in the face shape model is also restored so as to coincide with the relative positional relationship of the feature points CP (FIG. 18) specified in the face feature detection process. The color value of the corresponding pixel in the target image OI is replaced with the color value of each pixel of the restored face image so that the positions of the feature points CP are matched (step S690). As a result, as shown in the lower part of FIG. 24, it is possible to obtain an attention image OI having a face image that has been corrected so that the visual axis VA is orthogonal to the image plane IP. Finally, the target image OI is output to the print processing unit 320, and the target image OI with the corrected red-eye is printed (step S695).

As described above, in this embodiment, correction can be performed so that the line-of-sight axis VA is orthogonal to the image plane IP, and even when the subject person is not looking at the front at the time of shooting, correction is made to the so-called camera line of sight. can do. Since only the texture parameter λ ₁ that correlates with the pupil position is corrected in the eye texture model, it is possible to correct the ideal line-of-sight direction while maintaining characteristics (such as the pupil color) other than the pupil position.

B. Variation:
The present invention is not limited to the above-described examples and embodiments, and can be implemented in various modes without departing from the gist thereof. For example, the following modifications are possible.

B1. Modification 1:
FIG. 25 is a flowchart showing the flow of the line-of-sight determination / correction processing according to this modification. This modification is substantially the same as the line-of-sight determination / correction process of the previous embodiment, but differs from the previous embodiment in that step S617 is executed between step S615 and step S620 of the previous embodiment. The line-of-sight determination unit 250 refers to the angle table T (FIG. 1), acquires the shape parameter p ₁ face swing angle θ, and determines whether the absolute value of the swing angle θ is greater than 25 degrees. (Step S617). If the absolute value of the swing angle θ is greater than 25 degrees, the process is terminated without correcting the pupil position. That is, the pupil position is corrected only when the absolute value of the swing angle θ is 25 degrees or less. This is because when the absolute value of the face swing angle θ is larger than 25 degrees, it becomes unnatural if the pupil position is corrected so that the visual axis VA is orthogonal to the image plane IP. According to this modification, only natural correction can be performed.

B2. Modification 2:
FIG. 26 is a schematic diagram for explaining an ideal direction of the visual axis VA in the present modification. In this modification, the position of the pupil is corrected so that the visual axis VA coincides with the face front axis FA. In this case, it suffices that the relative gaze angle γ can be corrected to 0 regardless of the value of the face swing angle θ. As described in FIG. 11A, when the texture parameter λ ₁ is 0, the eyes are not inward or outward, and the line of sight is directed to the front, so the texture parameter λ ₁ is corrected to 0. Similarly to the previous embodiment, the right eye and left eye texture A (x) may be calculated as the ideal eye image.

B3. Modification 3:
FIG. 27 is a schematic diagram for explaining an ideal direction of the visual axis VA in the present modification. In this modified example, the subject distance L, the lens focal length, the sensor width, and the number of horizontal pixels are attached and recorded on the target image OI by the digital still camera DSC. The subject distance L can be considered as a distance between the image plane IP formed at the face position of the subject and the digital still camera DSC. In this embodiment, the ideal line-of-sight axis VA is not perpendicular to the image plane IP, but the left and right center position of the face area FA specified by face detection (the position of the left and right center of the target image OI is (B = 0). ) And the ideal angle κ along the straight line LL connecting the digital still camera DSC to the ideal viewing axis VA.

  If lens distortion is not taken into account, the substantial position B of the face area FA in the image plane IP where the face exists can be specified based on the subject distance L, the lens focal length, the sensor width, and the number of horizontal pixels. it can. In the previous embodiment, the line-of-sight correction unit 260 calculates the correction angle η, which is a relative line-of-sight angle γ such that the absolute line-of-sight angle φ is 0 (step S635). It is only necessary to calculate a correction angle η that is a relative line-of-sight angle γ such that is an ideal angle κ. That is, the correction angle may be calculated by (η = κ−θ−γ). The ideal angle κ is an angle that satisfies tan κ = B / L. Then, after step S640, the same processing as in the previous embodiment may be executed based on the correction angle η.

B4. Modification 4:
In the feature point CP setting position correction process of the above embodiment, the average shape image I (W (x; p)) is calculated based on the target image OI to determine the installation position of the feature point CP of the target image OI as the average face image A. _Although it is matched with the installation position of the feature point CP of ₀ (x), the arrangement of both feature points CP may be matched by performing image conversion on the average face image A ₀ (x). That is, in the above embodiment, the average shape image I (W (x; p)) as the “face region” is changed, but the average face image A ₀ (x) is changed to the shape parameter p _i and the texture parameter λ. You may change according to the change of _k . In any case, the shape parameter p _i and the texture parameter λ _k that reduce the difference between the “face region” included in the target image OI and the face image reproduced by the face model are corrected. Similarly, in the eye feature detection process of the above-described embodiment, the eye texture image J (T (x; λ)) is calculated based on the attention image OI, thereby converting the eye image of the attention image OI into the average eye image A ₀ (x). However, they may be matched by performing image conversion on the average eye image A ₀ (x).

B5. Modification 5:
The sample image SI (FIG. 3) in the above embodiment is merely an example, and the number and type of images employed as the sample image SI can be arbitrarily set. In the embodiment, the predetermined feature portion (see FIG. 4) of the face indicated by the position of the feature point CP is merely an example, and a part of the feature portion set in the embodiment may be omitted, Other parts may be adopted as the part.

In the embodiment, the size of the average face image A ₀ (x) is not limited to 56 pixels × 56 pixels, and may be other sizes. Further, the average face image A ₀ (x) does not need to include the mask area MA (FIG. 8), and may be configured only by the average shape area BSA. Further, instead of the average face image A ₀ (x), another reference face image set based on the statistical analysis of the sample image SI may be used.

  In the embodiment, the shape model and the texture model are set using AAM. However, the shape model and texture model using other modeling methods (for example, a method called Morphable Model or a method called Active Blob) are used. A texture model may be set.

  In the embodiment, the image stored in the memory card MC is set as the attention image OI. However, the attention image OI may be an image acquired via a network, for example. In the above embodiment, image processing by the printer 100 as an image processing apparatus has been described. However, part or all of the processing is executed by another type of image processing apparatus such as a personal computer, a digital still camera, or a digital video camera. It is good also as what is done. The printer 100 is not limited to an ink jet printer, and may be another type of printer such as a laser printer or a sublimation printer.

  In the embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware.

  In addition, when part or all of the functions of the present invention are realized by software, the software (computer program) can be provided in a form stored in a computer-readable recording medium. In the present invention, the “computer-readable recording medium” is not limited to a portable recording medium such as a flexible disk or a CD-ROM, but an internal storage device in a computer such as various RAMs and ROMs, a hard disk, etc. It also includes an external storage device fixed to the computer.

  DESCRIPTION OF SYMBOLS 100 ... Printer, 110 ... CPU, 120 ... Internal memory, 140 ... Operation part, 150 ... Display part, 160 ... Printing mechanism, 170 ... Card interface, 172 ... Card slot, 200 ... Image processing part, 210 ... Setting part, 220 ... feature position detection unit, 222, 242 ... generation unit, 224, 244 ... calculation unit, 226, 246 ... correction unit, 230 ... face area detection unit, 240 ... texture detection unit, 250 ... gaze determination unit, 260 ... gaze correction , 310... Display processing unit, 320... Print processing unit.

Claims (10)

An image processing device that corrects the position of a pupil in an eye included in an image of interest,
A face area detection unit for detecting a face area from the attention image;
An eye area detector for detecting an eye area from the face image;
An acquisition unit that acquires an eye model that reproduces an eye image while changing the position of a pupil in the eye based on a feature amount;
A feature amount detection unit that detects a feature amount when the eye image reproduced by the eye model approximates the eye region as an approximate feature amount;
The approximate feature amount is corrected so that the eye model reproduces an ideal eye image in which the position of the pupil is ideal, and the ideal eye image is reproduced by the model based on the corrected approximate feature amount. An image processing apparatus comprising: a correction unit that corrects an eye area. The acquisition unit performs principal component analysis on color values of each pixel of a plurality of sample images including eye images having different line-of-sight directions, and reproduces the eye image by linearly combining eigenvectors of the principal components. And get
The image processing apparatus according to claim 1, wherein the feature amount is a weighting coefficient related to each principal component when linearly combining the eigenvectors.   The image processing apparatus according to claim 1, wherein a face swing angle included in the face area is detected, and the eye area is corrected only when the swing angle is within a predetermined range.   4. The image processing apparatus according to claim 1, wherein, in the ideal eye image, a pupil position whose line-of-sight direction is perpendicular to the image plane of the target image is reproduced. 5.   4. The image processing apparatus according to claim 1, wherein, in the ideal eye image, a position of a pupil whose line-of-sight direction is a front direction of a face included in the face region is reproduced.   4. The image processing apparatus according to claim 1, wherein in the ideal eye image, a position of a pupil whose line-of-sight direction is toward the center of the target image is reproduced. An image processing method for correcting the position of a pupil in an eye included in a target image,
Detecting a face area from the attention image;
An eye area is detected from the face image;
Obtain an eye model that reproduces the eye image while changing the position of the pupil in the eye based on the feature amount,
Detecting the feature amount when the eye image reproduced by the eye model approximates the eye region as an approximate feature amount;
The approximate feature amount is corrected so that the eye model reproduces an ideal eye image in which the position of the pupil is ideal, and the ideal eye image is reproduced by the model based on the corrected approximate feature amount. An image processing device for correcting an eye area. An image processing method for correcting the position of a pupil in an eye included in a target image,
Detecting a face area from the attention image;
An eye area is detected from the face image;
Obtain an eye model that reproduces the eye image while changing the position of the pupil in the eye based on the feature amount,
Detecting the feature amount when the eye image reproduced by the eye model approximates the eye region as an approximate feature amount;
The approximate feature amount is corrected so that the eye model reproduces an ideal eye image in which the position of the pupil is ideal, and the ideal eye image is reproduced by the model based on the corrected approximate feature amount. An image processing method for correcting an eye area. An image processing program for causing a computer to realize a function of correcting the position of a pupil in an eye included in an image of interest,
A face area detection function for detecting a face area from the attention image;
An eye area detection function for detecting an eye area from the face image;
An acquisition function for acquiring an eye model that reproduces an eye image while changing the position of the pupil in the eye based on the feature amount;
A feature amount detection function for detecting, as an approximate feature amount, a feature amount when the eye image reproduced by the eye model approximates the eye region;
The approximate feature amount is corrected so that the eye model reproduces an ideal eye image in which the position of the pupil is ideal, and the ideal eye image is reproduced by the model based on the corrected approximate feature amount. An image processing program that causes a computer to realize a correction function for correcting an eye area. A printing device that prints an image of interest,
A face area detection unit for detecting a face area from the attention image;
An eye area detector for detecting an eye area from the face image;
An acquisition unit that acquires an eye model that reproduces an eye image while changing the position of a pupil in the eye based on a feature amount;
A feature amount detection unit that detects a feature amount when the eye image reproduced by the eye model approximates the eye region as an approximate feature amount;
The approximate feature amount is corrected so that the eye model reproduces an ideal eye image in which the position of the pupil is ideal, and the ideal eye image is reproduced by the model based on the corrected approximate feature amount. A correction unit for correcting the eye area;
And a printing unit that prints the attention image in which the eye area is corrected.