JP2010245721A - Face image processing - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily perform effective image processing not depending on face direction in a face image. <P>SOLUTION: An image processor has a face direction estimation part which estimates face direction in a face image, a face direction changing part which performs face direction changing processing so that the face direction estimated for the face image becomes close to a target face direction, an image processing part which performs image processing for the face image, the face direction of which is changed, and a face direction recovery part which performs face direction recovery processing to restore the face direction changed by the face direction changing processing for the face image after image processing. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a technique for performing image processing on a face image.

  A technique for performing various image processing (for example, image processing for enlarging the size of a pupil) on a face image is known (for example, see Patent Document 1).

JP 2005-31990 A

  There are various face orientations in the face image. Conventionally, depending on the face orientation of a face image, there have been cases where image processing cannot be executed effectively, for example, the result of image processing on the face image becomes unnatural.

  SUMMARY An advantage of some aspects of the invention is to easily realize effective image processing regardless of the face orientation in a face image.

  In order to solve at least a part of the above problems, the present invention can be realized as the following forms or application examples.

Application Example 1 An image processing apparatus that performs image processing on a face image,
A face orientation estimating unit for estimating a face orientation in the face image;
A face orientation changing unit that performs a face orientation changing process so that the estimated face orientation approaches the target face orientation with respect to the face image;
An image processing unit that performs the image processing on the face image after the face orientation change processing;
An image processing apparatus comprising: a face orientation restoring unit that performs a face orientation restoring process for restoring the face orientation changed by the face orientation changing process to the face image after the image processing.

  In this image processing apparatus, the face orientation in the face image is estimated, face orientation change processing is performed so that the estimated face orientation with respect to the face image approaches the target face orientation, and the face image after the face orientation change processing is processed. The image processing is performed on the face image and the face image after the image processing is subjected to the face orientation restoration process for restoring the face orientation changed by the face orientation changing process. Effective image processing can be realized even in the orientation, and image processing can be easily realized because it is not necessary to set the image processing mode for each face orientation.

[Application Example 2] The image processing apparatus according to Application Example 1,
The image processing apparatus, wherein the target face orientation is a front orientation.

  In this image processing apparatus, no matter what face orientation the face image has, the image processing is executed after the face orientation changing process is performed on the face image so that the face orientation approaches the front orientation. Therefore, good results can be obtained even if image processing that is assumed to be performed on a face image facing the front is performed on a face image facing any face.

[Application Example 3] The image processing apparatus according to Application Example 1,
The image processing unit can execute a plurality of types of the image processing,
The image processing apparatus further includes:
A storage unit that stores the correspondence between the type of image processing and the face orientation;
The face orientation changing unit is an image processing device that sets, in the correspondence relationship, a face orientation associated with a type of the image processing executed by the image processing unit as the target face orientation.

  In this image processing apparatus, the correspondence relationship between the type of image processing and the face orientation is stored, and the face orientation associated with the type of image processing in the correspondence relationship is set as the target face orientation. The processing can be realized effectively and easily.

[Application Example 4] The image processing apparatus according to any one of Application Examples 1 to 3,
The image processing unit performs the image processing on the face image after the face orientation change processing when the estimated face orientation is in a first range relatively far from the target face orientation, When the estimated face orientation is in the second range that is closer to the target face orientation compared to the first range, the image processing is performed on the face image before the face orientation change processing is performed. An image processing apparatus.

  In this image processing apparatus, when the estimated face orientation is in the first range relatively far from the target face orientation, image processing is performed on the face image after the face orientation change process, while the estimated face orientation is estimated. When the face orientation is in the second range that is closer to the target face orientation compared to the first range, the image processing is performed on the face image before the face orientation change processing is performed, and thus efficient processing Can be realized.

Application Example 5 The image processing apparatus according to any one of Application Examples 1 to 4, further comprising:
Defined by a shape model representing a face shape defined by the position of a predetermined feature part of the face by a reference shape and at least one face shape feature amount including a face orientation, and a pixel value of a face image having the reference shape Using a texture model that represents a face texture by a reference texture and at least one face texture feature amount, and a feature position specifying unit that specifies the position of the feature part in the face image,
The face direction changing unit is an image processing apparatus that executes the face direction changing process by changing the face shape feature amount representing a face direction in the shape model.

  In this image processing apparatus, the face orientation changing process can be realized by changing the face shape feature amount representing the face orientation in the shape model.

[Application Example 6] The image processing apparatus according to Application Example 5,
The characteristic part includes an eye outline,
The image processing apparatus includes a process of enlarging an eye region defined by an eye contour whose position is specified in the face image.

  In this image processing apparatus, image processing for enlarging the eye area can be effectively and easily realized regardless of the face orientation in the face image.

[Application Example 7] The image processing apparatus according to Application Example 5,
The characteristic part includes an eye outline,
The image processing apparatus includes a process of bringing the sizes of the left and right eye regions defined by the contours of the eyes whose positions are specified in the face image closer to each other.

  In this image processing apparatus, image processing for making the sizes of the left and right eye regions close to each other can be effectively and easily realized regardless of the face orientation in the face image.

Application Example 8 The image processing apparatus according to any one of Application Example 5 to Application Example 7,
The shape model and the texture model are set based on a statistical analysis for a plurality of sample face images whose arrangement of feature points indicating the positions of the feature parts is known,
The feature position specifying unit determines an initial arrangement of the feature points in the face image based on an arrangement of the feature points in the reference shape, and a reference defined by the face image, the reference shape, and the reference texture. An image processing apparatus that identifies the position of the feature portion in the face image by updating the arrangement of the feature points based on a comparison result with the face image.

  In this image processing apparatus, the position of the characteristic part in the face image can be specified by the characteristic position specifying unit.

[Application Example 9] The image processing apparatus according to Application Example 8,
The reference shape is an average shape representing an average position of the characteristic part in the plurality of sample face images,
The image processing apparatus, wherein the reference texture is an average texture that represents an average of pixel values at the positions of the characteristic portions of the plurality of sample face images that have been transformed into the average shape.

  In this image processing apparatus, it is possible to accurately specify the position of the feature part in the face image by the feature position specifying unit.

Application Example 10 An image processing method for performing image processing on a face image,
(A) estimating a face orientation in the face image;
(B) performing a face orientation change process on the face image so that the estimated face orientation approaches the target face orientation;
(C) performing the image processing on the face image after the face orientation change processing;
(D) An image processing method comprising: performing a face orientation restoring process for restoring the face orientation changed by the face orientation changing process to the face image after the image processing.

Application Example 11 A computer program for image processing that performs image processing on a face image,
A face direction estimating function for estimating a face direction in the face image;
A face orientation change function for performing a face orientation change process so that the estimated face orientation approaches the target face orientation with respect to the face image;
An image processing function for performing the image processing on the face image after the face orientation change processing;
The computer program which makes a computer implement | achieve the face orientation restoration function which performs the face orientation restoration process which returns the face orientation changed by the said face orientation change process with respect to the said face image after the said image processing.

Application Example 12 A printing apparatus,
A face direction estimation unit for estimating a face direction in a face image;
A face orientation changing unit that performs a face orientation changing process so that the estimated face orientation approaches the target face orientation with respect to the face image;
An image processing unit that performs image processing on the face image after the face orientation change processing;
A face orientation restoring unit that performs a face orientation restoring process for restoring the face orientation changed by the face orientation changing process to the face image after the image processing;
And a printing unit that prints the face image after the face orientation restoring process.

  It should be noted that the present invention can be realized in various modes, for example, an image processing method and apparatus, an image correction method and apparatus, a printing method and apparatus, and a computer for realizing the functions of these methods or apparatuses. The present invention can be realized in the form of a program, a recording medium recording the computer program, a data signal including the computer program and embodied in a carrier wave, and the like.

1 is an explanatory diagram schematically illustrating a configuration of a printer 100 as an image processing apparatus according to an embodiment of the present invention. It is a flowchart which shows the flow of the AAM setting process in a present Example. It is explanatory drawing which shows an example of sample face image SI. It is explanatory drawing which shows an example of the setting method of the feature point CP in the sample face image SI. It is explanatory drawing which shows an example of the coordinate of the feature point CP set to sample face image SI. Is an explanatory diagram showing an example of the average shape s _0. It is explanatory drawing which shows an example of the method of the warp W of sample face image SI. It is explanatory drawing which shows an example of average face image _A0 (x). It is a flowchart which shows the flow of the face characteristic position specific process in a present Example. It is explanatory drawing which shows an example of the detection result of the face area FA in the target image OI. It is a flowchart which shows the flow of the initial position determination processing of the feature point CP in a present Example. It is explanatory drawing which shows an example of temporary arrangement | positioning of the feature point CP in the target image OI. It is explanatory drawing which shows an example of the average shape image I (W (x; p)). It is explanatory drawing which shows an example of the initial arrangement | positioning of the feature point CP in the target image OI. It is a flowchart which shows the flow of the feature point CP arrangement | positioning update process in a present Example. It is explanatory drawing which shows an example of the result of a face feature position specific process. It is a flowchart which shows the flow of the specific image process in a present Example. It is explanatory drawing which shows an example of the content of the reference | standard face direction information SDI. It is explanatory drawing which shows the expansion method of an eye area | region.

Next, embodiments of the present invention will be described in the following order based on examples.
A. Example:
A-1. Configuration of image processing device:
A-2. AAM setting process:
A-3. Facial feature location processing:
A-4. Specific image processing:
B. Variation:

A. Example:
A-1. Configuration of image processing device:
FIG. 1 is an explanatory diagram schematically showing the configuration of a printer 100 as an image processing apparatus according to an embodiment of the present invention. The printer 100 of this embodiment is an ink jet color printer that supports so-called direct printing, in which an image is printed based on image data acquired from a memory card MC or the like. The printer 100 includes a CPU 110 that controls each unit of the printer 100, an internal memory 120 configured by a ROM and a RAM, an operation unit 140 configured by buttons and a touch panel, a display unit 150 configured by a liquid crystal display, and a printer. An engine 160 and a card interface (card I / F) 170 are provided. The printer 100 may further include an interface for performing data communication with other devices (for example, a digital still camera or a personal computer). Each component of the printer 100 is connected to each other via a bus.

  The printer engine 160 is a printing mechanism that performs printing based on print data. The card interface 170 is an interface for exchanging data with the memory card MC inserted into the card slot 172. In this embodiment, an image file including image data is stored in the memory card MC.

  The internal memory 120 stores an image processing unit 200, a display processing unit 310, and a print processing unit 320. The image processing unit 200 is a computer program for executing face feature position specifying processing and specific image processing under a predetermined operating system. The face feature position specifying process of the present embodiment is a process of specifying (detecting) the position of a predetermined feature part (for example, the corner of the eye, the nose head, or the face line) in the face image. The specific image processing of the present embodiment is processing for performing the selected type of image processing on the face image. The face feature position specifying process and the specific image process will be described in detail later.

  The image processing unit 200 includes a face feature position specifying unit 210, a face area detecting unit 230, a specific image processing unit 240, a face direction changing unit 250, and a face direction restoring unit 260 as program modules. . The face feature position specifying unit 210 includes an initial arrangement unit 211, an image conversion unit 212, a determination unit 213, an update unit 214, and a normalization unit 215. The functions of these units will be described in detail in the description of face feature position specifying processing and specific image processing described later.

  The display processing unit 310 is a display driver that controls the display unit 150 to display processing menus, messages, images, and the like on the display unit 150. The print processing unit 320 is a computer program for generating print data from image data, controlling the printer engine 160, and printing an image based on the print data. The CPU 110 reads out and executes these programs (the image processing unit 200, the display processing unit 310, and the print processing unit 320) from the internal memory 120, thereby realizing the functions of these units.

  The internal memory 120 also stores AAM information AMI and reference face orientation information SDI. The AAM information AMI is information set in advance by an AAM setting process described later, and is referred to in a face feature position specifying process and a specific image process described later. The reference face orientation information SDI is information for designating a reference face orientation for each of a plurality of types of specific image processing, and is referred to in specific image processing described later. The contents of the AAM information AMI and the reference face orientation information SDI will be described in detail later.

A-2. AAM setting process:
FIG. 2 is a flowchart showing the flow of AAM setting processing in this embodiment. The AAM setting process is a process for setting a shape model and a texture model used for modeling an image called AAM (Active Appearance Model).

  In step S110, a plurality of images representing a person's face are set as sample face images SI. FIG. 3 is an explanatory diagram showing an example of the sample face image SI. As shown in FIG. 3, the sample face image SI has various attributes such as personality, race / gender, facial expression (anger, laughter, embarrassment, surprise, etc.), and orientation (front, upward, downward, right, left, etc.). Are set to include different images. If the sample face image SI is set in such a manner, any face image can be accurately modeled by the AAM, and accurate face feature position specifying processing (described later) for any face image can be executed. It becomes possible. The sample face image SI is also called a learning face image.

  In step S120 (FIG. 2), a feature point CP is set in each sample face image SI. FIG. 4 is an explanatory diagram showing an example of a method for setting the feature point CP in the sample face image SI. The feature point CP is a point indicating the position of a predetermined feature part in the face image. In this embodiment, as predetermined feature parts, a predetermined position on the eyebrows in a person's face (for example, an end point or a four-divided point, the same applies hereinafter), a predetermined position on the eye contour, and a predetermined position on the nose and nose contours 68 parts such as a predetermined position on the contour of the upper and lower lips and a predetermined position on the contour of the face (face line) are set. In other words, in the present embodiment, predetermined positions in facial organs (eyebrows, eyes, nose, mouth) and facial contours that are commonly included in a person's face are set as feature parts. As shown in FIG. 4, the feature points CP are set (arranged) at positions representing 68 feature parts designated by the operator in each sample face image SI. Since each feature point CP set in this way corresponds to each feature part, the arrangement of the feature points CP in the face image can be expressed as defining the shape of the face.

  The position of the feature point CP in the sample face image SI is specified by coordinates. FIG. 5 is an explanatory diagram showing an example of the coordinates of the feature point CP set in the sample face image SI. In FIG. 5, SI (j) (j = 1, 2, 3,...) Indicates each sample face image SI, and CP (k) (k = 0, 1,..., 67) indicates each. A feature point CP is shown. CP (k) -X represents the X coordinate of the feature point CP (k), and CP (k) -Y represents the Y coordinate of the feature point CP (k). The coordinates of the feature point CP include predetermined reference points in the sample face image SI normalized with respect to the size of the face, the inclination of the face (inclination in the image plane), and the position in the X and Y directions of the face. For example, coordinates with the origin at the lower left point of the image are used. In the present embodiment, a case where a plurality of human faces are included in one sample face image SI is allowed (for example, two faces are included in the sample face image SI (2)). Each person in one sample face image SI is specified by a person ID.

  In step S130 (FIG. 2), an AAM shape model is set. Specifically, a principal component analysis is performed on a coordinate vector (see FIG. 5) composed of the coordinates (X coordinate and Y coordinate) of 68 feature points CP in each sample face image SI, and the position of the feature point CP is determined. The face shape s specified by is modeled by the following equation (1). The shape model is also referred to as a feature point CP arrangement model.

In the above formula (1), s ₀ is an average shape. FIG. 6 is an explanatory diagram showing an example of the average shape s ₀ . As shown in FIGS. 6A and 6B, the average shape s ₀ is a model representing the average face shape specified by the average position (average coordinate) for each feature point CP of the sample face image SI. It is. In the present embodiment, in the average shape s ₀ , a region surrounded by a straight line connecting feature points CP (face lines, eyebrows, feature points CP corresponding to the eyebrows, see FIG. 4) located on the outer periphery (FIG. 6 ( b) is indicated by hatching) and is referred to as “average shape region BSA”. In the average shape s ₀ , as shown in FIG. 6A, a plurality of triangular regions TA having the feature points CP as vertices are set so as to divide the average shape region BSA into a mesh shape.

In the above equation (1) representing the shape model, s _i is a shape vector, and p _i is a shape parameter representing the weight of the shape vector s _i . The shape vector s _i is a vector representing the characteristics of the face shape s, and is specifically an eigenvector corresponding to the i-th principal component obtained by principal component analysis. That is, the number of eigenvectors set based on the cumulative contribution rate is adopted as the shape vector s _{i in} order from the eigenvector corresponding to the principal component having a larger variance. In the present embodiment, the first shape vector s ₁ corresponding to the first principal component having the largest variance is a vector that is substantially correlated with the left / right swing of the face, and corresponds to the second principal component having the second largest variance. The second shape vector s ₂ is a vector that is substantially correlated with the vertical swing of the face. The third shape vector s ₃ corresponding to the third principal component having the third largest variance is a vector that is substantially correlated with the aspect ratio of the face shape, and corresponds to the fourth principal component having the fourth largest variance. The fourth shape vector s ₄ is a vector that is substantially correlated with the degree of mouth opening.

As shown in the above equation (1), in the shape model in the present embodiment, the face shape s representing the arrangement of the feature points CP is modeled as the sum of the average shape s ₀ and the linear combination of the n shape vectors s _i. It becomes. By appropriately setting the shape parameter p _i in the shape model, the face shape s in any image can be reproduced. The average shape s ₀ and shape vector s _i set in the shape model setting step (step S 130 in FIG. 2) are stored in the internal memory 120 as AAM information AMI (FIG. 1). The average shape s ₀ corresponds to the reference shape in the present invention, and the product of the shape vector s _i and the shape parameter p _i corresponds to the shape feature amount in the present invention.

In step S140 (FIG. 2), an AAM texture model is set. Specifically, first, for each sample face image SI, image conversion (hereinafter referred to as “warp”) is performed so that the arrangement of the feature points CP in the sample face image SI is equal to the arrangement of the feature points CP in the average shape s ₀ . W ”).

FIG. 7 is an explanatory diagram illustrating an example of a warp W method for the sample face image SI. In each sample face image SI, similarly to the average shape s ₀ , a plurality of triangular areas TA that divide the area surrounded by the feature points CP located on the outer periphery into mesh shapes are set. The warp W is a set of affine transformations for each of the plurality of triangular areas TA. That is, in the warp W, an image of a certain triangular area TA in the sample face image SI is affine transformed into an image of the corresponding triangular area TA in the average shape s ₀ . The warp W generates a sample face image SI (hereinafter referred to as “sample face image SIw”) in which the arrangement of the feature points CP is equal to the arrangement of the feature points CP in the average shape s ₀ .

  Each sample face image SIw has a rectangular frame containing an average shape region BSA (shown with hatching in FIG. 7) as an outer periphery, and a region other than the average shape region BSA (hereinafter also referred to as “mask region MA”). Is generated as a masked image. An image area in which the average shape area BSA and the mask area MA are combined is referred to as a reference area BA. Each sample face image SIw is normalized as an image having a size of 56 pixels × 56 pixels, for example.

  Next, a principal component analysis is performed on a luminance value vector composed of luminance values in each pixel group x of each sample face image SIw, and a facial texture (also referred to as “appearance”) A (x) is expressed by the following equation: Modeled by (2). The pixel group x is a set of pixels located in the average shape area BSA.

In the above equation (2), A ₀ (x) is an average face image. FIG. 8 is an explanatory diagram showing an example of the average face image A ₀ (x). The average face image A ₀ (x) is an average image of the sample face images SIw (see FIG. 7) after the warp W. That is, the average face image A ₀ (x) is an image calculated by taking the average of pixel values (luminance values) for each of the pixel groups x in the average shape area BSA of the sample face image SIw. Therefore, the average face image A ₀ (x) is a model representing the average face texture (appearance) in the average face shape. Note that the average face image A ₀ (x) is composed of an average shape area BSA and a mask area MA, similarly to the sample face image SIw. Also in the average face image A ₀ (x), an image area that is a combination of the average shape area BSA and the mask area MA is referred to as a reference area BA.

In the above equation (2) representing the texture model, A _i (x) is a texture vector, and λ _i is a texture parameter representing the weight of the texture vector A _i (x). The texture vector A _i (x) is a vector representing the characteristics of the facial texture A (x), and specifically, is an eigenvector corresponding to the i-th principal component obtained by principal component analysis. That is, the number m of eigenvectors set based on the cumulative contribution rate is adopted as the texture vector A _i (x) in order from the eigenvector corresponding to the principal component having a larger variance. In the present embodiment, the first texture vector A ₁ (x) corresponding to the first principal component having the largest variance is a vector that is substantially correlated with a change in face color (also considered as a gender difference). The second texture vector A ₂ (x) corresponding to the second principal component having a large variance is a vector that is substantially correlated with a change in the shadow component (also considered as a change in the light source position).

As shown in the above equation (2), in the texture model in the present embodiment, the facial texture A (x) representing the appearance of the face is the average face image A ₀ (x) and m texture vectors A _i (x ) And the linear combination. By appropriately setting the texture parameter λ _i in the texture model, it is possible to reproduce the facial texture A (x) in any image. Note that the average face image A ₀ (x) and the texture vector A _i (x) set in the texture model setting step (step S140 in FIG. 2) are stored in the internal memory 120 as AAM information AMI (FIG. 1). . The average face image A ₀ (x) corresponds to the reference texture in the present invention, and the product of the texture vector A _i (x) and the texture parameter λ _i corresponds to the texture feature amount in the present invention.

By the AAM setting process described above (FIG. 2), a shape model for modeling the face shape and a texture model for modeling the face texture are set. By combining the set shape model and the texture model, that is, the synthesized texture A (x) is converted from the average shape s ₀ to the shape s (inverse conversion of the warp W shown in FIG. 7). Thus, it is possible to reproduce the shape and texture of any face image.

A-3. Facial feature location processing:
FIG. 9 is a flowchart showing the flow of the face feature position specifying process in the present embodiment. The face feature position specifying process in the present embodiment is a process for specifying the position of the feature part of the face in the target image by determining the arrangement of the feature points CP in the target image using AAM. As described above, in the present embodiment, in the AAM setting process (FIG. 2), a total of 68 predetermined positions in the human facial organs (eyebrows, eyes, nose, mouth) and facial contour are set as the characteristic parts. (See FIG. 4). Therefore, in the face feature position specifying process according to the present embodiment, the arrangement of 68 feature points CP indicating predetermined positions in the human face organ and face contour is determined.

  When the arrangement of the feature points CP in the target image is determined by the face feature position specifying process, it is possible to specify the shape / position of the human face organ and the face contour shape in the target image. Therefore, the processing result of the facial feature position specifying process is a facial expression determination for detecting a facial image of a specific facial expression (for example, a face with a smile or eyes closed) or a facial image in a specific direction (for example, rightward or downward). It can be used for face orientation estimation for detection, face deformation for deforming the face shape, and the like.

  In step S210 (FIG. 9), the image processing unit 200 (FIG. 1) acquires image data representing a target image that is a target of the face feature position specifying process. In the printer 100 of the present embodiment, when the memory card MC is inserted into the card slot 172, thumbnail images of image files stored in the memory card MC are displayed on the display unit 150. The user selects one or a plurality of images to be processed via the operation unit 140 while referring to the displayed thumbnail images. The image processing unit 200 acquires an image file including image data corresponding to one or more selected images from the memory card MC and stores it in a predetermined area of the internal memory 120. Note that the acquired image data is referred to as target image data, and an image represented by the target image data is referred to as a target image OI.

  In step S220 (FIG. 9), the face area detection unit 230 (FIG. 1) detects an image area including at least a part of the face image in the target image OI as the face area FA. The detection of the face area FA can be performed using a known face detection method. Known face detection methods include, for example, pattern matching methods, skin color region extraction methods, learning using sample face images (for example, learning using neural networks, learning using boosting, and support vector machines). There is a method using learning data set by learning.

  FIG. 10 is an explanatory diagram illustrating an example of the detection result of the face area FA in the target image OI. FIG. 10 shows the face area FA detected in the target image OI. In the present embodiment, a face detection method is used in which a rectangular area including the vertical direction of the face from the forehead to the chin and the horizontal direction from the outside of both ears is detected as the face area FA.

The assumed reference area ABA shown in FIG. 10 is an area assumed to correspond to the reference area BA (see FIG. 8), which is the entire area of the average face image A ₀ (x). The assumed reference area ABA is set based on the detected face area FA as an area having a predetermined relationship with the face area FA with respect to each of size, inclination, vertical and horizontal positions. The predetermined relationship between the face area FA and the assumed reference area ABA is such that when the face represented in the face area FA is an average face, the assumed reference area ABA corresponds to the reference area BA. It is set in advance in consideration of the characteristics of the face detection method employed for detection of the face area FA (what face range is detected as the face area FA).

  If the face area FA is not detected from the target image OI in step S220 (FIG. 9), the face feature position specifying process is terminated because the target image OI does not include a face image, or The face area FA detection process is executed again.

  In step S230 (FIG. 9), the face feature position specifying unit 210 (FIG. 1) determines the initial arrangement of the feature points CP in the target image OI. FIG. 11 is a flowchart showing the flow of the initial arrangement determining process of the feature point CP in the present embodiment. In step S310 of the feature point CP initial arrangement determination process, the initial arrangement unit 211 (FIG. 1) changes various values of size, inclination, and position (vertical position and horizontal position) as global parameters. Then, a temporary arrangement of the feature points CP is set on the target image OI.

FIG. 12 is an explanatory diagram showing an example of the temporary arrangement of the feature points CP in the target image OI. 12A and 12B, the temporary arrangement of the feature points CP in the target image OI is shown by a mesh. That is, each intersection of the mesh is a feature point CP. As shown in the center of FIGS. 12A and 12B, the initial placement unit 211 has an average face image A ₀ (x) (see FIG. 8) in the assumed reference area ABA (see FIG. 10) of the target image OI. ) Are set, a temporary arrangement specified by the feature point CP of the average face image A ₀ (x) (hereinafter also referred to as “reference temporary arrangement”) is set.

  The initial arrangement unit 211 also sets temporary arrangements in which global parameter values are variously changed with respect to the reference temporary arrangement. Changing global parameters (size, inclination, vertical position and horizontal position) is to enlarge / reduce, change inclination, or move parallel to the mesh that specifies the temporary arrangement of feature points CP. It corresponds to. Accordingly, as shown in FIG. 12A, the initial placement unit 211 indicates a temporary placement (shown below and above the reference temporary placement) specified by a mesh enlarged or reduced at a predetermined magnification with respect to the mesh of the reference temporary placement. ) Or a temporary arrangement (shown on the right and left of the reference temporary arrangement) specified by a mesh whose inclination is changed clockwise or counterclockwise by a predetermined angle. In addition, the initial placement unit 211 specifies a temporary placement (upper left, lower left, upper right, lower right of the reference temporary placement) that is obtained by performing a transformation that combines enlargement / reduction and inclination change on the mesh of the reference temporary placement. Also set).

  Also, as shown in FIG. 12 (b), the initial placement unit 211 sets the temporary placement (above and below the reference temporary placement) specified by a mesh that has been moved upward or downward by a predetermined amount from the reference temporary placement mesh. Or a temporary arrangement (shown on the left and right of the reference temporary arrangement) specified by the mesh moved in parallel to the left or right. In addition, the initial placement unit 211 performs provisional placement (upper left, lower left, upper right, lower right of the reference temporary placement) that is specified by a mesh obtained by performing a combination of vertical and horizontal parallel movements on the mesh of the reference temporary placement. Also set).

  Further, the initial placement unit 211 is identified by a mesh in which the vertical movement shown in FIG. 12B is performed on each of the eight temporary placements other than the reference temporary placement shown in FIG. The temporary arrangement to be performed is also set. Therefore, in this embodiment, the combination of the three levels of the reference temporary arrangement and each of the four global parameters (size, inclination, vertical position, horizontal position) with respect to the mesh in the reference temporary arrangement. A total of 81 types of temporary arrangements are set, including 80 types of temporary arrangements set by performing a total of 80 types (= 3 × 3 × 3 × 3-1) of conversions corresponding to.

In this embodiment, the correspondence between the average face image A ₀ (x) in the reference temporary arrangement and the assumed reference area ABA of the target image OI is referred to as “reference correspondence”. Setting temporary arrangement includes a mean reference relationship as reference face image A ₀ (x) average face image A ₀ after a total of 80 kinds of the above conversion for one of the target image OI is performed and (x) A correspondence relationship with the target image OI (hereinafter also referred to as “conversion correspondence relationship”) is set, and the arrangement of the feature points CP of the average face image A ₀ (x) in the reference correspondence relationship and the conversion correspondence relationship is a feature in the target image OI. It can be expressed as realized by provisional arrangement of points CP.

In step S320 (FIG. 11), the image conversion unit 212 (FIG. 1) calculates an average shape image I (W (x; p)) corresponding to each set temporary arrangement. FIG. 13 is an explanatory diagram showing an example of the average shape image I (W (x; p)). The average shape image I (W (x; p)) is a face image having the average shape s ₀ . The average shape image I (W (x; p)) is calculated by conversion such that the arrangement of the feature points CP in the input image is equal to the arrangement of the feature points CP in the average shape s ₀ .

The transformation for calculating the average shape image I (W (x; p)) is a warp that is a set of affine transformations for each triangular area TA, similarly to the transformation for calculating the sample face image SIw (see FIG. 7). W. Specifically, the average shape region BSA (region surrounded by the feature points CP located on the outer periphery, see FIG. 6) in the target image OI is specified by the feature points CP (see FIG. 12) arranged in the target image OI. The average shape image I (W (x; p)) is calculated by performing affine transformation for each triangular area TA on the average shape area BSA. In this embodiment, the average shape image I (W (x; p) ) is composed of an average face image A ₀ (x) and an average shape area BSA and a mask area MA, average face image A ₀ (x) And is calculated as an image of the same size. FIG. 13 shows an example of nine average shape images I (W (x; p)) corresponding to the nine temporary arrangements shown in FIG.

As described above, the pixel group x is a set of pixels located in the average shape region BSA in the average shape s ₀ . The pixel group in the image (average shape area BSA of the target image OI) before execution of the warp W corresponding to the pixel group x in the image after execution of the warp W (face image having the average shape s ₀ ) is expressed as W (x; p). To express. Since the average shape image is an image composed of luminance values in each of the pixel groups W (x; p) in the average shape region BSA of the target image OI, it is expressed as I (W (x; p)).

In step S330 (FIG. 11), the initial arrangement unit 211 (FIG. 1) calculates a difference image Ie between each average shape image I (W (x; p)) and the average face image A ₀ (x). Since 81 types of temporary arrangements of feature points CP are set and 81 average shape images I (W (x; p)) are set, the initial arrangement unit 211 calculates 81 difference images Ie. .

In step S340 (FIG. 11), the initial placement unit 211 (FIG. 1) calculates the norm of each difference image Ie, and the provisional placement corresponding to the difference image Ie having the smallest norm value (hereinafter “norm minimum provisional placement”). Is also set as the initial arrangement of the feature points CP in the target image OI. The norm minimum provisional arrangement is a provisional arrangement corresponding to the average shape image I (W (x; p)) having the smallest degree of difference from the average face image A ₀ (x) (closest, most similar). . Note that selecting the norm minimum provisional arrangement means that the average shape image I (W (x; p)) after normalization processing and the average face image A are selected from the above-described reference correspondence relationship and 80 types of conversion correspondence relationships. _This is synonymous with selecting a correspondence relationship having the smallest difference from ₀ (x) and selecting a provisional arrangement in the selected correspondence relationship. By the initial arrangement processing of the feature points CP, approximate values of global parameters that define the overall size, inclination, and position (vertical position and horizontal position) of the characteristic points CP are set in the target image OI. It will be done.

  FIG. 14 is an explanatory diagram showing an example of the initial arrangement of the feature points CP in the target image OI. In FIG. 14, the initial arrangement of the feature points CP determined in the target image OI is represented by a mesh. That is, each intersection of the mesh is a feature point CP.

  When the feature point CP initial arrangement determination process (step S230 in FIG. 9) is completed, the face feature position specifying unit 210 (FIG. 1) updates the arrangement of the feature points CP in the target image OI (step S240). FIG. 15 is a flowchart showing the flow of the feature point CP arrangement update process in the present embodiment.

In step S410 of the feature point CP arrangement update process (FIG. 15), the image conversion unit 212 (FIG. 1) calculates an average shape image I (W (x; p)) from the target image OI. The average shape image I (W (x; p)) is a face image having the average shape s ₀ . The average shape image I (W (x; p)) is calculated by conversion such that the arrangement of the feature points CP in the input image is equal to the arrangement of the feature points CP in the average shape s ₀ (see FIG. 6).

The transformation for calculating the average shape image I (W (x; p)) is a warp that is a set of affine transformations for each triangular area TA, similarly to the transformation for calculating the sample face image SIw (see FIG. 7). W. Specifically, the average shape region BSA (region surrounded by the feature points CP located on the outer periphery, see FIG. 6) in the target image OI is specified by the feature points CP (see FIG. 14) arranged in the target image OI. The average shape image I (W (x; p)) is calculated by performing affine transformation for each triangular area TA on the average shape area BSA. In this embodiment, the average shape image I (W (x; p) ) is composed of an average face image A ₀ (x) and an average shape area BSA and a mask area MA, average face image A ₀ (x) And is calculated as an image of the same size.

In step S412 (FIG. 15), the normalization unit 215 (FIG. 1) refers to the index value representing the luminance value distribution of the average face image A ₀ (x), and calculates the average shape image I (W (x; p). ) Is normalized. In this embodiment, the AAM information AMI includes information indicating an average value and a variance value as index values representing the luminance value distribution in the average shape area BSA (see FIG. 8) of the average face image A ₀ (x). Yes. The normalization unit 215 calculates the average value and the variance value of the luminance values in the average shape region BSA of the average shape image I (W (x; p)), and the calculated average value and variance value are the average face image A _0. Image conversion (normalization processing) is performed on the average shape area BSA of the average shape image I (W (x; p)) so as to be equal to the average value and the variance value of the luminance values of (x).

In step S420 (FIG. 15), the face feature position specifying unit 210 (FIG. 1) calculates a difference image between the average shape image I (W (x; p)) after normalization and the average face image A ₀ (x). Ie is calculated. In step S430, the determination unit 213 (FIG. 1) determines whether or not the feature point CP arrangement update processing has converged based on the difference image Ie. The determination unit 213 calculates the norm of the difference image Ie, determines that it has converged when the norm value is smaller than a preset threshold value, and has not yet converged when the norm value is greater than or equal to the threshold value. judge. The norm of the difference image Ie is an index value that represents the degree of difference between the average shape image I (W (x; p)) and the average face image A ₀ (x).

  In the convergence determination in step S430, the determination unit 213 determines that the convergence has occurred when the calculated norm value of the difference image Ie is smaller than the value calculated in the previous step S430, and is greater than or equal to the previous value. In some cases, it may be determined that it has not yet converged. Or the determination part 213 is good also as what performs determination of convergence combining the determination by a threshold value, and the determination by comparison with the last time value. For example, the determination unit 213 determines that the calculated norm value has converged only when the calculated norm value is smaller than the threshold and smaller than the previous value, and otherwise determines that it has not yet converged. Also good.

If it is determined in step S430 that the convergence has not yet been completed, the updating unit 214 (FIG. 1) calculates a parameter update amount ΔP (step S440). The parameter update amount ΔP means the amount of change in the values of the four global parameters (total size, inclination, X direction position, Y direction position) and n shape parameters p _i (see equation (1)). is doing. Immediately after the initial arrangement of the feature point CP, the global parameter is set to the value determined in the feature point CP initial arrangement determination process (FIG. 11). In addition, since the difference between the initial arrangement of the feature points CP and the arrangement of the feature points CP of the average shape s ₀ at this time is limited to the difference in overall size, inclination, and position, the shape parameter p _i in the shape model. The values of are all zero.

  The parameter update amount ΔP is calculated by the following equation (3). That is, the parameter update amount ΔP is a product of the update matrix R and the difference image Ie.

The update matrix R in the expression (3) is a matrix of M rows and N columns set in advance by learning in order to calculate the parameter update amount ΔP based on the difference image Ie, and the internal memory 120 as the AAM information AMI (FIG. 1). Stored in In this embodiment, the number M of rows of the update matrix R is equal to the sum ((4 + n)) of the number of global parameters (4) and the number of shape parameters p _i (n), and the number of columns N is It is equal to the number of pixels in the average shape area BSA of the average face image A ₀ (x) (FIG. 8). The update matrix R is calculated by the following formulas (4) and (5).

In step S450 (FIG. 15), the updating unit 214 (FIG. 1) updates the parameters (four global parameters and n shape parameters p _i ) based on the calculated parameter update amount ΔP. Thereby, the arrangement of the feature points CP in the target image OI is updated. After the parameter update in step S450, the average shape image I (W (x; p)) is calculated again from the target image OI whose arrangement of the feature points CP has been updated (step S410), and the difference image Ie is calculated ( In step S420), a convergence determination (step S430) based on the difference image Ie is performed. If it is determined that the convergence has not occurred in the convergence determination again, the parameter update amount ΔP based on the difference image Ie is calculated (step S440), and the feature point CP is updated by updating the parameters (step S450). Done.

When the processing from step S410 to step S450 in FIG. 15 is repeatedly executed, the position of the feature point CP corresponding to each feature part in the target image OI approaches the position of the actual feature part (correct position) as a whole, At a certain point in time, it is determined that convergence has occurred in the convergence determination (step S430). If it is determined in the convergence determination that convergence has been achieved, the face feature position specifying process is completed (step S460). The arrangement of the feature points CP specified by the global parameter and the shape parameter p _i set at this time is determined as the arrangement of the feature points CP in the final target image OI.

  FIG. 16 is an explanatory diagram illustrating an example of a result of the face feature position specifying process. FIG. 16 shows the arrangement of the feature points CP finally determined in the target image OI. By the arrangement of the feature points CP, the positions of the characteristic portions (predetermined positions in the human face organs (eyebrows, eyes, nose, mouth) and face contour) in the target image OI are specified, and the face of the person in the target image OI is identified. It is possible to specify the shape / position of the organ and the contour shape of the face.

As described above, in the facial feature position specifying process (FIG. 9) of the present embodiment, the initial arrangement of the feature points CP in the target image OI is determined, and then the average shape image I (W The arrangement of the feature points CP in the target image OI is updated based on the comparison result between (x; p)) and the average face image A ₀ (x). That is, in the initial arrangement determination process of the feature point CP (FIG. 11), the global parameter approximate value that defines the overall size, inclination, and position (vertical position and horizontal position) of the characteristic point CP. In the subsequent feature point CP arrangement update process (FIG. 15), the arrangement of the feature points CP is updated in accordance with the parameter update based on the difference image Ie, and the final arrangement of the feature points CP in the target image OI is determined. It is determined. As described above, in this embodiment, first, by determining the approximate value of the global parameter in which the variation of the entire arrangement of the feature points CP is large (the variance is large) in the initial arrangement determination process, Speeding up and accuracy improvement (final determination of feature point CP arrangement based on a global optimal solution rather than a so-called local optimal solution) can be realized.

Further, in the feature point CP arrangement update process (FIG. 15) of the present embodiment, a difference image between the average shape image I (W (x; p)) calculated from the target image OI and the average face image A ₀ (x). Prior to the calculation of Ie (step S420 in FIG. 15), the luminance between the average shape area BSA of the average shape image I (W (x; p)) and the average shape area BSA of the average face image A ₀ (x) Image conversion (normalization processing) is performed on the average shape image I (W (x; p)) so that the average value and the variance value of the values are equal to each other (step S412). As a result, the influence of the feature of the luminance value distribution of each target image OI on the difference image Ie is suppressed, the accuracy of convergence determination (step S430) based on the difference image Ie is improved, and consequently the accuracy of the face feature position specifying process Will improve. In the convergence determination, as described above, highly accurate determination can be performed by determination using an absolute threshold. Therefore, for example, the processing speed can be increased compared to the case where the convergence determination is made by comparing the norm value of the difference image Ie with the previous value.

A-4. Specific image processing:
FIG. 17 is a flowchart showing the flow of specific image processing in the present embodiment. The specific image processing in the present embodiment is a process of performing the specific image processing of the type selected by the user on the target image OI in which the arrangement of the feature points CP is specified by the face feature position specifying process (FIG. 9) described above. It is.

  In step S610 (FIG. 17), the specific image processing unit 240 (FIG. 1) sets the type of specific image processing to be performed on the target image OI. In step S620, the specific image processing unit 240 refers to the reference face direction information SDI and sets a target face direction corresponding to the set type of specific image processing.

  FIG. 18 is an explanatory diagram showing an example of the content of the reference face orientation information SDI. In this embodiment, a plurality of specific image processing types that can be executed by the specific image processing unit 240 of the printer 100 are set. Specifically, the specific image processing unit 240 can execute an eye enlargement process, an eye balance adjustment process, a facial expression change process, a makeup simulation process, and a shaping simulation process. The eye enlargement process in the present embodiment is a process for enlarging the image portion (eye area) of the eye in the target image OI. Specifically, for each of the left and right eyes, six feature points CP indicating the eye contours This is a process of moving the image (see FIG. 4) and performing image deformation such that the area (eye area) defined by the six feature points CP is enlarged. The eye enlargement process will be described in detail later. The eye balance adjustment process is a process for adjusting (enlarging or reducing) the size of the left and right eye image portions (eye regions) in the target image OI so that the contours of the eyes are similar to the eye enlargement process. This is a process of performing image transformation such that the eye area is enlarged or reduced by moving the six feature points CP shown. The facial expression changing process is a process of changing the facial expression in the target image OI to a predetermined facial expression (for example, a smile). For example, the image transformation that moves the feature point CP indicating the outline of the mouth and raises the mouth corner is performed. It is a process to perform. The makeup simulation process is a process of generating a predicted image as a result of applying a predetermined makeup (for example, makeup that applies lipstick or blush) to the face in the target image OI. Is a process of changing the value of the pixel corresponding to 1 to a value according to the makeup mode. The shaping simulation process is a process of generating a predicted image as a result of performing a predetermined shaping (for example, shaping the nose muscles or making the chin narrower) on the face in the target image OI. Move the feature point CP to deform the image so that the nose passes, or move the feature point CP indicating the chin line to deform the image so that the chin becomes thin (so-called small face) It is processing to do.

  In the present embodiment, a reference face orientation is set for each of a plurality of specific image processing types. The reference face orientation is a reference face orientation set to realize effective image processing for each type of specific image processing. For example, as shown in FIG. 18, in the eye enlargement process, it is considered that the enlargement rate and the enlargement direction of the eye area from which a natural image processing result is obtained differ depending on the inclination of the face in the horizontal direction and the vertical direction. It is done. For this reason, in this embodiment, the “front direction” having no horizontal tilt or vertical tilt is set as the reference face direction for the eye enlargement process. Similarly, in the eye balance adjustment process, the expression change process, and the makeup simulation process for applying lipstick, the reference face direction is set to “front direction”.

  Further, as shown in FIG. 18, the makeup simulation process for applying blusher and the shaping simulation process for narrowing the chin are considered to be face orientations suitable for setting the method and degree of simulating makeup and shaping. The face orientation of “45 ° orientation” or “45 ° right” is set as the reference face orientation. In addition, for the shaping simulation process through the nose, the face orientation “75 degrees left” or “75 degrees right”, which is considered to be a suitable face orientation for setting the method and degree of shaping, It is set as the face orientation.

  The specific image processing unit 240 causes the display processing unit 310 to display a menu screen indicating the type of executable specific image processing on the display unit 150 (FIG. 1), and the specific image selected by the user via the operation unit 140. The type of processing is set as the type of specific image processing to be performed on the target image OI (step S610 in FIG. 17). Further, the specific image processing unit 240 sets the reference face direction associated with the set specific image processing type as the target face direction in the reference face direction information SDI (step S620). In the following, a case will be mainly described where the specific image processing type to be executed by the eye enlargement process is set, and the “front direction” that is the reference face direction associated with the eye enlargement process is set as the target face direction. I do.

  In step S630 (FIG. 17), the specific image processing unit 240 (FIG. 1) performs an approximate determination between the face orientation and the target face orientation in the target image OI. The approximation determination is performed in order to determine whether or not to execute a face orientation change and face orientation restoration process (steps S640 and S660) described later. That is, in the present embodiment, the face orientation changing / restoring process is not executed when it is determined that the face orientation in the target image OI is close to the target face orientation to some extent, and the face orientation in the target image OI is determined from the target face orientation. It is executed only when it is determined that it is some distance away (not close).

Specifically, the specific image processing unit 240 determines the face among the shape parameters p _i in the shape s (see the above formula (1)) of the target image OI specified in the face feature position specifying process (FIG. 9) described above. The target image is based on the value of the shape parameter p _i of the first shape vector s ₁ that is substantially correlated with the horizontal swing of the face and the value of the shape parameter p ₂ of the second shape vector s ₂ that is substantially correlated with the vertical swing of the face. Estimate the face orientation in OI. In the present embodiment, the correspondence between the value of the shape parameter p _i of the first shape vector s ₁ and the angle of the left / right swing of the face, the value of the shape parameter p ₂ of the second shape vector s ₂ and the up / down swing of the face The correspondence with the angle is set in advance. Since the average face image A ₀ (x) (FIG. 8) set in the AAM setting process (FIG. 2) is a front-facing face image, the values of the shape parameter p _i and the shape parameter p ₂ are zero. It is estimated that the face orientation of the target image OI is “front-facing” with no left-right swing and no up-down swing. Also, it is estimated that the larger the absolute value of the shape parameter p _i is, the larger the angle of swinging the face of the target image OI is, and the larger the absolute value of the shape parameter p ₂ is, the larger the angle of swinging the face of the target image OI is. It is estimated to be. At this time, the specific image processing unit 240 functions as a face orientation estimating unit in the present invention.

  Next, the specific image processing unit 240 determines that the face orientation of the target image OI is the target face orientation when the swing of the face orientation of the target image OI with respect to the target face orientation is within 5 degrees in all the left, right, top, and bottom directions. Judge as close. On the other hand, the specific image processing unit 240 determines that the face orientation of the target image OI is determined from the target face orientation when the swing of the face orientation of the target image OI with respect to the target face orientation exceeds 5 degrees in at least one of the left, right, up and down directions. It is determined that they are separated (not near).

When it is determined that the face orientation of the target image OI is away from the target face orientation (step S630: No), the face orientation changing unit 250 (FIG. 1) sets the face orientation in the target image OI to the target face orientation. A face orientation changing process is performed so as to be close to each other (step S640). Specifically, the face orientation changing unit 250 determines the value of the shape parameter p _i of the first shape vector s _{1 and} the shape parameter of the second shape vector s _{2 in} the shape s (see the above formula (1)) of the target image OI. and the value of p _2, changed to a value corresponding to the target face orientation. For example, when the eye enlargement process is set as the type of the specific image process to be executed in step S610, the target face direction is set to “front direction” (see FIG. 18), so the value of the shape parameter p _i And the value of the shape parameter p ₂ of the second shape vector s ₂ are both changed to zero, which is a value corresponding to “front direction”. The coordinates of each feature point CP of the target image OI when the face direction in the target image OI is changed to the target face direction are specified by the shape s after the parameter value is changed. In the face orientation changing process in the present embodiment, the target image OI (image having a texture) whose face orientation is changed is not generated, but each feature point of the target image OI when the face orientation is changed. Only the coordinates after the CP change are specified.

  In step S650 (FIG. 17), the specific image processing unit 240 (FIG. 1) performs specific image processing on the target image OI after the face orientation change processing. For example, when the eye enlargement process is set as the type of the specific image process to be executed in step S610, the eye area is enlarged by changing the position of the feature point CP corresponding to the outline of the eye.

  FIG. 19 is an explanatory diagram showing a method of enlarging the eye area. In each of FIGS. 19A to 19C, for each of the left eye and the right eye, there are six feature points CP indicating the outline of the eye, and center points EC (EC (r) and EC (6) of the six feature points CP. l)). In the eye enlargement process in the present embodiment, as shown in FIG. 19A, the point at the tip of the enlarged vector obtained by enlarging the vector from the center point EC to each of the six feature points CP at a constant enlargement ratio is changed. Calculated as a feature point CPc. An area partitioned by the calculated six changed feature points CPc (an area surrounded by a broken line) is an eye area after the eye enlargement process.

  In the eye enlargement process, as shown in FIG. 19B, the positions of the feature points CP (CP (i)) corresponding to the eyes are not changed for each of the left eye and the right eye, and the remaining five feature points. The position of only the CP may be changed. In addition, as shown in FIG. 19C, the positions of the feature point CP (CP (i)) corresponding to the top of the eye and the feature point CP (CP (o)) corresponding to the corner of the eye are changed for each of the left eye and the right eye. Instead, the positions of only the remaining four feature points CP may be changed. By executing the eye enlargement process according to the mode shown in FIGS. 19B and 19C, it is possible to suppress the processing result from becoming unnatural due to, for example, changing the position of the black eye. . Further, in any of the cases shown in FIGS. 19A to 19C, the vector from the center point EC to each of the six feature points CP is individually set for each feature point CP (that is, each vector). The point at the tip of the enlarged vector enlarged at a rate may be calculated as the changed feature point CPc. By doing so, a more natural and preferable eye enlargement process can be realized.

  Note that in step S650 (FIG. 17), the process of actually obtaining the pixel values for the entire eye area of the target image OI is not executed, and the coordinate calculation of the changed feature point CPc and the coordinates of the changed feature point CPc are performed. The reflected shape s is calculated. The calculation of the coordinates of the changed feature point CPc and the calculation of the shape s reflecting the coordinates of the changed feature point CPc are (part of) specific image processing.

In step S660 (FIG. 17), the face orientation restoring unit 260 (FIG. 1) performs a face orientation restoring process on the target image OI. Specifically, the face orientation restoring unit 260 reflects the value of the shape parameter p _i of the first shape vector s _{1 and} the shape of the second shape vector s ₂ in the shape s of the target image OI reflecting the coordinates of the changed feature point CPc. the value of the parameter p _2, to restore to a value corresponding to the original face direction of the target image OI. The coordinates of each feature point CP including the changed feature point CPc in the initial face direction of the target image OI is specified by the shape s after the parameter value restoration. For example, when eye enlargement processing is set as the type of specific image processing to be executed in step S610, the face orientation is the initial face orientation, and the result of changing the position of the feature point CP corresponding to the eyes is reflected. The coordinates of each feature point CP in the target image OI are specified.

  On the other hand, if it is determined in step S630 (FIG. 17) that the face orientation of the target image OI is close to the target face orientation (step S630: Yes), the face orientation changing process (step S640) is not executed and the target Specific image processing (change of the position of the feature point CP corresponding to the eye contour) for the image OI is executed (step S670). The processing content of step S670 is the same as the processing content of step S650. Further, when the process of step S670 is completed, naturally the face orientation restoring process (step S660) is not executed, and the process proceeds to step S680.

  In step S680 (FIG. 17), the specific image processing unit 240 (FIG. 1) generates a post-image processing target image OIc. The target image OIc after image processing is the target image OI after execution of specific image processing. For example, if eye enlargement processing is set as the type of specific image processing to be executed in step S610, the feature points CP (including changed feature points CPc) after execution of specific image processing are processed by the processing before step S670. The location is specified. Accordingly, in step S680, an eye region (post-processing eye region) defined by the position of the feature point CP corresponding to the eye after execution of the specific image processing is specified in the original target image OI, and the original target image OI The image of the eye area (pre-processing eye image) specified by the feature point CP corresponding to the eye is converted into an image that matches the post-processing eye area. This conversion is executed by affine transformation for each triangular area TA (see FIG. 6).

  Note that, when generating the post-image processing target image OIc, affine transformation may be performed so that the image of the triangular area TA surrounding the eye area in the original target image OI is reduced by the amount of enlargement of the eye area. . In this way, in the eye enlargement process, the image of the triangular area TA surrounding the eye area is not overwritten by the image of the enlarged eye area.

  As described above, in the specific image processing of the present embodiment, the face orientation in the target image OI is estimated, and when it is determined that the estimated face orientation is away (not close) from the target face orientation, Specific image processing (for example, the coordinate calculation of the change feature point CPc and the coordinates of the change feature point CPc related to the eye enlargement processing are performed on the target image OI after the face orientation change processing is performed so that the face orientation approaches the target face orientation. The calculation of the reflected shape s) is executed, and then the face orientation restoring process for restoring the face orientation is executed. Since the target face orientation is set based on the reference face orientation set to realize effective image processing for each type of specific image processing, in this embodiment, the face orientation of the original target image OI is Effective specific image processing can be realized regardless of the face orientation. For example, good results can be obtained even if image processing that is supposed to be performed on a face image facing the front is performed on a face image facing any face. Further, in this embodiment, in order to obtain a preferable specific image processing result, it is not necessary to set the specific image processing mode (for example, the eye region enlargement method and enlargement degree shown in FIG. 19) for each face direction. Since only one reference face direction needs to be set, effective specific image processing can be easily realized.

  In this embodiment, the reference face direction is set for each type of specific image processing, and the reference face direction is set to the target face direction. Therefore, various types of specific image processing can be effectively and easily realized. be able to. In the present embodiment, when it is determined that the face orientation in the estimated target image OI is close to the target face orientation, the face orientation changing process and the face orientation restoring process are not performed, and the original target image is executed. Since specific image processing is executed for OI, efficient processing can be realized.

B. Variation:
The present invention is not limited to the above-described examples and embodiments, and can be implemented in various modes without departing from the gist thereof. For example, the following modifications are possible.

B1. Modification 1:
In the above embodiment, in the approximate determination of the face orientation and the target face orientation in the estimated target image OI (step S630 in FIG. 17), the face orientation in the target image OI is far from (not close to) the target face orientation. It is assumed that the face orientation changing process and the face orientation restoring process are executed only when determined, but the approximate judgment is omitted and the face orientation changing process and the face orientation restoring process are executed in all cases. Good.

  Further, the threshold value in the approximation determination of the above embodiment is merely an example, and the threshold value can be arbitrarily changed. In the approximate determination, if the face orientation in the target image OI is considerably away from the target face orientation, the target image OI is not appropriate as the target of the set specific image processing type, and the process is stopped. It may be done.

  In the above-described embodiment, the reference face orientation associated with the type of specific image processing is set as the target face orientation in the reference face orientation information SDI (FIG. 18), but it does not depend on the type of specific image processing. The predetermined face direction (for example, the front direction) may be uniformly set as the target face direction. Further, the content of the reference face orientation information SDI shown in FIG. 18 is merely an example, and the correspondence between the type of specific image processing and the reference face orientation can be changed. Further, the specific image processing unit 240 need not be able to execute all the types of specific image processing shown in FIG. 18, and may be able to execute specific image processing types other than those shown in FIG. 18. For example, the specific image processing unit 240 sets a deformation area based on the face area FA (see FIG. 10), divides the deformation area into a plurality of small areas having the dividing point as a vertex, and moves the dividing point to move the small area. It may be possible to execute image processing for performing image deformation of the deformation area (for example, image deformation to make a small face) by deforming. Also in this case, if the face direction is estimated and the image process is executed after the face direction is changed to, for example, the front direction, the image process for performing the image deformation can be effectively realized.

In the above embodiment, the face orientation changing process and the face orientation restoration process, the value of the first shape vector s ₁ of shape parameter p _i to be approximately correlated with the left and right swing in shape s of the target image OI, substantially vertical appearance This is executed by changing the value of the shape parameter p ₂ of the correlated second shape vector s ₂ , but only one of the value of the shape parameter p _{i and} the value of the shape parameter p ₂ is changed. It may be executed depending on the situation. That is, the face orientation changing process and the face orientation restoring process may be executed only in one of the left and right directions and the up and down direction. Further, the face orientation changing process may be performed so that the face orientation in the target image OI approaches the target face orientation and does not necessarily match the target face orientation.

  Further, the estimation of the face orientation, the face orientation changing process, and the face orientation restoring process in the target image OI are not necessarily performed using the AAM shape model, and may be performed by other methods.

  The eye enlargement method shown in FIG. 19 is merely an example, and the eye enlargement process may be performed by another method. Further, when eye enlargement or eye balance adjustment is set as the type of specific image processing, the content of the specific image processing may be changed based on the size of the eye area in the target image OI. For example, when eye enlargement is set as the type of specific image processing, if the size of the eye area of the target image OI is larger than a predetermined threshold, execution of eye enlargement as specific image processing may be stopped. It is also possible to adjust the enlargement ratio of the eye enlargement to a smaller value. Further, when eye balance adjustment is set as the type of specific image processing, if the size of the eye area of the target image OI is smaller than a predetermined threshold, execution of eye balance adjustment as specific image processing is stopped. It is good.

B2. Modification 2:
In the above embodiment, the facial feature position specifying process (FIG. 9) is executed using AAM. However, the facial feature position specifying process is not necessarily executed using AAM. It may be executed.

  Further, although the normalization process (step S412) is executed in the feature point CP arrangement update process (FIG. 15), the normalization process is not necessarily executed.

B3. Modification 3:
In the above embodiment, in the feature point CP initial arrangement determination process (step S230 in FIG. 9), the difference image Ie between each of the average face image groups and the target image OI, the average face image A ₀ (x), and a plurality of averages A difference image Ie from each of the shape images I (W (x; p)) is calculated, and based on the difference image Ie, the overall value of the global parameter having a large variation (large variance) in the entire arrangement of the flat feature points CP is obtained. Although it is determined that the initial arrangement of the feature points CP in the target image OI is not necessarily calculated, it is not always necessary to calculate the difference image Ie or determine the global parameter approximate value. The arrangement of the points CP (for example, the arrangement in the above-described reference correspondence relationship) may be determined as the initial arrangement.

B4. Modification 4:
In the above embodiment, the average shape image I (W (x; p)) and the average face image A ₀ (x) are used as determination index values in the convergence determination (step S430) of the feature point CP arrangement update process (FIG. 15). The norm of the difference image Ie is used, but other index values representing the degree of difference between the average shape image I (W (x; p)) and the average face image A ₀ (x) are used as determination index values. May be used.

B5. Modification 5:
In the feature point CP arrangement update process (FIG. 15) of the above embodiment, the average shape image I (W (x; p)) is calculated based on the target image OI, and the arrangement of the feature points CP of the target image OI is averaged. Although matched to the arrangement of the characteristic points CP of the image a ₀ (x), may be configured to be aligned arrangement of the characteristic points CP of the two performs an image conversion for the average face image a ₀ (x) .

B6. Modification 6:
In the above embodiment, the face area FA is detected and the assumed reference area ABA is set based on the face area FA. However, the detection of the face area FA does not necessarily have to be executed. For example, the assumed reference area ABA may be set directly according to the designation by the user.

B7. Modification 7:
The sample face image SI (FIG. 3) in the above embodiment is merely an example, and the number and type of images employed as the sample face image SI can be arbitrarily set. In each of the above embodiments, the predetermined feature portion of the face indicated by the position of the feature point CP (see FIG. 4) is merely an example, and a part of the feature portion set in the embodiment may be omitted, Other parts may be adopted as the characteristic part.

  In the above embodiment, the texture model is set by the principal component analysis for the luminance value vector constituted by the luminance values in each of the pixel groups x of the sample face image SIw, but represents the texture (appearance) of the face image. A texture model may be set by principal component analysis for index values other than luminance values (for example, RGB values).

In the above embodiment, the average face image A ₀ (x) may have various sizes. Further, the average face image A ₀ (x) does not need to include the mask area MA (FIG. 8), and may be configured only by the average shape area BSA. Further, instead of the average face image A ₀ (x), another reference face image set based on the statistical analysis of the sample face image SI may be used.

  In the above embodiment, the shape model and the texture model are set using AAM. However, the shape model and the texture model using other modeling methods (for example, a method called Morphable Model or a method called Active Blob) are used. A texture model may be set.

  In the above embodiment, the image stored in the memory card MC is set as the target image OI. However, the target image OI may be an image acquired via a network, for example.

  Further, the configuration of the printer 100 as the image processing apparatus in the above embodiment is merely an example, and the configuration of the printer 100 can be variously changed. In the above embodiment, image processing by the printer 100 as the image processing apparatus has been described. However, part or all of the processing is executed by another type of image processing apparatus such as a personal computer, a digital still camera, or a digital video camera. It is good also as what is done. The printer 100 is not limited to an ink jet printer, and may be another type of printer such as a laser printer or a sublimation printer.

  In the above embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware.

  In addition, when part or all of the functions of the present invention are realized by software, the software (computer program) can be provided in a form stored in a computer-readable recording medium. In the present invention, the “computer-readable recording medium” is not limited to a portable recording medium such as a flexible disk or a CD-ROM, but an internal storage device in a computer such as various RAMs and ROMs, a hard disk, etc. It also includes an external storage device fixed to the computer.

100 ... Printer 110 ... CPU
DESCRIPTION OF SYMBOLS 120 ... Internal memory 140 ... Operation part 150 ... Display part 160 ... Printer engine 170 ... Card interface 172 ... Card slot 200 ... Image processing part 210 ... Face feature position specification part 211 ... Initial arrangement part 212 ... Image conversion part 213 ... Determination part 214 ... Update unit 215 ... Normalization unit 230 ... Face region detection unit 240 ... Specific image processing unit 250 ... Face orientation change unit 260 ... Face orientation restoration unit 310 ... Display processing unit 320 ... Print processing unit

Claims (12)

An image processing apparatus that performs image processing on a face image,
A face orientation estimating unit for estimating a face orientation in the face image;
A face orientation changing unit that performs a face orientation changing process so that the estimated face orientation approaches the target face orientation with respect to the face image;
An image processing unit that performs the image processing on the face image after the face orientation change processing;
An image processing apparatus comprising: a face orientation restoring unit that performs a face orientation restoring process for restoring the face orientation changed by the face orientation changing process to the face image after the image processing. The image processing apparatus according to claim 1,
The image processing apparatus, wherein the target face orientation is a front orientation. The image processing apparatus according to claim 1,
The image processing unit can execute a plurality of types of the image processing,
The image processing apparatus further includes:
A storage unit that stores the correspondence between the type of image processing and the face orientation;
The face orientation changing unit is an image processing device that sets, in the correspondence relationship, a face orientation associated with a type of the image processing executed by the image processing unit as the target face orientation. An image processing apparatus according to any one of claims 1 to 3,
The image processing unit performs the image processing on the face image after the face orientation change processing when the estimated face orientation is in a first range relatively far from the target face orientation, When the estimated face orientation is in the second range that is closer to the target face orientation compared to the first range, the image processing is performed on the face image before the face orientation change processing is performed. An image processing apparatus. The image processing apparatus according to any one of claims 1 to 4, further comprising:
Defined by a shape model representing a face shape defined by the position of a predetermined feature part of the face by a reference shape and at least one face shape feature amount including a face orientation, and a pixel value of a face image having the reference shape Using a texture model that represents a face texture by a reference texture and at least one face texture feature amount, and a feature position specifying unit that specifies the position of the feature part in the face image,
The face direction changing unit is an image processing apparatus that executes the face direction changing process by changing the face shape feature amount representing a face direction in the shape model. The image processing apparatus according to claim 5,
The characteristic part includes an eye outline,
The image processing apparatus includes a process of enlarging an eye region defined by an eye contour whose position is specified in the face image. The image processing apparatus according to claim 5,
The characteristic part includes an eye outline,
The image processing apparatus includes a process of bringing the sizes of left and right eye regions defined by the contours of eyes whose positions are specified in the face image closer to each other. An image processing apparatus according to any one of claims 5 to 7,
The shape model and the texture model are set based on a statistical analysis for a plurality of sample face images whose arrangement of feature points indicating the positions of the feature parts is known,
The feature position specifying unit determines an initial arrangement of the feature points in the face image based on an arrangement of the feature points in the reference shape, and a reference defined by the face image, the reference shape, and the reference texture. An image processing apparatus that identifies the position of the feature portion in the face image by updating the arrangement of the feature points based on a comparison result with the face image. The image processing apparatus according to claim 8,
The reference shape is an average shape representing an average position of the characteristic part in the plurality of sample face images,
The image processing apparatus, wherein the reference texture is an average texture that represents an average of pixel values at the positions of the characteristic portions of the plurality of sample face images that have been transformed into the average shape. An image processing method for performing image processing on a face image,
(A) estimating a face orientation in the face image;
(B) performing a face orientation change process on the face image so that the estimated face orientation approaches the target face orientation;
(C) performing the image processing on the face image after the face orientation change processing;
(D) An image processing method comprising: performing a face orientation restoring process for restoring the face orientation changed by the face orientation changing process to the face image after the image processing. A computer program for image processing that performs image processing on a face image,
A face direction estimating function for estimating a face direction in the face image;
A face orientation change function for performing a face orientation change process so that the estimated face orientation approaches the target face orientation with respect to the face image;
An image processing function for performing the image processing on the face image after the face orientation change processing;
The computer program which makes a computer implement | achieve the face orientation restoration function which performs the face orientation restoration process which returns the face orientation changed by the said face orientation change process with respect to the said face image after the said image processing. A printing device,
A face direction estimation unit for estimating a face direction in a face image;
A face orientation changing unit that performs a face orientation changing process so that the estimated face orientation approaches the target face orientation with respect to the face image;
An image processing unit that performs image processing on the face image after the face orientation change processing;
A face orientation restoring unit that performs a face orientation restoring process for restoring the face orientation changed by the face orientation changing process to the face image after the image processing;
And a printing unit that prints the face image after the face orientation restoring process.

Images