JP2010271955A - Image processing apparatus, image processing method, image processing program, and printer - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To discriminate expressions with high accuracy. <P>SOLUTION: In determining the expression of the face included in an image under consideration, an image region including at least part of the face image is detected as a face region from the image under consideration, and a basic face model reproducing at least the position of the feature site of the face on the basis of the plurality of basic feature values and a face model for expression for reproducing at least variation of the position of the feature site corresponding to the expression on the basis of a plurality of feature values for expressions associated with the expressions are set. The basic feature values are corrected so that the position of the feature site reproduced by the basic face model can approximate the face region. The feature values for expression are corrected so that the position of the feature site when the position of the feature site reproduced by applying the corrected basic feature value to the basic face model is varied by the face model for expression can approximate the face region, and the expression of the face is determined on the basis of the corrected feature values for expression. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, an image processing program, and a printing apparatus that determine facial expressions included in an image.

  A printing apparatus that estimates facial expression based on facial feature information such as position, size, and ratio of eyes, nose, mouth, hairstyle, color, etc. is disclosed (see Patent Document 1, paragraph 0031). .

JP 2007-206921 A

  However, if the detection of the position and shape of the facial organ and the determination of the facial expression are performed at the same time, there is a problem that the detection accuracy of the facial organ and the accuracy of the facial expression determination are deteriorated.

  The present invention has been made to solve the above-described problems, and an object thereof is to provide an image processing apparatus, an image processing method, an image processing program, and a printing apparatus capable of accurately discriminating facial expressions. And

  In order to solve at least a part of the problems, the present invention adopts the following aspects. That is, a face area detecting unit is provided, and the face area detecting unit detects an image area including at least a part of the face image as the face area from the attention image. The setting unit includes a basic face model that reproduces at least the position of a facial feature portion based on a plurality of basic feature amounts, and a feature portion corresponding to a facial expression based on a plurality of facial expression feature amounts associated with facial expressions. A facial expression model that reproduces the change in position is set. The basic feature amount correction unit corrects the basic feature amount so that the position of the feature portion reproduced by the basic face model approximates the face region. On the other hand, the feature amount correcting unit for facial expression further changes the position of the characteristic part reproduced by applying the corrected basic feature amount to the basic face model according to the facial expression model. The facial expression feature quantity is corrected so that the position of the part approximates the face area. The determination unit determines the facial expression based on the corrected facial expression feature.

  In such a configuration, first, the basic feature amount correction unit corrects the basic feature amount, whereby the basic position and shape of the feature portion can be specified with high accuracy. Next, the facial expression feature value correction unit corrects the facial expression feature value based on the position of the identified facial feature part. In this way, by specifying the basic arrangement of characteristic parts such as the contours of facial organs in advance, the facial model for facial expression can be used to change the shape and position of facial organs caused by facial expressions. Can be identified. Therefore, the facial expression can be accurately determined based on the corrected facial feature. The basic face model and the facial expression model for expression need only reproduce at least the position of the characteristic part, and also reproduce the state of other faces (for example, texture, etc.) together with the position of the characteristic part. It may be.

  In the image processing device according to another aspect, at least one of the basic feature amount and the facial expression feature amount is an eigenvector of each principal component obtained by principal component analysis of the position of the feature portion of a plurality of sample images. It is a coefficient. That is, a face model is prepared using principal component analysis, and the feature amount is set as a coefficient of an eigenvector of each principal component. Furthermore, the appearance of the face may be modeled by performing principal component analysis on the color values of the pixels of the plurality of sample images. Further, the basic face model is specialized to reproduce the basic arrangement of the characteristic part, and the facial expression model for facial expression preferably specifies a fine variation of the characteristic part due to an expression, It is desirable that the sample image is different between the basic face model and the facial expression model.

  Note that the present invention can be realized in various modes, for example, an image processing method and apparatus, a feature position detection method and apparatus, a facial expression determination method and apparatus, and the functions of these methods or apparatuses. For example, a recording medium storing the image processing program, a data signal including the computer program and embodied in a carrier wave, and the like.

1 is an explanatory diagram schematically illustrating a configuration of a printer 100 as an image processing apparatus according to an embodiment of the present invention. It is a flowchart which shows the flow of the AAM setting process in an Example. It is explanatory drawing which shows an example of sample image SI. It is explanatory drawing which shows an example of the setting method of the feature point CP in the sample image SI. It is explanatory drawing which shows an example of the coordinate of the feature point CP set to sample image SI. It is explanatory drawing which shows an example of average shape s0. It is explanatory drawing which illustrated the relationship between shape vector s _i and shape parameter p _i, and face shape s. It is explanatory drawing which shows an example of the method of the warp W of the sample image SI. It is explanatory drawing which shows an example of average face image _A0 (x). It is a flowchart which shows the flow of the face feature detection process in an Example. It is explanatory drawing which shows an example of the detection result of the face area FA in the attention image OI. It is a flowchart which shows the flow of the initial position setting process of the feature point CP in an Example. It is explanatory drawing which shows an example of the temporary setting position of the feature point CP by changing the value of a global parameter. It is explanatory drawing which shows an example of the average shape image I (W (x; p)). It is a flowchart which shows the flow of the feature point CP setting position correction process in an Example. It is explanatory drawing which shows an example of the result of CP setting position correction processing. It is a flowchart which shows the flow of a texture detection process. It is a flowchart which shows the flow of a facial expression determination process. It is a figure which shows the image for facial expression determination. It is a flowchart which shows the flow of the feature point CP setting position correction process in a facial expression determination process. It is a flowchart which shows the flow of the texture detection process in a facial expression determination process.

  Hereinafter, a printer which is an aspect of an image processing apparatus according to the present invention will be described based on examples with reference to the drawings.

A. Example:
A1. Configuration of image processing device:
FIG. 1 is an explanatory diagram schematically showing a configuration of a printer 100 as an image processing apparatus in the present embodiment of the present invention. The printer 100 of this embodiment is an ink jet color printer that supports so-called direct printing, in which an image is printed based on image data acquired from a memory card MC or the like. The printer 100 includes a CPU 110 that controls each unit of the printer 100, an internal memory 120 configured by a ROM and a RAM, an operation unit 140 configured by buttons and a touch panel, a display unit 150 configured by a liquid crystal display, and printing. A mechanism 160 and a card interface (card I / F) 170 are provided. The printer 100 may further include an interface for performing data communication with other devices (for example, a digital still camera or a personal computer). Each component of the printer 100 is connected via a bus so that bidirectional communication is possible.

  The printing mechanism 160 performs printing based on the print data. The card interface 170 is an interface for exchanging data with the memory card MC inserted into the card slot 172. In this embodiment, an image file including image data is stored in the memory card MC.

  The internal memory 120 stores an image processing unit 200, a display processing unit 310, and a print processing unit 320. The image processing unit 200 is a computer program, and performs facial feature detection processing and facial expression determination processing by being executed by the CPU 110 under a predetermined operating system. The face feature detection process is a process of detecting the position of a predetermined feature part (for example, the corner of the eye, the nose head, or the face line) in the face image. The face feature detection process will be described in detail later. The display processing unit 310 and the print processing unit 320 are also executed by the CPU 110 to realize the respective functions.

  The image processing unit 200 includes a setting unit 210, a feature position detection unit 220, a face area detection unit 230, a texture detection unit 240, and an expression determination unit 250 as program modules. The feature position detection unit 220 and the texture detection unit 240 include generation units 222 and 242, calculation units 224 and 244, and correction units 226 and 246, respectively. The functions of these units will be described in detail in the description of face feature detection processing and facial expression determination processing described later.

  The display processing unit 310 is a display driver that controls the display unit 150 to display processing menus, messages, images, and the like on the display unit 150. The print processing unit 320 is a computer program for generating print data from image data, controlling the printing mechanism 160, and printing an image based on the print data. The CPU 110 reads out and executes these programs (the image processing unit 200, the display processing unit 310, and the print processing unit 320) from the internal memory 120, thereby realizing the functions of these units.

  The internal memory 120 also stores AAM information AMI. The AAM information AMI is information set in advance by an AAM setting process described later, and is referred to in a face feature detection process described later. The contents of the AAM information AMI will be described in detail in the description of AAM setting processing described later.

A2. AAM setting process:
FIG. 2 is a flowchart showing the flow of AAM setting processing in this embodiment. The AAM setting process is a process for setting a shape model, a texture model, an expression shape model, and an expression texture model used for modeling an image called an AAM (Active Appearance Model). In the present embodiment, the AAM setting process is performed by an operator using a computer. The computer has a CPU, a RAM, a ROM, an HDD, a display, an input device, and the like, which are connected by a bus. The CPU reads out a program recorded in the HDD and executes arithmetic processing according to the program, thereby causing the computer to execute each process described later. The computer that sets the shape model, the texture model, the expression shape model, and the expression texture model corresponds to a “setting unit” in the claims. The shape model and the texture model correspond to “basic face model” in the claims, and the shape model for expression and the texture model for expression correspond to the “face model for expression” in the claims.

  First, the operator prepares a plurality of images including a person's face as a sample image SI (step S110). FIG. 3 is an explanatory diagram showing an example of the sample image SI. As shown in FIG. 3, the sample image SI is prepared so as to include different face images with respect to various attributes such as individuality / race / gender / orientation (front, up, down, right, left). The In this embodiment, a sample group G1 constituted by all of these sample images SI, and a sample group G2 including only face images that differ from each other with respect to attributes of facial expressions (joyful face, sad face, surprised face, and angry face) Prepare each. According to the sample group G1, all face images can be accurately modeled by AAM, and accurate face feature detection processing (described later) for all face images can be executed. On the other hand, according to the sample group G2, only the feature of the facial expression can be selectively modeled by the AAM, and it is possible to execute an accurate facial expression determination process (described later). In the sample group G2, the sample image SI is selected so that attributes other than facial expressions such as personality, race, sex, and orientation are as diverse as possible. The sample image SI is also called a learning image.

  Hereinafter, processing performed on the sample images SI belonging to the sample group G1 will be described as an example. A feature point CP is set to the face image included in each sample image SI belonging to the sample group G1 (step S120). FIG. 4 is an explanatory diagram showing an example of a method for setting the feature point CP in the sample image SI. The feature point CP is a point indicating the position of a predetermined feature part in the face image. In this embodiment, as predetermined feature parts, a predetermined position on the eyebrows in a person's face (for example, an end point or a four-divided point, the same applies hereinafter), a predetermined position on the eye contour, and a predetermined position on the nose and nose contours 68 parts such as a predetermined position on the contour of the upper and lower lips and a predetermined position on the contour of the face (face line) are set. That is, in the present embodiment, a predetermined position in the facial organs (eyebrows, eyes, nose, mouth) and facial contour that are commonly included in the human face are set as the characteristic parts. As shown in FIG. 4, the feature points CP are set (arranged) at positions representing 68 feature parts designated by the operator in each sample image SI. Since each feature point CP set in this way corresponds to each feature part, the arrangement of the feature points CP in the face image can be expressed as specifying the shape of the face.

  The position of the feature point CP in the sample image SI is specified by coordinates. FIG. 5 is an explanatory diagram showing an example of the coordinates of the feature point CP set in the sample image SI. In FIG. 5, SI (g) (g = 1, 2, 3...) Indicates each sample image SI, and CP (k) (k = 0, 1,..., 67) indicates each feature. Point CP is shown. CP (k) -X represents the X coordinate of the feature point CP (k), and CP (k) -Y represents the Y coordinate of the feature point CP (k). The coordinates of the feature point CP include predetermined reference points (for example, in the sample image SI normalized with respect to the size of the face, the inclination of the face (inclination in the image plane), and the position of the face in the X direction and the Y direction) The coordinates with the origin at the lower left point of the image are used. Further, in the present embodiment, a case where a plurality of human face images are included in one sample image SI is allowed (for example, two sample face images are included in the sample image SI (2)). Each person in one sample image SI is specified by a person ID. Each sample image SI, the coordinates of each feature point CP, and the person ID are input to the computer.

  Subsequently, the operator sets the AAM shape model (step S130). Specifically, the computer performs principal component analysis on a coordinate vector (see FIG. 5) composed of the coordinates (X coordinate and Y coordinate) of 68 feature points CP in each sample image SI, and the computer further uses the feature points. The face shape s specified by the position of the CP is modeled by the following equation (1). The shape model is also referred to as a feature point CP arrangement model.

In the formula (1), s ₀ is an average shape. FIG. 6 is an explanatory diagram showing an example of the average shape s ₀ . As shown in FIGS. 6A and 6B, the average shape s ₀ is a model representing the average face shape specified by the average position (average coordinate) for each feature point CP of the sample image SI. is there. In the present embodiment, in the average shape s ₀ , a region surrounded by a straight line connecting feature points CP (face lines, eyebrows, feature points CP corresponding to the eyebrows, see FIG. 4) located on the outer periphery (FIG. 6 ( b) is indicated by hatching) and is referred to as “average shape region BSA”. As shown in FIG. 6A, the computer sets a plurality of triangular areas TA having the feature points CP as vertices so as to divide the average shape area BSA into a mesh shape with respect to the average shape s ₀ .

In the equation (1) representing the shape model, s _i is a shape vector, and p _i is a shape parameter representing the weight of the shape vector s _i . The shape vector s _i is a vector representing the characteristics of the face shape s and is an eigenvector corresponding to the i-th principal component obtained by principal component analysis. As shown in the equation (1), in the shape model in the present embodiment, the face shape s representing the arrangement of the feature points CP is the model as the sum of the average shape s ₀ and the linear combination of the n shape vectors s _i. It becomes. As shown in the equation (1a), it is possible to reproduce the face shape s in any image by appropriately setting the shape parameter p _i in the shape model. Further, as shown in the equation (1b), by appropriately setting the shape parameter p _i in the shape model, it is possible to reversely convert from any shape s to the average shape s ₀ .

FIG. 7 is an explanatory diagram illustrating the relationship between the shape vector s _{i, the} shape parameter p _i, and the face shape s. As shown in FIG. 7 (a), in order to identify the face shape s, the computer selects the number n (the number) selected from the eigenvector corresponding to the principal component having the larger contribution rate in the descending order of the cumulative contribution rate. eigenvectors of that.) and 10 pieces in the embodiment, used as the shape vectors s _i. Each of the shape vectors s _i corresponds to the movement direction / movement amount of each feature point CP, as indicated by the arrows in FIG. In this embodiment, most first shape vector s ₁ corresponding to the first principal component having a large contribution rate is a vector that is approximately correlated with the left and right appearance of a face, by the magnitude of the shape parameter p _1, 7 As shown in (b), the horizontal face direction of the face shape s changes. Second shape vector s ₂ corresponding to the large second principal component of the contribution rate to the second is a vector that is approximately correlated with the vertical appearance of a face, by the magnitude of the shape parameter p _2, FIG. 7 (c) As shown in FIG. 4, the vertical face direction of the face shape s changes. The third shape vector s ₃ corresponding to the third principal component having the third largest contribution ratio is a vector that is substantially correlated with the aspect ratio of the face shape, and the fourth principal component having the fourth largest contribution ratio. The fourth shape vector s ₄ corresponding to is a vector that is substantially correlated with the degree of mouth opening. As described above, the value of the shape parameter represents the feature of the face image such as facial expression and face orientation. The n “shape parameters” in the shape model correspond to “basic features” in the claims.

The average shape s ₀ and the shape vector s _i set by the computer in the shape model setting step (step S130) are stored in the internal memory 120 of the printer 100 as AAM information AMI (FIG. 1). Subsequently, the computer sets an AAM texture model (step S140). Specifically, first, for each sample image SI, image conversion (hereinafter referred to as “warp”) is performed so that the setting position of the feature point CP in the sample image SI is equal to the setting position of the feature point CP in the average shape s ₀ . Also called “W”).

FIG. 8 is an explanatory diagram showing an example of a method of warping W of the sample image SI. In each sample image SI, similarly to the average shape s ₀ , the computer sets a plurality of triangular regions TA that divide the region surrounded by the feature points CP located on the outer periphery into a mesh shape. The warp W is a set of affine transformations for each of the plurality of triangular areas TA. That is, in the warp W, an image of a certain triangular area TA in the sample image SI is affine transformed into an image of the corresponding triangular area TA in the average shape s ₀ . The computer generates a sample image SI (hereinafter referred to as “sample image SIw”) in which the setting position of the feature point CP is equal to the setting position of the feature point CP in the average shape s ₀ by the warp W.

  Each sample image SIw has a rectangular frame containing an average shape area BSA (shown with hatching in FIG. 8) as an outer periphery, and an area other than the average shape area BSA (hereinafter also referred to as “mask area MA”). Generated as a masked image. An image area in which the average shape area BSA and the mask area MA are combined is referred to as a reference area BA. In addition, the computer normalizes each sample image SIw as an image having a size of, for example, 56 pixels × 56 pixels.

Next, the computer performs a principal component analysis on a luminance value vector composed of luminance values in each pixel group x of each sample image SIw, and a facial texture (also referred to as “appearance”) A (x) is expressed as follows: Modeled by equation (2). The pixel group x is a set of pixels located in the average shape area BSA.

In the formula (2), A ₀ (x) is an average face image. FIG. 9 is an explanatory diagram showing an example of the average face image A ₀ (x). The average face image A ₀ (x) is an image representing the average face of the sample image SIw (see FIG. 8) after the warp W. That is, the average face image A ₀ (x) is an image calculated by taking the average of the pixel values (luminance values) of the pixel group x in the average shape area BSA of the sample image SIw. Therefore, the average face image A ₀ (x) is a model representing the average face texture (appearance) in the average face shape. Note that the average face image A ₀ (x) is composed of the average shape area BSA and the mask area MA, similarly to the sample image SIw, and is calculated as an image having a size of 56 pixels × 56 pixels, for example.

In the equation (2) representing the texture model, a _k (x) is a texture vector, and λ _k is a texture parameter representing the weight of the texture vector a _k (x). The texture vector a _k (x) is a vector representing the characteristics of the facial texture A (x), and specifically, is an eigenvector corresponding to the j-th principal component obtained by principal component analysis. That is, the eigenvectors of the number m (four in this embodiment) set based on the cumulative contribution rate are adopted as the texture vector a _k (x) in order from the eigenvector corresponding to the principal component having the larger contribution rate. Is done. In the present embodiment, the first texture vector A ₁ (x) corresponding to the first principal component having the largest contribution rate is a vector that is substantially correlated with a change in face color (also regarded as a gender difference).

As shown in the equation (2), in the texture model in the present embodiment, the facial texture A (x) representing the appearance of the face is the average face image A ₀ (x) and m texture vectors a _k (x ) And the linear combination. As shown in the equation (2a), it is possible to reproduce the facial texture A (x) in any image by appropriately setting the texture parameter λ _k in the texture model. On the contrary, as shown in the equation (2b), by appropriately setting the texture parameter λ _k in the texture model, the face texture A (x) in every image is reversed to the average face image A ₀ (x). Can be converted. The m “texture parameters” in the texture model correspond to “basic features” in the claims.

The average face image A ₀ (x) and the texture vector a _k (x) set in the texture model setting step (step S140 in FIG. 2) are stored in the internal memory 120 of the printer 100 as AAM information AMI (FIG. 1). Stored. The computer records the AAM information AMI on any recording medium, and when the printer 100 is manufactured, the internal memory 120 that records the AAM information AMI read from the recording medium is incorporated in the printer 100 or rarely incorporated in the printer 100. The AAM information AMI read from the recording medium is recorded in the internal memory 120.

Through the processing so far, a shape model for modeling the shape of the face and a texture model for modeling the texture of the face are set. By combining the set shape model and the texture model, that is, the synthesized texture A (x) is converted from the average shape s ₀ to the shape s (inverse conversion of the warp W shown in FIG. 8). Thus, it is possible to reproduce the shape and texture of any face image.

Next, AAM setting processing (steps S110 to S140) is performed on the sample images SI belonging to the sample group G2. As a result, the average shape s ₀ , the average face image A ₀ (x), the shape vector s _i, and the texture vector a _k (x) are different from those when the AAM setting process is performed on the sample images SI belonging to the sample group G1. Is stored as AAM information AMI. The expression shape model is set by the average shape s ₀ and the shape vector s _i , and the expression texture model is set by the average face image A ₀ (x) and the texture vector a _k (x). It should be noted that the average shape s ₀ and the average face image A ₀ (x) in the expression shape model and the expression texture model respectively correspond to no expression.

To perform the AAM setting process sample image SI belonging to the sample group G2 as the target, the first to fourth shape vector s ₁ ~s ₄ corresponding to the large first to fourth principal component of most contribution rate in look for shape model, Each vector is correlated with whether it is a happy face, whether it is a sad face, whether it is a surprise face, or whether it is an angry face. That is, since the AAM setting process is performed on the sample group G2 including only face images that are different from each other with respect to the attributes of facial expressions (happy face, sad face, surprised face, and angry face), it is determined whether or not each face is an expression. The cumulative contribution rate of the shape vector s _i shown is the highest. The principal components used for the facial shape model are the first to fourth principal components (n = 4). For example, the shape vector s ₁ corresponding to whether or not the face is a happy face is an eigenvector that comprehensively varies the angle of the corner of the eye, the angle of the mouth, and the degree of lowering of the corner of the eye. The shape parameters p ₁ ~p ₄ is a coefficient for the first to fourth shape vector s ₁ ~s ₄ corresponds to the "expression feature quantity" in the claims.

On the other hand, also in the texture model for facial expressions, are the first to fourth texture vectors a ₁ (x) to a ₄ (x) corresponding to the first to fourth principal components having the largest contribution rate each being a happy face? No, whether it is a sad face, whether it is an amazing face, or whether it is an angry face. The main components used here are also the first to fourth main components (m = 4). For example, the texture vector a ₁ (x) corresponding to whether or not the face is a happy face is an eigenvector that comprehensively fluctuates color changes and the like caused by wrinkles around the corners of the eyes and wrinkles around the mouth. The coefficient texture parameter lambda ₁ to [lambda] ₄ is about first to fourth shape vector s ₁ ~s ₄ corresponds to the "expression feature quantity" in the claims.

A3. Face feature detection processing:
FIG. 10 is a flowchart illustrating a flow of face feature detection processing in the embodiment. The face feature detection processing in this embodiment is feature point CP setting position correction processing that determines the arrangement of feature points CP in the face image included in the target image using AAM (basic face model (shape model, texture model)). (Step S250) is a process for detecting the position of the characteristic part in the face image, and further detecting the texture of the face by the texture detection process (Step S260). As described above, in the present embodiment, in the AAM setting process (FIG. 2), a total of 68 predetermined positions in the human facial organs (eyebrows, eyes, nose, mouth) and facial contour are set as the characteristic parts. (See FIG. 4). Therefore, in the face feature detection process of the present embodiment, the arrangement of 68 feature points CP indicating predetermined positions in the human face organ and face contour is determined.

When the arrangement of the feature points CP in the face image is determined by the face feature detection process, the value of the shape parameter p _i for the face image is specified. Accordingly, the processing result of the face feature detection process can be used for face orientation determination for detecting a face image in a specific direction (for example, rightward or downward), face deformation for deforming the face shape, face shading correction, and the like. .

  First, the image processing unit 200 (FIG. 1) acquires image data representing an attention image that is a target of face feature detection processing (step S210). In the printer 100 of the present embodiment, when the memory card MC is inserted into the card slot 172, thumbnail images of image files stored in the memory card MC are displayed on the display unit 150. One or more images to be processed are selected by the user via the operation unit 140. The image processing unit 200 acquires an image file including image data corresponding to one or more selected images from the memory card MC and stores it in a predetermined area of the internal memory 120. Note that the acquired image data is referred to as attention image data, and an image represented by the attention image data is referred to as attention image OI.

  The face area detection unit 230 (FIG. 1) detects an image area including at least a part of the face image included in the target image OI as the face area FA (step S230). The detection of the face area FA can be performed using a known face detection method. Known face detection methods include, for example, pattern matching methods, skin color region extraction methods, learning using sample images (for example, learning using neural networks, learning using boosting, and support vector machines). For example, a method using learning data set by learning).

  FIG. 11 is an explanatory diagram illustrating an example of a detection result of the face area FA in the target image OI. FIG. 11 shows the face area FA detected in the target image OI. In the present embodiment, a face detection method is used in which a rectangular area including the vertical direction of the face from the forehead to the chin and the horizontal direction from the outside of both ears is detected as the face area FA.

  The setting unit 210 (FIG. 1) sets the initial position of the feature point CP in the target image OI (Step S240). FIG. 12 is a flowchart showing a flow of initial position setting processing of the feature point CP in the present embodiment. In the present embodiment, the setting unit 210 changes the values of global parameters representing the size, inclination, and position (vertical position and horizontal position) of the face image with respect to the face area FA to change the feature point CP. The temporary setting position on the attention image OI is set (step S310).

FIG. 13 is an explanatory diagram showing an example of a temporary setting position of the feature point CP by changing the value of the global parameter. 13A and 13B show a mesh formed by connecting feature points CP and feature points CP in the target image OI. As shown in the center of FIGS. 13A and 13B, the setting unit 210 temporarily sets the feature point CP (hereinafter, referred to as an average shape s _{0) at} which the average shape s ₀ is formed at the center of the face area FA. (Also referred to as “reference temporary setting position”).

  The setting unit 210 also sets a plurality of temporary setting positions obtained by changing various global parameter values with respect to the reference temporary setting position. Changing the global parameters (size, inclination, vertical position and horizontal position) means that the mesh formed by the feature point CP in the target image OI is enlarged / reduced, the inclination is changed, and the parallel movement is performed. Equivalent to. Accordingly, as shown in FIG. 13 (a), the setting unit 210 forms a temporary setting position that forms a mesh obtained by enlarging or reducing the reference temporary setting position mesh at a predetermined magnification (below the reference temporary setting position diagram and Or a temporary setting position (shown on the right and left in the drawing of the reference temporary setting position) that forms a mesh whose inclination is changed clockwise or counterclockwise by a predetermined angle. Further, the setting unit 210 forms a temporary setting position (upper left of the figure of the reference temporary setting position, which forms a mesh obtained by performing a combination of enlargement / reduction and change in inclination with respect to the mesh of the reference temporary setting position. Also set in the lower left, upper right, and lower right).

  Also, as shown in FIG. 13B, the setting unit 210 forms a temporary setting position (a reference temporary setting position diagram) that forms a mesh that has been translated upward or downward by a predetermined amount from the reference temporary setting position mesh. And a temporary setting position (shown on the left and right in the drawing of the reference temporary setting position) that forms a mesh moved in parallel to the left or right. Further, the setting unit 210 forms a temporary setting position (upper left and lower left in the figure of the reference temporary setting position) that forms a mesh obtained by performing a combination of vertical and horizontal parallel movements on the mesh of the reference temporary setting position. , Shown in the upper right and lower right).

  The setting unit 210 also has temporary setting positions at which parallel movement in the vertical and horizontal directions shown in FIG. 13B is executed with respect to the mesh at each of the eight temporary setting positions other than the reference temporary setting position shown in FIG. Set. Therefore, in the present embodiment, the four global parameters (size, inclination, vertical position, horizontal position) are set in 80 ways (= 3 × 3 × 3) that are set as known three-level values. A total of 81 temporary setting positions of (3-1) temporary setting positions and reference temporary setting positions are set.

The generation unit 222 (FIG. 1) generates an average shape image I (W (x; p)) corresponding to each set temporary setting position (step S320). FIG. 14 is an explanatory diagram showing an example of the average shape image I (W (x; p)). The average shape image I (W (x; p)) is calculated by conversion such that the arrangement of the feature points CP in the input image is equal to the arrangement of the feature points CP in the average shape s ₀ .

The transformation for calculating the average shape image I (W (x; p)) is a warp W that is a set of affine transformations for each triangular area TA, similarly to the transformation for calculating the sample image SIw (see FIG. 8). Is done. Specifically, the average shape region BSA (region surrounded by the feature points CP located on the outer periphery) is specified by the feature points CP (see FIG. 13) arranged in the attention image OI, and the average shape region in the attention image OI. An average shape image I (W (x; p)) is calculated by performing affine transformation for each triangular area TA on the BSA. In this embodiment, the average shape image I (W (x; p) ) is composed of an average face image A ₀ (x) and an average shape area BSA and a mask area MA, average face image A ₀ (x) And is calculated as an image of the same size.

As described above, the pixel group x is a set of pixels located in the average shape region BSA in the average shape s ₀ . The pixel group in the image (average shape area BSA of the target image OI) before execution of the warp W corresponding to the pixel group x in the image after execution of the warp W (face image having the average shape s ₀ ) is expressed as W (x; p). To express. Since the average shape image is an image composed of luminance values in each of the pixel groups W (x; p) in the average shape region BSA of the target image OI, it is represented as I (W (x; p)). FIG. 14 shows nine average shape images I (W (x; p)) corresponding to the nine temporarily set positions shown in FIG.

The calculation unit 224 (FIG. 1) calculates an average shape image I (W (x; p)) corresponding to each temporarily set position and an average face image A ₀ (x) as a “comparison face image” in the claims. The difference image Ie is calculated (step S330). Since 81 types of temporary setting positions of the feature points CP are set, the calculation unit 224 (FIG. 1) calculates 81 difference images Ie.

The setting unit 210 calculates a norm from the pixel values of each difference image Ie, and sets a temporary installation position corresponding to the difference image Ie having the smallest norm value (hereinafter also referred to as “norm minimum temporary setting position”) as the target image OI. Is set as the initial position of the feature point CP at (step S340). The pixel value for calculating the norm may be a luminance value or an RGB value. Thus, the feature point CP initial position setting process is completed. Note that the smaller the norm value, the smaller the difference between the average shape image I (W (x; p)) and the average face image A ₀ (x) (both approximate).

A3-1. Feature point CP setting position correction processing:
When the feature point CP initial position setting process is completed, the feature position detector 220 (FIG. 1) corrects the set position of the feature point CP in the target image OI (step S250). FIG. 15 is a flowchart showing the flow of the feature point CP setting position correction process in the present embodiment.

  The generation unit 222 (FIG. 1) calculates an average shape image I (W (x; p)) from the attention image OI (step S410). The calculation method of the average shape image I (W (x; p)) is the same as that in step S320 in the feature point CP initial position setting process.

The feature position detection unit 220 calculates a difference image Ie between the average shape image I (W (x; p)) and the average face image A ₀ (x) (step S420). The feature position detection unit 220 determines whether or not the feature point CP setting position correction process has converged based on the difference image Ie (step S430). The feature position detection unit 220 calculates the norm of the difference image Ie, determines that it has converged if the norm value is smaller than a preset threshold value, and still converges if the norm value is greater than or equal to the threshold value. Judge that there is no. The feature position detection unit 220 determines that the calculated norm value of the difference image Ie has converged when the value is smaller than the value calculated in the previous step S430, and still determines that the value is equal to or greater than the previous value. It is good also as what determines with not having converged. Alternatively, the feature position detection unit 220 may perform the convergence determination by combining the determination based on the threshold and the determination based on comparison with the previous value. For example, the feature position detection unit 220 determines that the calculated norm value has converged only when the calculated norm value is smaller than the threshold value and smaller than the previous value, and otherwise determines that it has not yet converged. It may be a thing.

If it is determined in step S430 that the convergence is not yet completed, the correction unit 226 (FIG. 1) calculates the parameter update amount ΔP (step S450). The parameter update amount ΔP includes four global parameters (total size, inclination, X direction position, Y direction position), and n shape parameters p _i (i = 1 to n) as feature amounts. .) Means the change amount of the value. Immediately after setting the feature point CP to the initial position, the global parameter is set to the value determined in the feature point CP initial position setting process (FIG. 12). Further, difference between the initial position of the characteristic point CP and sets the position of the characteristic points CP of the average shape s ₀ in this case, the overall size, the tilt, because it is limited to the difference in position, shape parameters in the shape model p The value of _i and the texture parameter λ _k in the texture model are all zero.

  The parameter update amount ΔP is calculated by the following equation (3). That is, the parameter update amount ΔP is a product of the update matrix R and the difference image Ie.

The update matrix R in the equation (3) is a matrix of M rows and N columns set in advance by learning in order to calculate the parameter update amount ΔP based on the difference image Ie.
It is stored in the internal memory 120 as AAM information AMI (FIG. 1). In this embodiment, the number of rows M in the update matrix R is equal to the sum ((4 + n)) of the number of global parameters (4) and the number of shape parameters p _i (n), and the number of columns N is , average face image a ₀ average number of pixels in the shape area BSA of (x) - is equal to (56 pixels × 56 pixels the number of pixels in the mask area MA). The update matrix R is calculated by the following formulas (4) and (5).

The correction unit 226 (FIG. 1) updates the parameters (four global parameters and n shape parameters p _i ) based on the calculated parameter update amount ΔP (step S460). Thereby, the setting position of the feature point CP in the target image OI is corrected. The correcting unit 226 corrects so that the norm of the difference image Ie is reduced. After the parameter update, the average shape image I (W (x; p)) is calculated again from the target image OI in which the installation position of the feature point CP is corrected (step S410), and the difference image Ie is calculated (step S420), a convergence determination based on the difference image Ie (step S430) is performed. The average shape image I (W (x; p)) changes according to the update (change) of the shape parameter p _i and corresponds to a “face region” in the claims. If it is determined that the convergence has not occurred in the convergence determination again, the parameter update amount ΔP based on the difference image Ie is calculated (step S450), and the setting position correction of the feature point CP by the parameter update (step S460). ) Is performed.

When the processing from step S410 to step S460 in FIG. 15 is repeatedly executed, the position of the feature point CP corresponding to each feature part in the target image OI and the position of the actual feature part approach as a whole and converge at a certain time. In the determination (step S430), it is determined that convergence has occurred. If it is determined in the convergence determination that convergence has been achieved, the face feature detection process is completed (step S470). The setting position of the feature point CP specified by the global parameter and the shape parameter p _i set at this time is specified as the setting position of the feature point CP in the final target image OI.

  FIG. 16 is an explanatory diagram illustrating an example of a result of the feature point CP setting position correction process. FIG. 16 shows the setting positions of the feature points CP finally specified in the target image OI. Since the position of the feature point CP specifies the position of the facial feature part (predetermined position in the facial organs (eyebrows, eyes, nose, mouth) and facial contour) included in the attention image OI. It is possible to detect the shape and position of the human face organ and the face contour shape in the image OI.

  When the feature point CP setting position correction processing is completed, the texture detection unit 240 (FIG. 1) corrects (detects) the texture (appearance) of the average shape image I (W (x; p)) set in the target image OI. Is performed (step S260).

A3-2. Texture detection processing:
FIG. 17 is a flowchart showing the flow of texture detection processing. The generation unit 242 (FIG. 1) calculates an average shape image I (W (x; p)) from the attention image OI (step S510). The calculation method of the average shape image I (W (x; p)) is the same as that in step S320 in the feature point CP initial position setting process and step S410 in the feature point CP setting position correction process. position correction processing completion time using the values of the global parameters and the shape parameters p _i of the average shape image I (W (x; p) ) is calculated.

前記式（６）は、前記式（２ｂ）における顔のテクスチャーＡ（ｘ）にＷ（ｘ；ｐ）を代入したものであり、テクスチャーパラメーターλ_kが適切に設定されれば、Ｔ（ｘ；λ）が平均顔画像Ａ₀（ｘ）と等しくなる。 Further, the generation unit 242 of the texture detection unit 240 calculates an average texture image J (T (x; λ)) based on the average shape image I (W (x; p)) (step S515). The average texture image J (T (x; λ)) is calculated by the following equation (6).

The equation (6) is obtained by substituting W (x; p) into the facial texture A (x) in the equation (2b). If the texture parameter λ _k is appropriately set, T (x; λ) is equal to the average face image A ₀ (x).

Then, the calculation unit 244 calculates a difference image Ie between the average texture image J (T (x; λ)) and the average face image A ₀ (x) (step S520). The texture detection unit 240 determines whether the texture detection process has converged based on the difference image Ie (step S530). The texture detection unit 240 calculates the norm of the difference image Ie, determines that it has converged if the norm value is smaller than a preset threshold value, and has not yet converged if the norm value is greater than or equal to the threshold value. Is determined. Note that the texture detection unit 240 determines that the calculated norm value of the difference image Ie has converged if it is smaller than the value calculated in the previous step S530, and still converges if it is greater than or equal to the previous value. It is good also as what determines with not doing. The convergence determination can be performed by the same method as in step S430 in the feature point CP setting position correction process.

If it is determined in step S530 that the convergence has not yet been completed, the correction unit 246 (FIG. 1) calculates the parameter update amount Δλ (step S550). The parameter update amount ΔΛ means a change amount of the value of m texture parameters λ _k (k = _{1 to} m), which is a feature amount. The parameter update amount ΔΛ is calculated by the following equation (7). That is, the parameter update amount ΔΛ is a product of the update matrix U and the difference image Ie.

The update matrix U in Equation (7) is a matrix of M rows and N columns set in advance by learning in order to calculate the parameter update amount ΔΛ based on the difference image Ie.
It is stored in the internal memory 120 as AAM information AMI (FIG. 1). In this embodiment, the number M of rows in the update matrix U is equal to the texture parameter λ _k (m), and the number N of columns is the number of pixels in the average shape area BSA (56 pixels × 56 pixels−pixels in the mask area MA). Number). The update matrix U is calculated by the following formulas (8) and (9).

The correction unit 246 (FIG. 1) updates the texture parameter λ _k based on the calculated parameter update amount ΔΛ (step S560). Thereby, the facial texture A (x) is corrected. That is, the correction unit 246 corrects so that the norm of the difference image Ie is reduced. After the parameter update, the average texture image J (T (x; λ)) is calculated again based on the corrected texture parameter λ _k (step S515), the difference image Ie is calculated (step S520), and the difference image. A convergence determination (step S530) based on Ie is performed. If it is determined that the convergence has not been made in the convergence determination again, the parameter update amount ΔΛ based on the difference image Ie is calculated (step S550), and the texture is corrected by updating the parameter (step S560). .

When the processing from step S510 to S560 in FIG. 17 is repeatedly executed, the luminance value (color value) of each pixel of the facial texture A (x) approaches the luminance value of the actual face image as a whole, At a certain point in time, it is determined that convergence has been made in the convergence determination (step S530). If it is determined in the convergence determination that the image has converged, the texture detection process is completed (step S570). The facial texture A (x) specified by the value of the texture parameter λ _k set at this time is specified as the final facial appearance. Based on the facial texture A (x), facial color correction and the like can be performed. When the shape parameter p _i and the texture parameter λ _k for the face area FA in the target image OI can be specified as described above, the facial expression determination process is executed next.

A4. Facial expression determination processing:
FIG. 18 is a flowchart showing the flow of facial expression determination processing. The facial expression determination process also includes a feature point CP setting position correction process (step S620) and a texture detection process (step S630), similar to the face feature detection process described above. These processes include the feature position detection unit 220 and the texture. This is executed by the detection unit 240. However, it differs from the above-described face feature detection processing in that a shape model for facial expression and a texture model for facial expression are used as AAM. In the facial feature detection process, the basic shape and position of facial organs such as eyes and mouth are specified, but in the facial expression determination process, the facial features that have changed in detail from the basic shape and position of the facial organ specified in advance. The facial expression is determined based on the shape and position of the organ. The feature position detection unit 220 and the texture detection unit 240 that perform the feature point CP setting position correction process and the texture detection process in the face feature detection process correspond to a “basic feature amount correction unit” in the claims. On the other hand, the feature position detection unit 220 and the texture detection unit 240 that execute the feature point CP setting position correction process and the texture detection process in the facial expression determination process correspond to the “facial expression feature amount correction unit” in the claims. First, using the global parameter, shape parameter p _i, and texture parameter λ _k (corresponding to “corrected basic feature amount” in the claims) finally obtained by the face feature detection process described above, An average texture image J (T (x; λ)) is calculated (step S610). The average texture image J (T (x; λ)) can be calculated in the same manner as steps S510 and S515 described above.

The average texture image J (T (x; λ)) indicates an image obtained by averaging the face area FA in the target image OI based on the global parameter, the shape parameter p _i, and the texture parameter λ _k . However, some eigenvectors selected using the AAM (shape model and texture model) modeled using the sample group G1 including the sample image SI that does not include the presence or absence of facial expression attributes, in order of the highest cumulative contribution rate Since the averaging is performed using, features resulting from facial expressions remain in the average texture image J (T (x; λ)). In the shape model and texture model trained with the sample image SI that does not include the presence or absence of facial expression attributes, the cumulative contribution rate of eigenvectors related to facial expressions is low, and the cumulative contribution rate of eigenvectors that change facial expressions is the top n, m It is because it will not be within the rank.

FIG. 19 is a diagram illustrating an example of the average texture image J (T (x; λ)). In the figure, an average texture image J (T (x; λ)) for a smiling expression (happy face) is shown. For example, the original face area FA in the target image OI indicates a left-handed face, but is corrected in the front direction in the average texture image J (T (x; λ)) by the correction based on the first shape vector s _1. Yes. In addition, individuality such as race and gender is averaged to be lost. However, since the sample image SI belonging to the sample group G1 does not include the presence or absence of the expression attribute, fine features (mouth corners, corners of eyes, laughter, etc.) due to the happy face are not included in the average texture image J ( T (x; λ)). When such an average texture image J (T (x; λ)) can be calculated, the average texture image J (T (x; λ)) is used as a facial expression for facial expression determination FE for facial expression determination.

20 and 21 are flowcharts showing the flow of the feature point CP setting position correction process (step S620) and the texture detection process (step S630) in the facial expression determination process. The overall flow of the feature point CP setting position correction process and the texture detection process is the same as the feature point CP setting position correction process (step S250) and the texture detection process (step S260) in the face feature detection process described above. However, in step S710 of the feature point CP setting position correction process, the facial expression image FE for facial expression determination is acquired, and the facial shape model for facial expression and the facial texture model for facial expression are processed. It differs from the face feature detection process in that it is used. In step S710, feature points CP are initially set for facial expression determination face image FE. For determining a facial expression face image FE is approximately the average for will have a position similar characteristic site and disposition of the characteristic points CP in shape s _0, the characteristic point CP in the average shape s ₀ without setting a plurality of global parameters The layout may be initialized. At the initial setting stage, basically, the fine feature point CP shifts due to the facial expression.

In step S750 of the feature point CP setting position correction process, four (n = 4) shape parameters of the shape model for expression indicating whether each expression is a facial expression (a happy face, a sad face, a surprised face, an angry face). p _i will be updated sequentially. In this way, the arrangement of the feature points CP of the facial expression determination face image FE is changed in association with the change in facial expression (change in the shape parameter p _i ), which is close to the average shape s ₀ indicating no expression. The shape parameter p _i when the arrangement of the feature points CP is obtained can be specified. Similarly, in the texture detection process S850, four (m = 4) facial texture parameters λ _k indicating whether each facial expression (a happy face, a sad face, a surprised face, and an angry face) is present. Will be updated sequentially. In this way, the brightness value of each pixel of the facial expression determination face image FE is changed in association with the change in facial expression (change in the texture parameter λ _k ), and the average facial image A ₀ (no expression) is shown. It is possible to specify the texture parameter λ _k when a luminance value distribution close to x) is obtained.

  As shown in FIG. 19, at the stage of the average texture image J (T (x; λ)) in which the basic shape and position of the facial organ are specified by the face feature detection process described above, the feature points corresponding to the facial expression The position of the feature point CP that varies depending on the facial expression by performing the feature point CP setting position correction process and the texture detection process using the facial expression shape model and the facial expression texture model. Or the luminance value of each pixel. For example, by the feature point CP setting position correction process, it is possible to correct to the position of the feature point CP that matches the mouth angle of the happy face. In addition, laughter and the like on a happy face can be reflected by the texture detection process.

When the texture detection process ends, the facial expression determination unit 250 acquires the shape parameter p _i and the texture parameter λ _k at the end of the texture detection process (step S640), and determines the facial expression based on these (step S650). As described above, the first to fourth shape vector s ₁ ~s ₄ corresponding to the large first to fourth principal component of most contribution rate in look for shape model, whether each Hee face, is悲顔Since it is a vector that substantially correlates whether or not it is a surprise face or an angry face, the magnitude of the shape parameter p _i (i = 1 to 4) as these coefficients is , Each facial expression will be shown. The size of the texture parameter λ _k (k = 1 to 4) of the texture model for facial expression also indicates the likelihood of each facial expression. Therefore, it is possible to determine which facial expression is based on the values of the shape parameter p _i and the texture parameter λ _k . In this embodiment, each value of the shape parameter p _i and the texture parameter λ _k is normalized by a conversion formula that normalizes the shape parameter p _i and the texture parameter λ _k with respect to the degree of the facial expression. Because the degree of each facial expression likelihood depends on human sense, the specimen corresponding to each value of shape parameter p _i and texture parameter lambda _k, is the conversion formula on the basis of a result of the operator is determined each facial likeness prepared It is desirable to be done.

Then, the largest component of the normalized shape parameter p _i and the largest component of the normalized texture parameter λ _k are detected, and if they correspond to the same facial expression, It is determined that When the shape parameter p _i and the texture parameter λ _k show contradictory expressions, the larger one of them may be adopted, or the shape parameter p _i may be prioritized. Further, if the value of the largest component of the normalized shape parameter p _i and texture parameter λ _k is smaller than a predetermined threshold value, it may be determined that there is no expression. In this embodiment, after tag information for identifying each expression is attached to the attention image OI, if the attention image OI includes a happy face, the attention image OI is output to the printing mechanism 160 and printing is performed ( Step S660).

In the present embodiment, independent AAM is used for the facial feature detection process and the facial expression determination process, so that the shape parameter p _i and the texture parameter λ _k in each process can be updated at high speed and with high accuracy. That is, in the face feature detection process and the facial expression determination process, the number of elements of the shape parameter p _i and the texture parameter λ _k to be changed can be suppressed, and the process can be made efficient. For example, by using the AAM learned from the sample image SI in which the sample group G1 and the sample group G2 are integrated so that the facial expression determination can be performed simultaneously with the facial feature detection process, the facial expression determination can be performed simultaneously with the facial feature detection process. It is possible to do it. However, the number of shape parameters p _i and texture parameters λ _k to be updated in the feature point CP setting position correction process and the texture detection process increases, and the process becomes inefficient. In addition, after detecting the basic position, shape, and color of the facial organs reliably in the facial feature detection process, the facial expression detection process detects the changes in facial organ position, shape, and color caused by facial expressions. Accurate facial expression change can be determined accurately.

B. Variation:
The present invention is not limited to the above-described examples and embodiments, and can be implemented in various modes without departing from the gist thereof. For example, the following modifications are possible.

B1. Modification 1:
In the above-described embodiment, the sample group G2 includes only the sample images SI including only face images that differ from each other with respect to the attributes of facial expressions (happy face, sad face, surprised face, and angry face). It is not limited to including only facial images that differ from each other in terms of facial expression attributes. You may make it include the sample image SI which has another attribute in the sample group G2. For example, sample images SI having different race attributes may be included in the sample group G2.

B2. Modification 2:
On the contrary, the sample group G1 may include sample images SI including face images that differ from each other with respect to attributes of facial expressions (joyful face, sad face, surprised face, and angry face). By adding face images that differ from each other in terms of facial expression attributes to the sample group G1, it is possible to ensure the robustness of the facial feature detection processing with respect to changes in facial expressions. In this case, it is conceivable that a difference in facial expression appears in the shape parameter p _i and the texture parameter λ _k in the face feature detection process. Therefore, it is possible to determine the facial expression based on the shape parameter p _i and the texture parameter λ _k in the face feature detection process. However, since there is a possibility that sufficient facial expression determination accuracy may not be obtained only with the shape parameter p _i and the texture parameter λ _{k in} the face feature detection process, the expression of the expression of the shape parameter p _i and the texture parameter λ _k in the face feature detection process The facial expression determination may be performed based on these only when the method is more conspicuous (for example, when the linear combination value of a certain shape parameter p _i is equal to or greater than a predetermined threshold). As a result, the computation on the facial expression model can be omitted, and the processing speed can be increased. If the expression of the facial expression of the shape parameter p _i and the texture parameter λ _k is not remarkable in the facial feature detection processing, the facial expression is accurately determined using the facial expression shape model and facial expression texture model as in the above embodiment. Can be done.

B3. Modification 3:
In the above-described embodiment, it is determined that the expression has the largest single expression, but it is indicated that the shape parameter p _i and the texture parameter λ _k corresponding to a plurality of expressions are more likely to be expressions than a predetermined threshold. In this case, it may be determined that a plurality of facial expressions are mixed. For example, when both the values obtained by normalizing the shape parameters p _i corresponding to the happy face and the surprised face exceed the threshold value, it may be determined that both facial expressions are included.

B4. Modification 4:
In the embodiment, in the feature point CP initial position setting process, a total of 80 types corresponding to combinations of values in three stages for each of the four global parameters (size, inclination, vertical position, and horizontal position). A temporary setting position (= 3 × 3 × 3 × 3-1) is preset, but the type and number of parameters used for setting the temporary setting position and the number of parameter values can be changed. For example, only a part of the four global parameters may be used for setting the temporary setting position, or the temporary setting position may be set by combining five values for each parameter used. Good.

B5. Modification 5:
In the feature point CP setting position correction process of the above embodiment, the average shape image I (W (x; p)) is calculated based on the target image OI to determine the installation position of the feature point CP of the target image OI as the average face image A. _Although it is matched with the installation position of the feature point CP of ₀ (x), the arrangement of both feature points CP may be matched by performing image conversion on the average face image A ₀ (x). That is, in the above embodiment, the average shape image I (W (x; p)) as the “face region” is changed, but the average face image A ₀ (x) is changed to the shape parameter p _i and the texture parameter λ. You may change according to the change of _k . In any case, the shape parameter p _i and the texture parameter λ _k that reduce the difference between the “face region” included in the target image OI and the face image reproduced by the face model are corrected. Note that the global parameter, shape parameter p _i, and texture parameter λ _k (corresponding to “corrected basic feature amount” in the claims) finally obtained by the face feature detection process of the embodiment are described above. The texture A (x) obtained by substituting into the equations (1) and (2) corresponds to the “basic face image” in the claims. In the above-described embodiment, the “basic face image” is not calculated, and instead, the face area FA in the target image OI is inversely transformed so as to approach the average face image A ₀ (x) by the “corrected basic feature amount”. It is said.

B6. Modification 6:
The sample image SI (FIG. 3) in the above embodiment is merely an example, and the number and type of images employed as the sample image SI can be arbitrarily set. In the embodiment, the predetermined feature portion (see FIG. 4) of the face indicated by the position of the feature point CP is merely an example, and a part of the feature portion set in the embodiment may be omitted, Other parts may be adopted as the part.

  In the embodiment, the texture model is set by the principal component analysis for the brightness value vector formed by the brightness value in each of the pixel groups x of the sample image SIw, but the brightness representing the texture (appearance) of the face image. The texture model may be set by principal component analysis for index values other than the values (for example, RGB values).

In the embodiment, the size of the average face image A ₀ (x) is not limited to 56 pixels × 56 pixels, and may be other sizes. Further, the average face image A ₀ (x) does not need to include the mask area MA (FIG. 8), and may be configured only by the average shape area BSA. Further, instead of the average face image A ₀ (x), another reference face image set based on the statistical analysis of the sample image SI may be used.

  Moreover, the facial expression to be determined is not limited to a happy face, a sad face, a surprised face, and an angry face, and expressions such as a troubled face, a sleeping face, and a dark face may be determined. Of course, in the embodiment and the modified example, it is possible to determine not only all of a happy face, a sad face, a surprised face, and an angry face, but, for example, whether or not it is a happy face.

  In the embodiment, the shape model and the texture model are set using AAM. However, the shape model and texture model using other modeling methods (for example, a method called Morphable Model or a method called Active Blob) are used. A texture model may be set.

  In the embodiment, the image stored in the memory card MC is set as the attention image OI. However, the attention image OI may be an image acquired via a network, for example. In the above embodiment, image processing by the printer 100 as an image processing apparatus has been described. However, part or all of the processing is executed by another type of image processing apparatus such as a personal computer, a digital still camera, or a digital video camera. It is good also as what is done. The printer 100 is not limited to an ink jet printer, and may be another type of printer such as a laser printer or a sublimation printer.

  In the embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware.

  In addition, when part or all of the functions of the present invention are realized by software, the software (computer program) can be provided in a form stored in a computer-readable recording medium. In the present invention, the “computer-readable recording medium” is not limited to a portable recording medium such as a flexible disk or a CD-ROM, but an internal storage device in a computer such as various RAMs and ROMs, a hard disk, etc. It also includes an external storage device fixed to the computer.

  DESCRIPTION OF SYMBOLS 100 ... Printer, 110 ... CPU, 120 ... Internal memory, 140 ... Operation part, 150 ... Display part, 160 ... Printing mechanism, 170 ... Card interface, 172 ... Card slot, 200 ... Image processing part, 210 ... Setting part, 220 ... feature position detection unit, 222, 242 ... generation unit, 224, 244 ... calculation unit, 226, 246 ... correction unit, 230 ... face area detection unit, 240 ... texture detection unit, 250 ... facial expression determination unit, 310 ... display process Part, 320... Print processing part.

Claims (8)

An image processing apparatus for determining facial expressions included in a noticed image,
A face area detection unit that detects an image area including at least a part of a face image from the attention image as a face area;
A basic face model that reproduces at least the position of a facial feature part based on a plurality of basic feature quantities, and a variation in the position of the feature part according to at least a facial expression based on a plurality of facial expression feature quantities associated with the facial expression A setting unit for setting a facial model for facial expression that reproduces
A basic feature amount correction unit that corrects the basic feature amount so that the position of the feature portion reproduced by the basic face model approximates the face region;
The position of the feature part, which is reproduced by applying the corrected basic feature quantity to the basic face model, is further approximated to the face region when the position of the feature part is changed by the facial model for facial expression. A facial expression feature correction unit that corrects the facial expression feature,
An image processing apparatus comprising: a determination unit that determines a facial expression based on the corrected facial feature.   The image processing apparatus according to claim 1, wherein the basic feature quantity and the facial expression feature quantity include eigenvector coefficients of each principal component obtained by performing principal component analysis on the positions of feature portions of a plurality of sample images.   The image processing apparatus according to claim 2, wherein the basic face model and the facial expression model are set using different sample images.   4. The basic feature amount and the facial expression feature amount include coefficients of eigenvectors of respective principal components obtained by performing principal component analysis on color values of pixels of a plurality of sample images. Image processing apparatus.   5. The image processing apparatus according to claim 1, wherein each facial expression feature amount indicates a facial expression characteristic of a different facial expression. 6. An image processing method for determining a facial expression included in an image of interest,
An image area including at least a part of a face image is detected as a face area from the attention image,
A basic face model that reproduces at least the position of a facial feature part based on a plurality of basic feature quantities, and a variation in the position of the feature part according to at least a facial expression based on a plurality of facial expression feature quantities associated with the facial expression Set the facial model for facial expression to reproduce
Correcting the basic feature amount so that the position of the feature part reproduced by the basic face model approximates the face region;
The position of the feature part, which is reproduced by applying the corrected basic feature quantity to the basic face model, is further approximated to the face region when the position of the feature part is changed by the facial model for facial expression. Correct the facial expression feature amount to
An image processing method for determining a facial expression based on the corrected facial feature. An image processing program for image processing for determining a facial expression included in an image of interest,
A face area detection function for detecting, as a face area, an image area including at least a part of a face image from the attention image;
A basic face model that reproduces at least the position of a facial feature part based on a plurality of basic feature quantities, and a variation in the position of the feature part according to at least a facial expression based on a plurality of facial expression feature quantities associated with the facial expression A setting function to set a facial model for facial expression that reproduces
A basic feature amount correction function for correcting the basic feature amount so that the position of the feature portion reproduced by the basic face model approximates the face region;
The position of the feature part, which is reproduced by applying the corrected basic feature quantity to the basic face model, is further approximated to the face region when the position of the feature part is changed by the facial model for facial expression. A facial expression feature correction function for correcting the facial expression feature,
An image processing program for causing a computer to realize a determination function for determining a facial expression based on the corrected facial expression feature. A face area detection unit that detects an image area including at least a part of a face image from the attention image as a face area;
A basic face model that reproduces at least the position of a facial feature part based on a plurality of basic feature quantities, and a variation in the position of the feature part according to at least a facial expression based on a plurality of facial expression feature quantities associated with the facial expression A setting unit for setting a facial model for facial expression that reproduces
A basic feature amount correction unit that corrects the basic feature amount so that the position of the feature portion reproduced by the basic face model approximates the face region;
The position of the feature part, which is reproduced by applying the corrected basic feature quantity to the basic face model, is further approximated to the face region when the position of the feature part is changed by the facial model for facial expression. A facial expression feature correction unit that corrects the facial expression feature,
A determination unit that determines a facial expression based on the corrected facial expression feature;
A printing apparatus comprising: a printing unit that prints the attention image including a face image determined to have a predetermined facial expression among the attention images.

Images