JP2010271956A - Image processing apparatus, image processing method, image processing program, and printer - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To determine a plurality of expressions with high accuracy. <P>SOLUTION: An image processing apparatus is configured to detect an image region including at least part of a face image from an image under consideration as a face region in the case of determining the expression of the face included in the image under consideration, to set a face model for reproducing at least the position of the feature site of the face on the basis of a plurality of feature values, to correct the feature values so that the position of the feature site reproduced by the face model can approximate the face region, to calculate a decision value for each of the plurality of expressions on the basis of the corrected feature values, and to determine the expression of the face by integrating the decision values of the respective expressions. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, an image processing program, and a printing apparatus that determine facial expressions included in an image.

  A printing apparatus that estimates facial expression based on facial feature information such as position, size, and ratio of eyes, nose, mouth, hairstyle, color, etc. is disclosed (see Patent Document 1, paragraph 0031). .

JP 2007-206921 A

However, when a plurality of facial expressions are mixed in a single face image (for example, a surprised smile), it is difficult to determine which facial expression.
The present invention has been made to solve the above problems, and provides an image processing apparatus, an image processing method, an image processing program, and a printing apparatus capable of accurately discriminating a plurality of facial expressions. With the goal.

  In order to solve at least a part of the problems, the present invention adopts the following aspects. That is, a face area detecting unit is provided, and the face area detecting unit detects an image area including at least a part of the face image as the face area from the attention image. The setting unit sets a face model that reproduces the position of the feature portion of the face based on a plurality of feature amounts. Further, the correction unit corrects the feature amount so that the position of the feature part reproduced by the face model approximates the face region. Then, the determination unit calculates a determination value indicating the likelihood of the expression for each of a plurality of expressions based on the corrected feature amount, and determines a facial expression by combining the determination values for each expression.

  Thus, by calculating the determination value indicating the expression-likeness for each of a plurality of facial expressions, the expression-likeness of each expression can be accurately evaluated. Further, since the determination value for each expression is comprehensively determined to determine which expression is present, the most appropriate expression can be determined even when a plurality of expressions are mixed. For example, by determining that the face area has an expression corresponding to the determination value that most strongly indicates the likelihood of expression, only the most appropriate expression can be determined even when a plurality of expressions are mixed. Note that the present invention is not limited to determining that a single facial expression is present. For example, when a determination value of a plurality of facial expressions exceeds a predetermined threshold, it is determined that the facial expression has a plurality of facial expressions. Good. The facial expression may be determined by comparing the determination value obtained by linearly combining at least two of the corrected feature quantities with a threshold value. By doing so, it is possible to calculate the determination value in consideration of a plurality of the feature amounts.

  In an image processing apparatus according to another aspect, at least one of the feature amounts is a coefficient of an eigenvector of each principal component obtained by performing principal component analysis on the positions of feature portions of a plurality of sample images. That is, a face model is prepared using principal component analysis, and the feature amount is set as a coefficient of an eigenvector of each principal component. Furthermore, the appearance of the face can be modeled by using at least one of the feature values as a coefficient of an eigenvector of each principal component obtained by principal component analysis of the color values of pixels of a plurality of sample images. .

  In the image processing device according to another aspect, when sequentially determining a plurality of facial expressions included in the plurality of attention images associated with different times, the attention image is determined to have a certain expression. If the face included in the pixel of interest associated with a later time is not determined to be the facial expression, the calculation of the determination value for the other facial expression and the determination based on the determination value are performed. Will not be performed. By doing in this way, the facial expression determination with respect to each frame, such as a moving image matched with different time, can be performed efficiently.

  Note that the present invention can be realized in various modes, for example, an image processing method and apparatus, a feature position detection method and apparatus, a facial expression determination method and apparatus, and the functions of these methods or apparatuses. For example, a recording medium storing the image processing program, a data signal including the computer program and embodied in a carrier wave, and the like.

FIG. 2 is an explanatory diagram schematically showing a configuration of a printer 100 as an image processing apparatus in the embodiment of the present invention. It is a flowchart which shows the flow of the AAM setting process in the said Example. It is explanatory drawing which shows an example of sample image SI. It is explanatory drawing which shows an example of the setting method of the feature point CP in the sample image SI. It is explanatory drawing which shows an example of the coordinate of the feature point CP set to sample image SI. It is explanatory drawing which shows an example of average shape s0. It is explanatory drawing which illustrated the relationship between shape vector s _i and shape parameter pi, and face shape s. It is explanatory drawing which shows an example of the method of the warp W of the sample image SI. It is explanatory drawing which shows an example of average face image _A0 (x). It is a flowchart which shows the flow of the face feature detection process in the said Example. It is explanatory drawing which shows an example of the detection result of the face area FA in the attention image OI. It is a flowchart which shows the flow of the initial position setting process of the feature point CP in the said Example. It is explanatory drawing which shows an example of the temporary setting position of the feature point CP by changing the value of a global parameter. It is explanatory drawing which shows an example of the average shape image I (W (x; p)). It is a flowchart which shows the flow of the feature point CP setting position correction process in the said Example. It is explanatory drawing which shows an example of the result of CP setting position correction processing. It is a flowchart which shows the flow of a texture detection process. It is a flowchart which shows the flow of a facial expression determination process. It is a graph which shows a judgment value. It is a flowchart which shows the flow of a judgment value setting process. It is a graph explaining the correlation of whether it is a feature amount and a happy face. It is a flowchart which shows the flow of the facial expression determination process concerning a modification. It is a flowchart which shows the flow of the facial expression determination process concerning a modification. It is a flowchart which shows the flow of the process which sets a facial expression parameter in a modification.

  Hereinafter, a printer which is an aspect of an image processing apparatus according to the present invention will be described based on examples with reference to the drawings.

A. Example:
A1. Configuration of image processing device:
FIG. 1 is an explanatory diagram schematically showing a configuration of a printer 100 as an image processing apparatus in the present embodiment of the present invention. The printer 100 of this embodiment is an ink jet color printer that supports so-called direct printing, in which an image is printed based on image data acquired from a memory card MC or the like. The printer 100 includes a CPU 110 that controls each unit of the printer 100, an internal memory 120 configured by a ROM and a RAM, an operation unit 140 configured by buttons and a touch panel, a display unit 150 configured by a liquid crystal display, and printing. A mechanism 160 and a card interface (card I / F) 170 are provided. The printer 100 may further include an interface for performing data communication with other devices (for example, a digital still camera or a personal computer). Each component of the printer 100 is connected via a bus so that bidirectional communication is possible.

  The printing mechanism 160 performs printing based on the print data. The card interface 170 is an interface for exchanging data with the memory card MC inserted into the card slot 172. In this embodiment, an image file including image data is stored in the memory card MC.

  The internal memory 120 stores an image processing unit 200, a display processing unit 310, and a print processing unit 320. The image processing unit 200 is a computer program, and performs facial feature detection processing and facial expression determination processing by being executed by the CPU 110 under a predetermined operating system. The face feature detection process is a process of detecting the position of a predetermined feature part (for example, the corner of the eye, the nose head, or the face line) in the face image. The face feature detection process will be described in detail later. The display processing unit 310 and the print processing unit 320 are also executed by the CPU 110 to realize the respective functions.

  The image processing unit 200 includes a setting unit 210, a feature position detection unit 220, a face area detection unit 230, a texture detection unit 240, and an expression determination unit 250 as program modules. The feature position detection unit 220 and the texture detection unit 240 include generation units 222 and 242, calculation units 224 and 244, and correction units 226 and 246, respectively. The facial expression determination unit 250 includes a selection unit 252, a determination value calculation unit 254, and a determination unit 256. The functions of these units will be described in detail in the description of the face feature detection process described later.

  The display processing unit 310 is a display driver that controls the display unit 150 to display processing menus, messages, images, and the like on the display unit 150. The print processing unit 320 is a computer program for generating print data from image data, controlling the printing mechanism 160, and printing an image based on the print data. The CPU 110 reads out and executes these programs (the image processing unit 200, the display processing unit 310, and the print processing unit 320) from the internal memory 120, thereby realizing the functions of these units.

  The internal memory 120 also stores AAM information AMI. The AAM information AMI is information set in advance by an AAM setting process described later, and is referred to in a face feature detection process described later. The contents of the AAM information AMI will be described in detail in the description of AAM setting processing described later.

A2. AAM setting process:
FIG. 2 is a flowchart showing the flow of AAM setting processing in this embodiment. The AAM setting process is a process for setting a shape model and a texture model used for modeling an image called AAM (Active Appearance Model). The shape model and the texture model correspond to a “face model” in the claims. In the present embodiment, the AAM setting process is performed by an operator using a computer. The computer has a CPU, a RAM, a ROM, an HDD, a display, an input device, and the like, which are connected by a bus. The CPU reads out a program recorded in the HDD and executes arithmetic processing according to the program, thereby causing the computer to execute each process described later. The computer that sets the shape model and the texture model corresponds to a “setting unit” in the claims.

  First, the operator prepares a plurality of images including a person's face as a sample image SI (step S110). FIG. 3 is an explanatory diagram showing an example of the sample image SI. As shown in FIG. 3, the sample image SI has various characteristics such as personality / race / gender, facial expression (happy face, sad face, surprised face, angry face), direction (front direction, upward direction, downward direction, right direction, left direction, etc.). Are prepared so as to include face images that are different from each other with respect to the attributes of. If the sample image SI is prepared in such a manner, any face image can be accurately modeled by AAM, and accurate face feature detection processing (described later) for any face image can be executed. Become. The sample image SI is also called a learning image.

  A feature point CP is set to the face image included in each sample image SI (step S120). FIG. 4 is an explanatory diagram showing an example of a method for setting the feature point CP in the sample image SI. The feature point CP is a point indicating the position of a predetermined feature part in the face image. In this embodiment, as predetermined feature parts, a predetermined position on the eyebrows in a person's face (for example, an end point or a four-divided point, the same applies hereinafter), a predetermined position on the eye contour, and a predetermined position on the nose and nose contours 68 parts such as a predetermined position on the contour of the upper and lower lips and a predetermined position on the contour of the face (face line) are set. That is, in the present embodiment, a predetermined position in the facial organs (eyebrows, eyes, nose, mouth) and facial contour that are commonly included in the human face are set as the characteristic parts. As shown in FIG. 4, the feature points CP are set (arranged) at positions representing 68 feature parts designated by the operator in each sample image SI. Since each feature point CP set in this way corresponds to each feature part, the arrangement of the feature points CP in the face image can be expressed as specifying the shape of the face.

  The position of the feature point CP in the sample image SI is specified by coordinates. FIG. 5 is an explanatory diagram showing an example of the coordinates of the feature point CP set in the sample image SI. In FIG. 5, SI (g) (g = 1, 2, 3...) Indicates each sample image SI, and CP (k) (k = 0, 1,..., 67) indicates each feature. Point CP is shown. CP (k) -X represents the X coordinate of the feature point CP (k), and CP (k) -Y represents the Y coordinate of the feature point CP (k). The coordinates of the feature point CP include predetermined reference points (for example, in the sample image SI normalized with respect to the size of the face, the inclination of the face (inclination in the image plane), and the position of the face in the X direction and the Y direction) The coordinates with the origin at the lower left point of the image are used. Further, in the present embodiment, a case where a plurality of human face images are included in one sample image SI is allowed (for example, two sample face images are included in the sample image SI (2)). Each person in one sample image SI is specified by a person ID. Each sample image SI, the coordinates of each feature point CP, and the person ID are input to the computer.

  Subsequently, the operator sets the AAM shape model (step S130). Specifically, the computer performs principal component analysis on a coordinate vector (see FIG. 5) composed of the coordinates (X coordinate and Y coordinate) of 68 feature points CP in each sample image SI, and the computer further uses the feature points. The face shape s specified by the position of the CP is modeled by the following equation (1). The shape model is also referred to as a feature point CP arrangement model.

In the formula (1), s ₀ is an average shape. FIG. 6 is an explanatory diagram showing an example of the average shape s ₀ . As shown in FIGS. 6A and 6B, the average shape s ₀ is a model representing the average face shape specified by the average position (average coordinate) for each feature point CP of the sample image SI. is there. In the present embodiment, in the average shape s ₀ , a region surrounded by a straight line connecting feature points CP (face lines, eyebrows, feature points CP corresponding to the eyebrows, see FIG. 4) located on the outer periphery (FIG. 6 ( b) is indicated by hatching) and is referred to as “average shape region BSA”. As shown in FIG. 6A, the computer sets a plurality of triangular areas TA having the feature points CP as vertices so as to divide the average shape area BSA into a mesh shape with respect to the average shape s ₀ .

In the equation (1) representing the shape model, s _i is a shape vector, and p _i is a shape parameter representing the weight of the shape vector s _i . The shape vector s _i is a vector representing the characteristics of the face shape s and is an eigenvector corresponding to the i-th principal component obtained by principal component analysis. As shown in the equation (1), in the shape model in the present embodiment, the face shape s representing the arrangement of the feature points CP is the model as the sum of the average shape s ₀ and the linear combination of the n shape vectors s _i. It becomes. As shown in the equation (1a), it is possible to reproduce the face shape s in any image by appropriately setting the shape parameter p _i in the shape model. Further, as shown in the equation (1b), by appropriately setting the shape parameter p _i in the shape model, it is possible to reversely convert from any shape s to the average shape s ₀ .

FIG. 7 is an explanatory diagram illustrating the relationship between the shape vector s _{i, the} shape parameter p _i, and the face shape s. As shown in FIG. 7 (a), in order to identify the face shape s, the computer selects the number n (shown in the order from the highest cumulative contribution ratio in order from the eigenvector corresponding to the principal component having the larger contribution ratio. 7, the eigenvector of the top four of the n principal components is shown as the shape vector s _i . Each of the shape vectors s _i corresponds to the movement direction / movement amount of each feature point CP, as indicated by the arrows in FIG. In this embodiment, most first shape vector s ₁ corresponding to the first principal component having a large contribution rate is a vector that is approximately correlated with the left and right appearance of a face, by the magnitude of the shape parameter p _1, 7 As shown in (b), the horizontal face direction of the face shape s changes. Second shape vector s ₂ corresponding to the large second principal component of the contribution rate to the second is a vector that is approximately correlated with the vertical appearance of a face, by the magnitude of the shape parameter p _2, FIG. 7 (c) As shown in FIG. 4, the vertical face direction of the face shape s changes. The third shape vector s ₃ corresponding to the third principal component having the third largest contribution ratio is a vector that is substantially correlated with the aspect ratio of the face shape, and the fourth principal component having the fourth largest contribution ratio. The fourth shape vector s ₄ corresponding to is a vector that is substantially correlated with the degree of mouth opening. As described above, the value of the shape parameter represents the feature of the face image such as facial expression and face orientation. The “shape parameter” in this embodiment corresponds to the “feature amount” in the claims.

The average shape s ₀ and the shape vector s _i set by the computer in the shape model setting step (step S130) are stored in the internal memory 120 of the printer 100 as AAM information AMI (FIG. 1). Subsequently, the computer sets an AAM texture model (step S140). Specifically, first, for each sample image SI, image conversion (hereinafter referred to as “warp”) is performed so that the setting position of the feature point CP in the sample image SI is equal to the setting position of the feature point CP in the average shape s ₀ . Also called “W”).

FIG. 8 is an explanatory diagram showing an example of a method of warping W of the sample image SI. In each sample image SI, similarly to the average shape s ₀ , the computer sets a plurality of triangular regions TA that divide the region surrounded by the feature points CP located on the outer periphery into a mesh shape. The warp W is a set of affine transformations for each of the plurality of triangular areas TA. That is, in the warp W, an image of a certain triangular area TA in the sample image SI is affine transformed into an image of the corresponding triangular area TA in the average shape s ₀ . The computer generates a sample image SI (hereinafter referred to as “sample image SIw”) in which the setting position of the feature point CP is equal to the setting position of the feature point CP in the average shape s ₀ by the warp W.

  Each sample image SIw has a rectangular frame containing an average shape area BSA (shown with hatching in FIG. 8) as an outer periphery, and an area other than the average shape area BSA (hereinafter also referred to as “mask area MA”). Generated as a masked image. An image area in which the average shape area BSA and the mask area MA are combined is referred to as a reference area BA. In addition, the computer normalizes each sample image SIw as an image having a size of, for example, 56 pixels × 56 pixels.

Next, the computer performs a principal component analysis on a luminance value vector composed of luminance values in each pixel group x of each sample image SIw, and a facial texture (also referred to as “appearance”) A (x) is expressed as follows: Modeled by equation (2). The pixel group x is a set of pixels located in the average shape area BSA.

In the formula (2), A ₀ (x) is an average face image. FIG. 9 is an explanatory diagram showing an example of the average face image A ₀ (x). The average face image A ₀ (x) is an image representing the average face of the sample image SIw (see FIG. 8) after the warp W. That is, the average face image A ₀ (x) is an image calculated by taking the average of the pixel values (luminance values) of the pixel group x in the average shape area BSA of the sample image SIw. Therefore, the average face image A ₀ (x) is a model representing the average face texture (appearance) in the average face shape. Note that the average face image A ₀ (x) is composed of the average shape area BSA and the mask area MA, similarly to the sample image SIw, and is calculated as an image having a size of 56 pixels × 56 pixels, for example.

In the equation (2) representing the texture model, a _k (x) is a texture vector, and λ _k is a texture parameter representing the weight of the texture vector a _k (x). The texture vector a _k (x) is a vector representing the characteristics of the facial texture A (x), and specifically, is an eigenvector corresponding to the j-th principal component obtained by principal component analysis. That is, the number m of eigenvectors set based on the cumulative contribution rate is adopted as the texture vector a _k (x) in order from the eigenvector corresponding to the principal component having the larger contribution rate. In the present embodiment, the first texture vector A ₁ (x) corresponding to the first principal component having the largest contribution rate is a vector that is substantially correlated with a change in face color (also regarded as a gender difference).

As shown in the equation (2), in the texture model in the present embodiment, the facial texture A (x) representing the appearance of the face is the average face image A ₀ (x) and m texture vectors a _k (x ) And the linear combination. As shown in the equation (2a), it is possible to reproduce the facial texture A (x) in any image by appropriately setting the texture parameter λ _k in the texture model. On the contrary, as shown in the equation (2b), by appropriately setting the texture parameter λ _k in the texture model, the face texture A (x) in every image is reversed to the average face image A ₀ (x). Can be converted.

The average face image A ₀ (x) and the texture vector a _k (x) set in the texture model setting step (step S140 in FIG. 2) are stored in the internal memory 120 of the printer 100 as AAM information AMI (FIG. 1). Stored. The computer records the AAM information AMI on any recording medium, and when the printer 100 is manufactured, the internal memory 120 that records the AAM information AMI read from the recording medium is incorporated in the printer 100 or rarely incorporated in the printer 100. The AAM information AMI read from the recording medium is recorded in the internal memory 120.

Through the processing so far, a shape model for modeling the shape of the face and a texture model for modeling the texture of the face are set. By combining the set shape model and the texture model, that is, the synthesized texture A (x) is converted from the average shape s ₀ to the shape s (inverse conversion of the warp W shown in FIG. 8). Thus, it is possible to reproduce the shape and texture of any face image.

A3. Face feature detection processing:
FIG. 10 is a flowchart illustrating a flow of face feature detection processing in the embodiment. In the face feature detection processing in this embodiment, the feature point CP setting position correction processing (step S250) for determining the arrangement of the feature points CP in the face image included in the target image using AAM is performed. In this process, the position is detected, and the texture of the face is further detected by the texture detection process (step S260). As described above, in the present embodiment, in the AAM setting process (FIG. 2), a total of 68 predetermined positions in the human facial organs (eyebrows, eyes, nose, mouth) and facial contour are set as the characteristic parts. (See FIG. 4). Therefore, in the face feature detection process of the present embodiment, the arrangement of 68 feature points CP indicating predetermined positions in the human face organ and face contour is determined.

When the arrangement of the feature points CP in the face image is determined by the face feature detection process, the value of the shape parameter p _i for the face image is specified. Accordingly, the processing result of the face feature detection process can be used for face orientation determination for detecting a face image in a specific direction (for example, rightward or downward), face deformation for deforming the face shape, face shading correction, and the like. .

  First, the image processing unit 200 (FIG. 1) acquires image data representing an attention image that is a target of face feature detection processing (step S210). In the printer 100 of the present embodiment, when the memory card MC is inserted into the card slot 172, thumbnail images of image files stored in the memory card MC are displayed on the display unit 150. One or more images to be processed are selected by the user via the operation unit 140. The image processing unit 200 acquires an image file including image data corresponding to one or more selected images from the memory card MC and stores it in a predetermined area of the internal memory 120. Note that the acquired image data is referred to as attention image data, and an image represented by the attention image data is referred to as attention image OI.

  The face area detection unit 230 (FIG. 1) detects an image area including at least a part of the face image included in the target image OI as the face area FA (step S230). The detection of the face area FA can be performed using a known face detection method. Known face detection methods include, for example, pattern matching methods, skin color region extraction methods, learning using sample images (for example, learning using neural networks, learning using boosting, and support vector machines). For example, a method using learning data set by learning).

  FIG. 11 is an explanatory diagram illustrating an example of a detection result of the face area FA in the target image OI. FIG. 11 shows the face area FA detected in the target image OI. In the present embodiment, a face detection method is used in which a rectangular area including the vertical direction of the face from the forehead to the chin and the horizontal direction from the outside of both ears is detected as the face area FA.

  The setting unit 210 (FIG. 1) sets the initial position of the feature point CP in the target image OI (Step S240). FIG. 12 is a flowchart showing a flow of initial position setting processing of the feature point CP in the present embodiment. In the present embodiment, the setting unit 210 changes the values of global parameters representing the size, inclination, and position (vertical position and horizontal position) of the face image with respect to the face area FA to change the feature point CP. The temporary setting position on the attention image OI is set (step S310).

FIG. 13 is an explanatory diagram showing an example of a temporary setting position of the feature point CP by changing the value of the global parameter. 13A and 13B show a mesh formed by connecting feature points CP and feature points CP in the target image OI. As shown in the center of FIGS. 13A and 13B, the setting unit 210 temporarily sets the feature point CP (hereinafter, referred to as an average shape s _{0) at} which the average shape s ₀ is formed at the center of the face area FA. (Also referred to as “reference temporary setting position”).

  The setting unit 210 also sets a plurality of temporary setting positions obtained by changing various global parameter values with respect to the reference temporary setting position. Changing the global parameters (size, inclination, vertical position and horizontal position) means that the mesh formed by the feature point CP in the target image OI is enlarged / reduced, the inclination is changed, and the parallel movement is performed. Equivalent to. Accordingly, as shown in FIG. 13 (a), the setting unit 210 forms a temporary setting position that forms a mesh obtained by enlarging or reducing the reference temporary setting position mesh at a predetermined magnification (below the reference temporary setting position diagram and Or a temporary setting position (shown on the right and left in the drawing of the reference temporary setting position) that forms a mesh whose inclination is changed clockwise or counterclockwise by a predetermined angle. Further, the setting unit 210 forms a temporary setting position (upper left of the figure of the reference temporary setting position, which forms a mesh obtained by performing a combination of enlargement / reduction and change in inclination with respect to the mesh of the reference temporary setting position. Also set in the lower left, upper right, and lower right).

  Also, as shown in FIG. 13B, the setting unit 210 forms a temporary setting position (a reference temporary setting position diagram) that forms a mesh that has been translated upward or downward by a predetermined amount from the reference temporary setting position mesh. And a temporary setting position (shown on the left and right in the drawing of the reference temporary setting position) that forms a mesh moved in parallel to the left or right. Further, the setting unit 210 forms a temporary setting position (upper left and lower left in the figure of the reference temporary setting position) that forms a mesh obtained by performing a combination of vertical and horizontal parallel movements on the mesh of the reference temporary setting position. , Shown in the upper right and lower right).

  The setting unit 210 also has temporary setting positions at which parallel movement in the vertical and horizontal directions shown in FIG. 13B is executed with respect to the mesh at each of the eight temporary setting positions other than the reference temporary setting position shown in FIG. Set. Therefore, in the present embodiment, the four global parameters (size, inclination, vertical position, horizontal position) are set in 80 ways (= 3 × 3 × 3) that are set as known three-level values. A total of 81 temporary setting positions of (3-1) temporary setting positions and reference temporary setting positions are set.

The generation unit 222 (FIG. 1) generates an average shape image I (W (x; p)) corresponding to each set temporary setting position (step S320). FIG. 14 is an explanatory diagram showing an example of the average shape image I (W (x; p)). The average shape image I (W (x; p)) is calculated by conversion such that the arrangement of the feature points CP in the input image is equal to the arrangement of the feature points CP in the average shape s ₀ .

The transformation for calculating the average shape image I (W (x; p)) is a warp W that is a set of affine transformations for each triangular area TA, similarly to the transformation for calculating the sample image SIw (see FIG. 8). Is done. Specifically, the average shape region BSA (region surrounded by the feature points CP located on the outer periphery) is specified by the feature points CP (see FIG. 13) arranged in the attention image OI, and the average shape region in the attention image OI. An average shape image I (W (x; p)) is calculated by performing affine transformation for each triangular area TA on the BSA. In this embodiment, the average shape image I (W (x; p) ) is composed of an average face image A ₀ (x) and an average shape area BSA and a mask area MA, average face image A ₀ (x) And is calculated as an image of the same size.

As described above, the pixel group x is a set of pixels located in the average shape region BSA in the average shape s ₀ . The pixel group in the image (average shape area BSA of the target image OI) before execution of the warp W corresponding to the pixel group x in the image after execution of the warp W (face image having the average shape s ₀ ) is expressed as W (x; p). To express. Since the average shape image is an image composed of luminance values in each of the pixel groups W (x; p) in the average shape region BSA of the target image OI, it is represented as I (W (x; p)). FIG. 14 shows nine average shape images I (W (x; p)) corresponding to the nine temporarily set positions shown in FIG.

The calculation unit 224 (FIG. 1) calculates an average shape image I (W (x; p)) corresponding to each temporarily set position and an average face image A ₀ (x) as a “comparison face image” in the claims. The difference image Ie is calculated (step S330). Since 81 types of temporary setting positions of the feature points CP are set, the calculation unit 224 (FIG. 1) calculates 81 difference images Ie.

The setting unit 210 calculates a norm from the pixel values of each difference image Ie, and sets a temporary installation position corresponding to the difference image Ie having the smallest norm value (hereinafter also referred to as “norm minimum temporary setting position”) as the target image OI. Is set as the initial position of the feature point CP at (step S340). The pixel value for calculating the norm may be a luminance value or an RGB value. Thus, the feature point CP initial position setting process is completed. Note that the smaller the norm value, the smaller the difference between the average shape image I (W (x; p)) and the average face image A ₀ (x) (both approximate).

A3-1. Feature point CP setting position correction processing:
When the feature point CP initial position setting process is completed, the feature position detector 220 (FIG. 1) corrects the set position of the feature point CP in the target image OI (step S250). FIG. 15 is a flowchart showing the flow of the feature point CP setting position correction process in the present embodiment.

  The generation unit 222 (FIG. 1) calculates an average shape image I (W (x; p)) from the attention image OI (step S410). The calculation method of the average shape image I (W (x; p)) is the same as that in step S320 in the feature point CP initial position setting process.

The feature position detection unit 220 calculates a difference image Ie between the average shape image I (W (x; p)) and the average face image A ₀ (x) (step S420). The feature position detection unit 220 determines whether or not the feature point CP setting position correction process has converged based on the difference image Ie (step S430). The feature position detection unit 220 calculates the norm of the difference image Ie, determines that it has converged if the norm value is smaller than a preset threshold value, and still converges if the norm value is greater than or equal to the threshold value. Judge that there is no. The feature position detection unit 220 determines that the calculated norm value of the difference image Ie has converged when the value is smaller than the value calculated in the previous step S430, and still determines that the value is equal to or greater than the previous value. It is good also as what determines with not having converged. Alternatively, the feature position detection unit 220 may perform the convergence determination by combining the determination based on the threshold and the determination based on comparison with the previous value. For example, the feature position detection unit 220 determines that the calculated norm value has converged only when the calculated norm value is smaller than the threshold value and smaller than the previous value, and otherwise determines that it has not yet converged. It may be a thing.

If it is determined in step S430 that the convergence is not yet completed, the correction unit 226 (FIG. 1) calculates the parameter update amount ΔP (step S450). The parameter update amount ΔP includes four global parameters (total size, inclination, X direction position, Y direction position), and n shape parameters p _i (i = 1 to n) as feature amounts. .) Means the change amount of the value. Immediately after setting the feature point CP to the initial position, the global parameter is set to the value determined in the feature point CP initial position setting process (FIG. 12). Further, difference between the initial position of the characteristic point CP and sets the position of the characteristic points CP of the average shape s ₀ in this case, the overall size, the tilt, because it is limited to the difference in position, shape parameters in the shape model p The value of _i and the texture parameter λ _k in the texture model are all zero.

  The parameter update amount ΔP is calculated by the following equation (3). That is, the parameter update amount ΔP is a product of the update matrix R and the difference image Ie.

The update matrix R in the equation (3) is a matrix of M rows and N columns set in advance by learning in order to calculate the parameter update amount ΔP based on the difference image Ie.
It is stored in the internal memory 120 as AAM information AMI (FIG. 1). In this embodiment, the number of rows M in the update matrix R is equal to the sum ((4 + n)) of the number of global parameters (4) and the number of shape parameters p _i (n), and the number of columns N is , average face image a ₀ average number of pixels in the shape area BSA of (x) - is equal to (56 pixels × 56 pixels the number of pixels in the mask area MA). The update matrix R is calculated by the following formulas (4) and (5).

The correction unit 226 (FIG. 1) updates the parameters (four global parameters and n shape parameters p _i ) based on the calculated parameter update amount ΔP (step S460). Thereby, the setting position of the feature point CP in the target image OI is corrected. The correcting unit 226 corrects so that the norm of the difference image Ie is reduced. After the parameter update, the average shape image I (W (x; p)) is calculated again from the target image OI in which the installation position of the feature point CP is corrected (step S410), and the difference image Ie is calculated (step S420), a convergence determination based on the difference image Ie (step S430) is performed. The average shape image I (W (x; p)) changes according to the update (change) of the shape parameter p _i and corresponds to a “face region” in the claims. If it is determined that the convergence has not occurred in the convergence determination again, the parameter update amount ΔP based on the difference image Ie is calculated (step S450), and the setting position correction of the feature point CP by the parameter update (step S460). ) Is performed.

When the processing from step S410 to step S460 in FIG. 15 is repeatedly executed, the position of the feature point CP corresponding to each feature part in the target image OI and the position of the actual feature part approach as a whole and converge at a certain time. In the determination (step S430), it is determined that convergence has occurred. If it is determined in the convergence determination that convergence has been achieved, the face feature detection process is completed (step S470). The setting position of the feature point CP specified by the global parameter and the shape parameter p _i set at this time is specified as the setting position of the feature point CP in the final target image OI.

  FIG. 16 is an explanatory diagram illustrating an example of a result of the feature point CP setting position correction process. FIG. 16 shows the setting positions of the feature points CP finally specified in the target image OI. Since the position of the feature point CP specifies the position of the facial feature part (predetermined position in the facial organs (eyebrows, eyes, nose, mouth) and facial contour) included in the attention image OI. It is possible to detect the shape and position of the human face organ and the face contour shape in the image OI.

  When the feature point CP setting position correction processing is completed, the texture detection unit 240 (FIG. 1) corrects (detects) the texture (appearance) of the average shape image I (W (x; p)) set in the target image OI. Is performed (step S260).

A3-2. Texture detection processing:
FIG. 17 is a flowchart showing the flow of texture detection processing. The generation unit 242 (FIG. 1) calculates an average shape image I (W (x; p)) from the attention image OI (step S510). The calculation method of the average shape image I (W (x; p)) is the same as that in step S320 in the feature point CP initial position setting process and step S410 in the feature point CP setting position correction process. position correction processing completion time using the values of the global parameters and the shape parameters p _i of the average shape image I (W (x; p) ) is calculated.

前記式（６）は、前記式（２ｂ）における顔のテクスチャーＡ（ｘ）にＷ（ｘ；ｐ）を代入したものであり、テクスチャーパラメーターλ_kが適切に設定されれば、Ｔ（ｘ；λ）が平均顔画像Ａ₀（ｘ）と等しくなる。 Further, the generation unit 242 of the texture detection unit 240 calculates an average texture image J (T (x; λ)) based on the average shape image I (W (x; p)) (step S515). The average texture image J (T (x; λ)) is calculated by the following equation (6).

The equation (6) is obtained by substituting W (x; p) into the facial texture A (x) in the equation (2b). If the texture parameter λ _k is appropriately set, T (x; λ) is equal to the average face image A ₀ (x).

Then, the calculation unit 244 calculates a difference image Ie between the average texture image J (T (x; λ)) and the average face image A ₀ (x) (step S520). The texture detection unit 240 determines whether the texture detection process has converged based on the difference image Ie (step S530). The texture detection unit 240 calculates the norm of the difference image Ie, determines that it has converged if the norm value is smaller than a preset threshold value, and has not yet converged if the norm value is greater than or equal to the threshold value. Is determined. Note that the texture detection unit 240 determines that the calculated norm value of the difference image Ie has converged if it is smaller than the value calculated in the previous step S530, and still converges if it is greater than or equal to the previous value. It is good also as what determines with not doing. The convergence determination can be performed by the same method as in step S430 in the feature point CP setting position correction process.

If it is determined in step S530 that the convergence has not yet been completed, the correction unit 246 (FIG. 1) calculates the parameter update amount Δλ (step S550). The parameter update amount ΔΛ means a change amount of the value of m texture parameters λ _k (k = _{1 to} m), which is a feature amount. The parameter update amount ΔΛ is calculated by the following equation (7). That is, the parameter update amount ΔΛ is a product of the update matrix U and the difference image Ie.

The update matrix U in Equation (7) is a matrix of M rows and N columns set in advance by learning in order to calculate the parameter update amount ΔΛ based on the difference image Ie.
It is stored in the internal memory 120 as AAM information AMI (FIG. 1). In this embodiment, the number M of rows in the update matrix U is equal to the texture parameter λ _k (m), and the number N of columns is the number of pixels in the average shape area BSA (56 pixels × 56 pixels−pixels in the mask area MA). Number). The update matrix U is calculated by the following formulas (8) and (9).

The correction unit 246 (FIG. 1) updates the texture parameter λ _k based on the calculated parameter update amount ΔΛ (step S560). Thereby, the facial texture A (x) is corrected. That is, the correction unit 246 corrects so that the norm of the difference image Ie is reduced. After the parameter update, the average texture image J (T (x; λ)) is calculated again based on the corrected texture parameter λ _k (step S515), the difference image Ie is calculated (step S520), and the difference image. A convergence determination (step S530) based on Ie is performed. If it is determined that the convergence has not been made in the convergence determination again, the parameter update amount ΔΛ based on the difference image Ie is calculated (step S550), and the texture is corrected by updating the parameter (step S560). .

When the processing from step S510 to S560 in FIG. 17 is repeatedly executed, the luminance value (color value) of each pixel of the facial texture A (x) approaches the luminance value of the actual face image as a whole, At a certain point in time, it is determined that convergence has been made in the convergence determination (step S530). If it is determined in the convergence determination that the image has converged, the texture detection process is completed (step S570). The facial texture A (x) specified by the value of the texture parameter λ _k set at this time is specified as the final facial appearance. Based on the facial texture A (x), facial color correction and the like can be performed. When the shape parameter p _i and the texture parameter λ _k for the face area FA in the target image OI can be specified as described above, the facial expression determination process is executed next.

A4. Facial expression determination processing:
FIG. 18 is a flowchart showing the flow of facial expression determination processing. The facial expression determination unit 250 selects a facial expression to be determined (step S610). In this embodiment, four types of facial expressions (a happy face, a sad face, a surprised face, and an angry face) are set as determination targets, and a determination target is selected in the order of a happy face → a sad face → a surprise face → an angry face. The selection unit 252 selects the h facial expression parameters q _l (used to calculate a judgment value E indicating the facial expression of the selected facial expression from the n shape parameters p _i and the m texture parameters λ _k. 1 is an integer from 1 to h.) (Step S620). Determination value calculation unit 254 calculates determination value E by substituting facial expression parameter q _l into the following equation (10) (step S630).

In the equation (10), α _l represents a weighting coefficient set for each facial expression parameter q _l , and kn represents a coefficient for normalization. That is, the determination value E is a value obtained by linearly combining the facial expression parameter q _l with a weight based on the weight coefficient α _l . When the determination value E is calculated, the facial expression determination unit 250 determines whether or not selection of all facial expressions has been completed (step S640). If not, the process returns to step S610 to select the next facial expression. . As a result, the determination value E for the happy face, the determination value E for the sad face, the determination value E for the surprised face, and the determination value E for the angry face can be calculated in order. For each facial expression, the h facial expression parameters q _l selected from the n shape parameters p _i and the m texture parameters λ _k are different, and the weighting coefficient corresponding to the facial expression parameter q _l α _{l is} also prepared for each facial expression. For example, the sign of the weight coefficient α _l for the facial expression parameter q _l related to the mouth angle is inverted between a happy face that tends to increase the mouth angle and a sad face that tends to decrease the mouth angle. Therefore, the determination value E is calculated based on a different calculation formula for each facial expression. As described above, since the judgment value E is calculated using the h facial expression parameters q _l having a high correlation with each facial expression among the geometric parameters _pi and the texture parameters λ _k , the geometrical and color (luminance) viewpoints are obtained. The facial expression can be accurately determined based on the above.

The facial expression parameter q _l selected corresponding to each facial expression is a vector element having a high correlation with each facial expression out of the shape parameter p _i and the texture parameter λ _k and is set by a judgment value setting process to be described later. It is stored in the internal memory 120 as information EMI (FIG. 1). When the determination value E can be calculated for each facial expression, the determination unit 256 determines the facial expression by combining the determination values E for each facial expression. Specifically, the determination unit 256 detects a facial expression with the largest determination value E (step S650). Next, the determination unit 256 determines whether or not the largest determination value E is greater than a predetermined threshold (step S660), and determines that the expression has the expression of the determination value E if greater than the threshold. (Step S670). On the other hand, if it is equal to or less than the largest determination value E, it is determined that no facial expression is present (step S680). In this embodiment, after tag information for identifying each expression is attached to the attention image OI, if the attention image OI includes a happy face, the attention image OI is output to the printing mechanism 160 and printing is performed ( Step S690).

  FIGS. 19A to 19D are graphs for comparing the determination values E of the facial expressions. If the determination value E for any facial expression does not exceed the threshold value as shown in FIG. 19A, it is determined that no facial expression is present (step S680). As shown in FIG. 19B, when the determination value E of any one facial expression exceeds the threshold value, it is determined that the facial expression value exceeds the threshold value. As shown in FIGS. 19C and 19D, when the determination value E of any two or more facial expressions exceeds the threshold value, the determination value E is the maximum among the facial expressions whose determination value E exceeds the threshold value. It is determined that it is a facial expression. Thus, by calculating the determination value E for each facial expression, it is possible to accurately determine whether each facial expression is present. Further, since it is determined that the expression corresponding to the largest determination value E is present, the most appropriate expression can be specified.

A5. Judgment value setting processing:
FIG. 20 is a flowchart showing the flow of the determination value setting process. Similar to the AAM setting process, the determination value setting process is a process executed by the computer prior to manufacturing the printer 100. First, a known sample image SI is prepared for whether it is a happy face, a sad face, a surprised face, or an angry face (step S710). For example, the operator determines which facial expression each sample image SI has, and the result is associated with each sample image SI and input to the computer. Next, the computer uses each sample image SI as the target image OI, and sequentially executes the same processing as the above-described feature point CP initial position setting processing, feature point CP setting position correction processing, and texture detection processing, thereby each sample image SI. the image SI and calculates a shape parameter p _i and texture parameter lambda _k (step S720). When the shape parameter p _i and the texture parameter λ _k are obtained, the computer selects a facial expression for which the facial expression parameter q _l is to be set (step S730). Again, the determination target is selected in the order of happy face → sad face → surprise face → angry face. If the facial expression can be selected, the computer calculates a correlation coefficient between each element of the shape parameter p _i and the texture parameter λ _k with respect to whether or not it is the selected facial expression (step S740).

FIG. 21 is a graph for explaining the correlation of whether or not the face is a happy face. The horizontal axis and the vertical axis in the figure indicate the values (0 to 100%) of the shape parameters p ₈ and p ₉ as examples of the shape parameter p _i . In the graph, which plots the shape parameter p _8, p ₉ of each sample image SI, what is Hee face indicated by ○, those that are not Hee face is indicated by ●. As shown in the figure, the smaller the value of the shape parameter p ₈ , the greater the distribution of the happy faces. Therefore, whether or not it is a happy face has a negative correlation with the shape parameter p ₈ . On the other hand, with respect to the shape parameters p _9, as is Hee face, has been substantially uniformly distributed not comply with Hee face, does not have a correlation. As an example, only the shape parameters p ₈ and p ₉ are listed, and all other shape parameters p _i and texture parameters λ _k are similarly investigated for correlation with facial expressions.

Here, the computer calculates the correlation coefficient by a known calculation formula, and expresses the shape parameter p _i and the texture parameter λ _k whose absolute value exceeds a predetermined threshold (for example, 0.5, etc.) The parameter is q _l (step S750). Therefore, the number (h) of elements of the facial expression parameter q _l is not necessarily the same for each facial expression. Of course, the number (h) of elements of the facial expression parameter q _l is set in advance, and the shape parameter p _i and the texture parameter λ _k having the highest absolute value of the correlation coefficient are set as the facial expression parameter q _l. It may be. Furthermore, the ratio of the number of elements of the shape parameter p _i and the texture parameter λ _k among the number (h) of elements of the facial expression parameter q _l may be defined. For example, the number of elements of the shape parameter p _i and texture parameter lambda _k is 1: be added to 1 and so as to limit, it is possible to evaluate a balanced shape and color (luminance) point of view. Here, the shape parameter p _i and the texture parameter λ _k having a high correlation coefficient are set as the expression parameter q _l . For example, in the case of a happy face, the shape parameter p _i related to the angle of the corner of the eye or the mouth angle is related to laughter. It is predicted that the texture parameter λ _k is set as the facial expression parameter q _l .

Next, the computer sets a value proportional to the calculated correlation coefficient as the weighting coefficient α ₁ (step S760). Accordingly, the facial expression parameters q _l having a positive correlation is set to a positive weight coefficient alpha _l, negative weight factor alpha _l is to be set for the expression parameter q _l having a negative correlation. If the weighting factor α _l for the selected facial expression can be set, the computer determines whether or not selection of all facial expressions has been completed (step S770), and returns to step S730 if not complete. As described above, when the facial expression parameter q _l and the weighting coefficient α _l can be set for all facial expressions, the computer determines that the maximum value of the judgment value E calculated by the equation (10) is between each facial expression. A coefficient kn for each facial expression is set so as to match (step S780). Thus, all the information used in the equation (10) has been set, and these pieces of information are stored in the internal memory 120 as the facial expression determination information EMI when the printer 100 is manufactured (step S790).

B. Variation:
The present invention is not limited to the above-described examples and embodiments, and can be implemented in various modes without departing from the gist thereof. For example, the following modifications are possible.

B1. Modification 1:
In the above-described embodiment, the sample images SI are prepared so as to include different face images with respect to various attributes such as personality / race / gender, facial expression (happy face, sad face, surprised face, angry face), and orientation. However, the composition ratio of the sample images SI having different attributes of expression (happy face, sad face, surprise face, angry face) may be increased. By increasing the composition ratio of the sample image SI with different facial expression attributes, the contribution ratio of the happy face, sad face, surprise face, and angry face attributes to other attributes such as personality, race, gender, and orientation On the other hand, since it becomes relatively high, in the principal component analysis, a principal component showing a certain degree appears as a happy face, a sad face, a surprised face, and an angry face. That is, any element of shape parameter p _i or texture parameter lambda _k is whether it Hee face, whether悲顔, whether Odorokikao, whether Okokao The degree can be shown.

When the determination value setting process is performed in the same manner as in the embodiment, as the component ratio of the sample image SI having different facial expression attributes is increased, either the shape parameter _pi or the texture parameter λ _k Correlation coefficient with facial expression increases. When the correlation coefficient between any element of the shape parameter p _i or the texture parameter λ _k and each facial expression is very close to 1 or −1 (for example, when the absolute value is 0.9 or more). Each facial expression can be determined by comparing the value of the element with a threshold value. For example, by comparing the shape parameter p ₁ corresponding to the first principal component with a threshold value, it is determined whether or not the face is a happy face, while comparing the shape parameter p ₂ corresponding to the second principal component with the threshold value. Thus, it can be determined whether or not it is a sad face. In addition, since it is necessary to detect feature points for various faces, it is desirable to include sample images SI having different personality, race, gender, and orientation attributes to some extent. In addition, if the detection of feature points is performed separately, the AAM information AMI may be set using only sample images SI having different facial expression attributes (happy face, sad face, surprise face, angry face). Is possible.

B2. Modification 2:
In the above embodiment, a single still image is set as the target image OI. However, a series of still images taken by a continuous shooting function of a moving image or a digital still camera in which images are continuously input in time series are illustrated. The target image OI may be sequentially set. For example, a frame with a significant expression can be detected from each frame of the moving image and the frame can be printed by the printing mechanism 160. For each frame, the same facial feature detection processing and facial expression determination processing as in the above embodiment may be executed, but in this modification, facial expression determination processing described below is executed.

  FIG. 22 is a flowchart showing the flow of facial expression determination processing according to the present modification. When selecting the facial expression to be determined, the facial expression determination unit 250 determines whether or not the face existing in the immediately preceding frame has been determined to be any facial expression (step S602). In the following description, it is assumed that each frame has only a single face. If it is not determined that the face existing in the immediately preceding frame is any facial expression, in the same manner as in the above embodiment, selection is made in the order of happy face → sad face → surprise face → angry face, and determination value E The facial expression having the largest value and larger than the threshold value is determined (steps S610 to S690 similar to the above embodiment). On the other hand, when it is determined that the face existing in the immediately preceding frame is any facial expression, the facial expression is selected (step S604).

Then, h facial expression parameters q _l are selected for the facial expression (step S622). The determination value E is calculated by substituting the facial expression parameter q _l into the above equation (10) (step S632). If the determination value E is larger than the threshold value (the same as in the above embodiment) (step S662), it is determined that the facial expression is selected in step S604 (step S672). On the other hand, if the determination value E is less than or equal to the threshold value, step S610 is executed, and steps S610 to S680 similar to those in the above embodiment are sequentially executed. However, since the determination of the determination value E has already been completed for the expression in the immediately preceding frame, the expression in the immediately preceding frame is not selected in step S610.

  It can be considered that each frame of a moving image input in an extremely short time is highly likely to have a face with the same expression as the previous frame. By using the flow of this modification, as long as the same facial expression continues, only the calculation of the judgment value E and the threshold value are determined for that facial expression. Since the determination value E is calculated for all facial expressions when it is no longer determined that the previous facial expression is determined, efficient facial expression determination can be performed on the moving image. Of course, the same effect can be obtained even with a still image taken in a very short time by the continuous shooting function. Furthermore, when a plurality of faces are included in a single target image OI, it can be estimated that each has the same expression. Therefore, the determination regarding the facial expression determined for the face for which the facial expression determination has been performed first may be performed first for the next face.

B3. Modification 3:
In the above embodiment, the determination value E is used for calculation if any element of the shape parameter p _i and the texture parameter λ _k has a high correlation with each facial expression. For example, only the shape parameter p _i is used. Thus, the determination value E may be calculated. Furthermore, depending on circumstances, whether to calculate the judgment value E using only the shape parameters p _i, and to switch whether to calculate a decision value E using both shape parameters p _i and texture parameter lambda _k Also good.

FIG. 23 is a flowchart showing the flow of facial expression determination processing according to this modification. In the present modification, a skin color correction mode for performing skin color correction can be specified by a user (not shown). In the skin color correction mode, processing for correcting the color value of the skin color pixel of the face existing in the target image OI to the color value of the memory color of the skin color specified in advance is executed at the time of printing. When a facial expression to be determined is selected as in the above embodiment (step S610), it is determined whether the skin color correction mode is selected (step S615). If the skin color correction mode is not selected, steps S620 to S680 are sequentially executed as in the above embodiment. That is, the determination value E is calculated using both the shape parameter p _i and the texture parameter λ _k , and each facial expression is determined.

On the other hand, when the skin color correction mode is selected, only the shape parameter p _i is selected (step S625). Then, Steps S630 to S680 are sequentially executed as in the above embodiment. That is, the determination value E is calculated using only the shape parameter p _i , and each facial expression is determined. As it can be determined value E using only the shape parameter p _i, in the determination value setting process it is sufficient to choose a facial expression parameters q _l only shape parameter p _i. If the absolute value of the correlation coefficient is to select only large shape parameter p _i than a predetermined threshold value, the facial expression parameters q than when selecting both from expression parameter q _l shape parameters p _i and texture parameter lambda _k The number of elements of _l will decrease. As in the above embodiment, the information necessary for calculating the equation (10) using both the shape parameter p _i and the texture parameter λ _{k and} the equation using only the shape parameter p _i are used. Information necessary for performing the calculation of (10) may be stored in the internal memory 120 as facial expression determination information EMI.

When the skin color correction mode is specified in this modification, the facial expression is determined without using the texture parameter λ _k . In the skin color correction mode, the color value of the face pixel before correction is corrected to the ideal skin color, so even if the color value of the face pixel before correction has a color that correlates with the facial expression, the print result It can be considered that there is little influence on the impression of facial expression. Therefore, even if facial expression determination is performed without using the texture parameter λ _k , there is a low possibility that a difference between the impression of the facial expression in the print result and the determined facial expression will occur. As described above, when facial expression determination is performed without using the texture parameter λ _k , the number of elements of the facial expression parameter q _l is reduced, so that the calculation of the equation (10) can be speeded up.

B4. Modification 4:
In the embodiment, the threshold value to be compared with the determination value E is constant, but the threshold value may be varied according to the situation. For example, the threshold value to be compared with the determination value E may be switched depending on whether the value of a certain shape parameter p _i or the texture parameter λ _k is larger than a predetermined reference value. For example, when the first shape vector s ₁ indicates that the face is greatly shaken to the left or right, the determination value E is corrected downward as compared with the case where the face is not greatly shaken to the left or right. May be. When the face is greatly shaken to the left or right, if the characteristics of each facial expression are less noticeable than usual (for example, the variation of the mouth angle due to the facial expression is ambiguous due to the facial swing), the judgment value It is desirable to reduce E.

B5. Modification 5:
In the determination value setting process of the embodiment, the elements of the shape parameter p _i and the texture parameter λ _k having higher correlation coefficients with each expression are set as the elements of the expression parameter q _l , but The element of the expression parameter q _l may be selected based on the above.

FIG. 24 is a flowchart showing a flow of processing for selecting the facial expression parameter q _l in the present modification. First, the computer selects one of the shape parameter p _i and the texture parameter λ _k (step S810). Here, the following description will be made assuming that the shape parameter p ₁ is selected. Next, the computer sets the selected shape parameter p ₁ to the lower limit value in the range (step S820). Next, the computer sets each value of the shape parameter p _i and the texture parameter λ _k not selected to 0 (step S830). The computer calculates the facial texture A (x) by performing the affine transformation while applying the shape parameter p _i and the texture parameter λ _k set as described above to the equations (1a) and (2a). (Step S840).

Then, the computer displays the texture A (x) on the display (step S850). Next, the computer determines whether the shape parameters p ₁ selected has been reached the upper limit value in the range (step S860), the current value of the shape parameter p ₁ selected. If it is judged range Is added (step S865), and the process returns to step S840. By repeatedly executing such processing, the selected shape parameter p ₁ is increased by 5% from the lower limit value of the range, and the facial texture A (x) that fluctuates accordingly is sequentially displayed on the display. be able to. Since the selected shape parameter p ₁ is a feature amount corresponding to the swing of the face, the swing angle of the face displayed on the display unit 150 varies with the variation of the shape parameter p ₁ .

If the selected shape parameters p ₁ has not reached the upper limit value in the range, the computer may display the display for inquiring whether or not to adopt the shape parameter p ₁ selected as the expression parameter q _l at the display, The operator's operation is accepted by the input device (step S870). Here, when the operation to adopt is accepted, the selected shape parameter p ₁ is adopted as the facial expression parameter q _l (S880). Next, it is determined whether or not all shape parameters p _i and texture parameters λ _k have been selected (step S890). If not all are selected, the next shape parameter p _i or texture parameter is determined in step S810. to select the λ _k.

By executing the above processing, the operator can observe the change in the facial texture A (x) on the display when the shape parameter p _i and the texture parameter λ _k are individually changed. It is possible to grasp which correlates with facial expressions. If it is felt that there is a correlation, one of the currently selected shape parameter p _i or texture parameter λ _k can be adopted as the expression parameter q _l . Further, an operation for designating the degree that the operator feels to be correlated may be accepted by the input device, and the weighting coefficient α _l may be set based on the degree.

B6. Modification 6:
In the above embodiment, it is determined that the expression corresponding to the largest determination value E is present. However, as shown in FIG. If both of the determination values E are close to each other, it may be determined that both expressions are present. That is, as shown in FIG. 19C, the difference between the second largest determination value E and the maximum determination value E is within a predetermined reference range (for example, 10% of the maximum determination value E). For example, it may be determined that two facial expressions are mixed.

B7. Modification 7:
In the embodiment, in the feature point CP initial position setting process, a total of 80 types corresponding to combinations of values in three stages for each of the four global parameters (size, inclination, vertical position, and horizontal position). A temporary setting position (= 3 × 3 × 3 × 3-1) is preset, but the type and number of parameters used for setting the temporary setting position and the number of parameter values can be changed. For example, only a part of the four global parameters may be used for setting the temporary setting position, or the temporary setting position may be set by combining five values for each parameter used. Good.

B8. Modification 8:
In the feature point CP setting position correction process of the above embodiment, the average shape image I (W (x; p)) is calculated based on the target image OI to determine the installation position of the feature point CP of the target image OI as the average face image A. _Although it is matched with the installation position of the feature point CP of ₀ (x), the arrangement of both feature points CP may be matched by performing image conversion on the average face image A ₀ (x). That is, in the above embodiment, the average shape image I (W (x; p)) as the “face region” is changed, but the average face image A ₀ (x) is changed to the shape parameter p _i and the texture parameter λ. You may change according to the change of _k . In any case, the shape parameter p _i and the texture parameter λ _k that reduce the difference between the “face region” included in the target image OI and the face image reproduced by the face model are corrected.

B9. Modification 9:
The sample image SI (FIG. 3) in the above embodiment is merely an example, and the number and type of images employed as the sample image SI can be arbitrarily set. In the embodiment, the predetermined feature portion (see FIG. 4) of the face indicated by the position of the feature point CP is merely an example, and a part of the feature portion set in the embodiment may be omitted, Other parts may be adopted as the part.

  In the embodiment, the texture model is set by the principal component analysis for the brightness value vector formed by the brightness value in each of the pixel groups x of the sample image SIw, but the brightness representing the texture (appearance) of the face image. The texture model may be set by principal component analysis for index values other than the values (for example, RGB values).

In the embodiment, the size of the average face image A ₀ (x) is not limited to 56 pixels × 56 pixels, and may be other sizes. Further, the average face image A ₀ (x) does not need to include the mask area MA (FIG. 8), and may be configured only by the average shape area BSA. Further, instead of the average face image A ₀ (x), another reference face image set based on the statistical analysis of the sample image SI may be used.

  Moreover, the facial expression to be determined is not limited to a happy face, a sad face, a surprised face, and an angry face, and expressions such as a troubled face, a sleeping face, and a dark face may be determined. Of course, in the embodiment and the modified example, it is possible to determine not only all of a happy face, a sad face, a surprised face, and an angry face, but, for example, whether or not it is a happy face.

  In the embodiment, the shape model and the texture model are set using AAM. However, the shape model and texture model using other modeling methods (for example, a method called Morphable Model or a method called Active Blob) are used. A texture model may be set.

  In the embodiment, the image stored in the memory card MC is set as the attention image OI. However, the attention image OI may be an image acquired via a network, for example. In the above embodiment, image processing by the printer 100 as an image processing apparatus has been described. However, part or all of the processing is executed by another type of image processing apparatus such as a personal computer, a digital still camera, or a digital video camera. It is good also as what is done. The printer 100 is not limited to an ink jet printer, and may be another type of printer such as a laser printer or a sublimation printer.

  In the embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware.

  In addition, when part or all of the functions of the present invention are realized by software, the software (computer program) can be provided in a form stored in a computer-readable recording medium. In the present invention, the “computer-readable recording medium” is not limited to a portable recording medium such as a flexible disk or a CD-ROM, but an internal storage device in a computer such as various RAMs and ROMs, a hard disk, etc. It also includes an external storage device fixed to the computer.

  DESCRIPTION OF SYMBOLS 100 ... Printer, 110 ... CPU, 120 ... Internal memory, 140 ... Operation part, 150 ... Display part, 160 ... Printing mechanism, 170 ... Card interface, 172 ... Card slot, 200 ... Image processing part, 210 ... Setting part, 220 ... feature position detection unit, 222, 242 ... generation unit, 224, 244 ... calculation unit, 226, 246 ... correction unit, 230 ... face area detection unit, 240 ... texture detection unit, 250 ... facial expression determination unit, 310 ... display process Part, 320... Print processing part.

Claims (9)

An image processing apparatus for determining facial expressions included in a noticed image,
A face area detection unit that detects an image area including at least a part of a face image from the attention image as a face area;
A setting unit that sets a face model that reproduces at least the position of a feature part of the face based on a plurality of feature amounts;
A correction unit that corrects the feature amount so that the position of the feature part reproduced by the face model approximates the face region;
A determination unit that calculates a determination value indicating the likelihood of the expression for each of a plurality of facial expressions based on the corrected feature amount, and determines a facial expression by combining the determination values for each expression; apparatus.   The image processing apparatus according to claim 1, wherein the determination unit determines that the face region has an expression corresponding to the determination value that most strongly indicates the likelihood of expression.   3. The image according to claim 1, wherein at least one of the feature amounts is a coefficient of an eigenvector of each principal component obtained by principal component analysis of the position of the feature portion of a plurality of sample images. Processing equipment.   The at least one of the feature amounts includes a coefficient of an eigenvector of each principal component obtained by principal component analysis of color values of pixels of a plurality of sample images. Image processing device. In sequentially determining a plurality of facial expressions included in the plurality of images of interest associated with different times,
When it is determined that the attention image has a certain expression,
Until the face included in the pixel of interest associated with a later time is determined not to be the facial expression, calculation of the determination value for other facial expressions and determination based on the determination value are not performed. The image processing apparatus according to any one of claims 1 to 4.   6. The expression according to claim 1, wherein the determination unit determines a facial expression by comparing the determination value obtained by linearly combining at least two of the corrected feature quantities with a threshold value. Image processing apparatus. An image processing method for determining a facial expression included in an image of interest,
An image area including at least a part of a face image is detected as a face area from the attention image,
Set a face model that reproduces at least the position of the facial feature based on multiple feature quantities,
Correcting the feature amount so that the position of the feature part reproduced by the face model approximates the face region;
An image processing method for calculating a determination value for each of a plurality of facial expressions based on the corrected feature amount, and determining a facial expression by combining the determination values for each facial expression. An image processing program for image processing for determining a facial expression included in an image of interest,
A face area detection function for detecting, as a face area, an image area including at least a part of a face image from the attention image;
A setting function for setting a face model that reproduces at least the position of a facial feature part based on a plurality of feature amounts;
A correction function for correcting the feature amount so that the position of the feature part reproduced by the face model approximates the face region;
An image processing program for causing a computer to realize a determination function for calculating a determination value for each of a plurality of facial expressions based on the corrected feature amount and determining a facial expression by combining the determination values for each facial expression. A face area detection unit that detects an image area including at least a part of a face image from the attention image as a face area;
A setting unit that sets a face model that reproduces at least the position of a feature part of the face based on a plurality of feature amounts;
A correction unit that corrects the feature amount so that the position of the feature part reproduced by the face model approximates the face region;
A determination unit that calculates a determination value for each of a plurality of facial expressions based on the corrected feature amount, and determines a facial expression by combining the determination values for each expression;
A printing apparatus comprising: a printing unit that prints the attention image including a face image determined to have a predetermined facial expression among the attention images.

Images