JP2011048747A - Image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accelerate the speed of the processing of the detection of a coordinate position of a characteristic region of a face, and to improve the accuracy of the detection. <P>SOLUTION: An image processor for detecting a coordinate position of a characteristic region of a face contained in an image includes: a face region detecting part for detecting an image region including at least a portion of the face from a target image as a face region; a trimming part for trimming an image in a prescribed range including the face region from the target image; a converting part for converting the size of the trimmed image into a predetermined size; and a characteristic position detecting part for detecting a coordinate position of the characteristic part based on the image whose size is converted. The converting part performs the conversion of the size and processing noise reduction with respect to the trimmed image. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that detects a coordinate position of a feature portion of a face included in an image.

  As a visual event modeling method, an active appearance model (Active Appearance Model, also referred to as “AAM” for short) is known. In AAM, for example, through statistical analysis of the positions (coordinates) and pixel values (for example, luminance values) of facial features (for example, the corners of the eyes, the nose and the face lines) included in a plurality of sample images, A shape model representing the shape of the identified face and a texture model representing “appearance” in the average face shape are set, and the face image is modeled using these models. According to AAM, it is possible to model (synthesize) an arbitrary face image, and to detect the position of a facial feature part included in the image (see Patent Document 1).

JP 2007-141107 A

  Here, in the detection processing of the position of the feature part of the face, pixels in an image range surrounded by points (feature points) assumed to correspond to the feature part of the face are extracted from the target image to be processed. Affinity transformation into an image with a fixed shape while sampling and comparing the image after the affine transformation with the average shape and appearance face image was assumed to correspond to the facial feature part. The point is evaluated (evaluation of whether or not it is close to the coordinate position of the characteristic part).

However, in the above-described detection process, when sampling is performed in the process of the affine transformation, pixels are scanned for the entire image of interest and necessary pixels are extracted, so that the detection process takes a long time. .
In the detection process, pixels are simply extracted at a required sampling rate at the time of sampling, so that the image obtained as a result of the affine transformation is an image that has low image quality and may contain noise. . Therefore, it cannot be said that the accuracy of the evaluation based on the image obtained as a result of such conversion is sufficiently high, and as a result, it is difficult to make the detection accuracy of the position of the characteristic part high.

  The present invention has been made to solve at least one of the above-described problems, and an object of the present invention is to increase the speed and accuracy of a process for detecting the position of a facial feature part included in an image.

One aspect of the present invention is an image processing apparatus that detects a coordinate position of a feature part of a face included in an image, and detects an image area including at least a part of the face as a face area from a target image. Based on a detection unit, a trimming unit that trims an image of a predetermined range including the face area from the attention image, a conversion unit that converts the size of the trimmed image to a predetermined size, and the image whose size has been converted And a feature position detector for detecting the coordinate position of the feature part.
According to the present invention, since the processing for detecting the coordinate position of the characteristic part is performed on an image after trimming the image of interest and converting it to a predetermined size, the image to be scanned during the detection processing is of interest. As a result, the detection process is faster than the conventional image. The characteristic part may be, for example, a part of eyebrows, eyes, nose, mouth, and face line. In this case, the coordinate position can be satisfactorily detected for a portion of the eyebrows, eyes, nose, mouth, and face line.

  The conversion unit may perform the conversion of the size together with a process for noise reduction on the trimmed image. According to this configuration, since the coordinate position detection process is performed on the image after the noise reduction process and the size converted as described above, the detection result is blurred under the influence of noise. Therefore, the detection process can be performed more accurately than in the past.

  The conversion unit may execute noise reduction processing using a noise reduction filter that varies depending on the size of the trimmed image. According to this configuration, it is possible to reduce noise by optimal filter processing according to the size of the image to be trimmed.

  The trimming unit may trim a range including all of the ranges specified on the basis of the face area according to each parameter defined in advance representing the size, angle, and position of the face image with respect to the face area. According to this configuration, it is possible to reliably trim an image range that is necessary and sufficient for detecting the coordinate position of the facial feature part.

  The feature position detection unit sets feature points for detecting the coordinate position of the feature part with respect to the image whose size has been converted, and a part of the image whose size has been converted is arranged with feature points. An average shape image that is an image converted to have a predetermined arrangement is generated, and a difference value between the average shape image and an average face image that is an image generated by averaging a plurality of sample images is calculated. Based on the calculated difference value, the setting position of the feature point in the image whose size has been converted is updated to correct the setting position so as to approach the coordinate position of the feature part, and the correction The set position may be detected as the coordinate position of the characteristic part. According to this configuration, in order to detect the coordinate position of the feature part by correcting the setting position of the feature point in the image whose size has been converted based on the difference value between the average shape image and the average face image, the image It is possible to satisfactorily detect the position of the feature part of the face included in.

  The feature position detection unit repeatedly executes update of the feature point setting position in the image whose size has been converted until the difference value is smaller than a predetermined threshold value, and the difference value is the threshold value. When the value is smaller than the value, the arrangement of the feature points in the image whose size has been converted is reproduced in the target image according to the ratio of the size of the image whose size has been changed and the trimmed image. To generate an average shape image that is an image obtained by converting a part of the target image so that the feature point is set on the target image, and the arrangement of the feature point in the target image becomes the predetermined configuration. By updating the setting position of the feature point in the target image based on the difference value between the shape image and the average face image, the setting position of the feature point in the target image is changed to the position of the feature part. Corrected so as to approach to the position, the setting position of the characteristic point in the corrected target image may be detected as the coordinate position of the feature portion. According to the configuration, on the image whose size has been converted, the setting position of the feature point can be corrected to a state where the matching rate with the position of the feature part of the face that is actually captured is extremely high, Then, the feature point arrangement in the image whose size has been converted is reproduced in the target image, and further, the feature point setting position in the target image is corrected so as to be close to the coordinate position of the feature part. It is possible to detect the position of the feature portion of the included face very accurately.

  The feature position detection unit may perform the process of updating the setting position of the feature point in the target image with a predetermined number of times as an upper limit. As described above, when the arrangement of the feature points after the setting position is appropriately corrected in the image whose size has been converted is reproduced in the attention image, the feature points related to the reproduced arrangement are included in the attention image. It can be said that the position of the characteristic part is shown fairly accurately. Therefore, the process of updating the setting position of the feature point in the target image can be performed with a predetermined number of times as the upper limit, and the position of the feature part of the face included in the target image can be detected very accurately.

  The conversion unit may execute the conversion of the size together with a correction process for bringing at least one element of brightness, contrast, and color of the trimmed image closer to the average face image. According to the configuration, the brightness and color of the average shape image generated by converting the size-converted image and the average shape image can be brought close to each other, and therefore, due to factors other than the setting position of the feature point at that time The inconvenience that the difference value does not become very small (does not converge) can be avoided, and as a result, the detection process can be further speeded up and highly accurate.

  The technical idea of the present invention can be realized by means other than an image processing apparatus. For example, it is possible to grasp an invention of an image processing method having processing steps executed by each part of the image processing apparatus and an invention of a computer-readable program that causes a computer to execute a function executed by each part of the image processing apparatus.

FIG. 2 is an explanatory diagram schematically illustrating a configuration of a printer as an image processing apparatus. It is a flowchart which shows an AAM setting process. It is explanatory drawing which shows an example of sample image SI. It is explanatory drawing which shows an example of the setting method of the feature point CP in the sample image SI. It is explanatory drawing which shows an example of the coordinate of the feature point CP set to sample image SI. Is an explanatory diagram showing an example of the average shape s _0. It is explanatory drawing which illustrated the relationship between shape vector s _i and shape parameter _pi, and face shape s. It is explanatory drawing which shows an example of the method of the warp W of the sample image SI. It is explanatory drawing which shows an example of average face image _A0 (x). It is a flowchart which shows a face feature position detection process. It is explanatory drawing which shows an example of the detection result of the face area FA in the attention image OI. It is a flowchart which shows the initial position setting process of the feature point CP. It is explanatory drawing which shows an example of the temporary setting position of the feature point CP by changing the value of a global parameter. It is explanatory drawing which shows an example of the average shape image I (W (x; p)). It is a flowchart which shows the feature point CP setting position correction process in a detection target image. It is explanatory drawing which shows an example of the result of a face feature position detection process. It is the figure which showed an example of each range specified on the basis of a face area according to a global parameter. It is the figure which showed an example of a mode that the image is trimmed from the attention image OI and affine transformation is performed. It is a flowchart which shows the feature point CP setting position correction process in an attention image.

Embodiments of the present invention will be described below with reference to the drawings.
1. Configuration of Image Processing Apparatus FIG. 1 is an explanatory diagram schematically illustrating a configuration of a printer 100 as an image processing apparatus according to the present embodiment. The printer 100 is an ink jet color printer that supports so-called direct printing, in which an image is printed based on image data acquired from a memory card MC or the like. The printer 100 includes a CPU 110 that controls each unit of the printer 100, an internal memory 120 configured by a ROM and a RAM, an operation unit 140 configured by buttons and a touch panel, a display unit 150 configured by a liquid crystal display, and printing. A mechanism 160 and a card interface (card I / F) 170 are provided. The printer 100 further includes an interface (I / F) 180 for performing data communication with other devices (for example, a digital still camera or the computer 300). Each component of the printer 100 is connected via a bus so that bidirectional communication is possible.

  The printing mechanism 160 performs printing based on the print data. The card interface 170 is an interface for exchanging data with the memory card MC inserted into the card slot 172. In this embodiment, an image file including image data is stored in the memory card MC.

  The internal memory 120 stores an image processing unit 200, a display processing unit 310, and a print processing unit 320. The image processing unit 200 is a computer program, and performs facial feature position detection processing by being executed by the CPU 110 under a predetermined operating system. The face feature position detection process is a process of detecting the coordinate position of a predetermined feature part (for example, the corner of the eye, the nose head, or the face line) in the face image. The face feature position detection process will be described in detail later. The display processing unit 310 and the print processing unit 320 are also executed by the CPU 110 to realize the respective functions.

  The image processing unit 200 includes a feature position detection unit 220, a face area detection unit 230, a trimming unit 250, and a conversion unit 260 as program modules. The feature position detection unit 220 includes a setting unit 221, a generation unit 222, a calculation unit 224, and a correction unit 226. The functions of these units will be described in detail in the description of the face feature position detection process described later.

  The display processing unit 310 is a display driver that controls the display unit 150 to display processing menus, messages, images, and the like on the display unit 150. The print processing unit 320 is a computer program for generating print data from image data, controlling the printing mechanism 160, and printing an image based on the print data. The CPU 110 reads out and executes these programs (the image processing unit 200, the display processing unit 310, and the print processing unit 320) from the internal memory 120, thereby realizing the functions of these units.

  The internal memory 120 also stores AAM information AMI. The AAM information AMI is information set in advance by an AAM setting process described later, and is referred to in a face feature position detection process described later. The contents of the AAM information AMI will be described in detail in the description of AAM setting processing described later.

2. AAM Setting Processing FIG. 2 is a flowchart showing the flow of AAM setting processing in this embodiment. The AAM setting process is a process for setting a shape model and a texture model used for modeling an image called AAM (Active Appearance Model). In this embodiment, the AAM setting process is performed when the user operates the computer 300 prepared for AAM setting. The computer 300 includes a CPU, a RAM, a ROM, an HDD, a display, an input device, and the like, which are connected by a bus. The CPU reads the program recorded in the HDD and executes arithmetic processing according to the program, whereby the computer 300 executes AAM setting processing described later.

First, the user prepares a plurality of images including a human face on the memory (HDD) of the computer 300 as sample images SI (step S110).
FIG. 3 is an explanatory diagram showing an example of the sample image SI. As shown in FIG. 3, the sample image SI is related to various attributes such as personality, race / gender, facial expression (anger, laughter, trouble, surprise, etc.), and orientation (front, upward, downward, right, left, etc.). It is prepared to include different face images. If the sample image SI is prepared in this way, any face image can be accurately modeled by AAM, and accurate face feature position detection processing (described later) for any face image can be executed. It becomes. The sample image SI is also called a learning image.

Next, the computer 300 sets a feature point CP to the face image included in each sample image SI by a predetermined operation of the user (step S120).
FIG. 4 is an explanatory diagram showing an example of a method for setting the feature point CP in the sample image SI. The feature point CP is a point indicating the position of a predetermined feature part in the face image. In this embodiment, as predetermined feature parts, a predetermined position on the eyebrows in a person's face (for example, an end point or a four-divided point, the same applies hereinafter), a predetermined position on the eye contour, and a predetermined position on the nose and nose contours 68 parts such as a predetermined position on the contour of the upper and lower lips and a predetermined position on the contour of the face (face line) are set. That is, in the present embodiment, a predetermined position in the facial organs (eyebrows, eyes, nose, mouth) and facial contour that are commonly included in the human face are set as the characteristic parts. As shown in FIG. 4, the feature points CP are set (arranged) at positions representing 68 feature parts designated on the screen by a user operation in the sample image SI displayed on the display by the computer 300. Since each feature point CP set in this way corresponds to each feature part, the arrangement of the feature points CP in the face image can be expressed as specifying the shape of the face. The position of the feature point CP in the sample image SI is specified by coordinates.

  FIG. 5 is an explanatory diagram showing an example of the coordinates of the feature point CP set in the sample image SI. In FIG. 5, SI (j) (j = 1, 2, 3,...) Indicates each sample image SI, and CP (k) (k = 0, 1,..., 67) corresponds to Each feature point CP of the sample image SI is shown. CP (k) -X represents the X coordinate of the feature point CP (k), and CP (k) -Y represents the Y coordinate of the feature point CP (k). The coordinates of the feature point CP include predetermined reference points (for example, in the sample image SI normalized with respect to the size of the face, the inclination of the face (inclination in the image plane), and the position of the face in the X direction and the Y direction) The coordinates with the origin at the lower left point of the image are used. Further, in the present embodiment, a case where a plurality of human face images are included in one sample image SI is allowed (for example, two sample face images are included in the sample image SI (2)). Each person in one sample image SI is specified by a person ID.

  Subsequently, the computer 300 sets an AAM shape model (step S130). Specifically, a principal component analysis is performed on a coordinate vector (see FIG. 5) composed of the coordinates (X coordinate and Y coordinate) of 68 feature points CP in each sample image SI, and specified by the position of the feature point CP. The face shape s to be formed is modeled by the following equation (1). The shape model is also referred to as a feature point CP arrangement model.

In the above formula (1), s ₀ is an average shape.
Figure 6 is an explanatory diagram showing an example of the average shape s _0. As shown in FIGS. 6A and 6B, the average shape s ₀ is a model representing the average face shape specified by the average position (average coordinate) for each feature point CP of the sample image SI. is there. In the present embodiment, in the average shape s ₀ , a region surrounded by a straight line connecting feature points CP (face line, eyebrows, feature points CP corresponding to the eyebrows, see FIG. 4) located on the outer periphery (FIG. 6 (FIG. 6). b) is indicated by hatching) and is referred to as “average shape region BSA”. In the average shape s ₀ , as shown in FIG. 6A, a plurality of triangular regions TA having the feature points CP as vertices are set so as to divide the average shape region BSA into a mesh shape.

In the above equation (1) representing the shape model, s _i is a shape vector, and p _i is a shape parameter representing the weight of the shape vector s _i . The shape vector s _i is a vector representing the characteristics of the face shape s, and is an eigenvector corresponding to the i-th principal component obtained by principal component analysis. As shown in the above equation (1), in the shape model in the present embodiment, the face shape s representing the arrangement of the feature points CP is a model as the sum of the average shape s ₀ and the linear combination of the n shape vectors s _i. It becomes. By appropriately setting the shape parameter p _i in the shape model, it is possible to reproduce the face shape s in any image.

FIG. 7 is an explanatory diagram illustrating the relationship between the shape vector s _{i, the} shape parameter _pi, and the face shape s. As shown in FIG. 7A, in order to specify the face shape s, the number n set based on the cumulative contribution rate in order from the eigenvector corresponding to the principal component having the larger contribution rate (in FIG. 7, n = The eigenvector of 4) is adopted as the shape vector s _i . Each of the shape vectors s _i corresponds to the movement direction / movement amount of each feature point CP, as indicated by the arrows in FIG. In the present embodiment, the first shape vector s ₁ corresponding to the first principal component having the largest contribution rate is a vector that is substantially correlated with the left / right swing of the face, and by increasing or decreasing the shape parameter p ₁ , FIG. As shown in (b), the horizontal face direction of the face shape s changes. The second shape vector s ₂ corresponding to the second principal component having the second largest contribution ratio is a vector that is substantially correlated with the vertical swing of the face, and FIG. 7C is obtained by increasing or decreasing the shape parameter p ₂ . As shown in FIG. 4, the vertical face direction of the face shape s changes. The third shape vector s ₃ corresponding to the third principal component having the third largest contribution ratio is a vector that is substantially correlated with the aspect ratio of the face shape, and the fourth principal component having the fourth largest contribution ratio. The fourth shape vector s ₄ corresponding to is a vector that is substantially correlated with the degree of mouth opening. As described above, the value of the shape parameter represents the feature of the face image such as facial expression and face orientation. The shape parameter is a feature amount calculated based on a plurality of sample images including a face image whose coordinates of the feature part are known.

The computer 300 transmits the average shape s ₀ and the shape vector s _i set in the shape model setting step (step S 130) to the printer 100. The printer 100 stores the transmitted information on the average shape s ₀ and the shape vector s _{i in} the internal memory 120 as AAM information AMI (FIG. 1).

Subsequently, the computer 300 sets an AAM texture model (step S140). Specifically, first, for each sample image SI, image conversion (hereinafter referred to as “warp”) is performed so that the setting position of the feature point CP in the sample image SI becomes equal to the setting position of the feature point CP in the average shape s ₀ . W ”).

FIG. 8 is an explanatory diagram showing an example of a method of warping W of the sample image SI. In each sample image SI, similarly to the average shape s ₀ , a plurality of triangular areas TA that divide the area surrounded by the feature points CP located on the outer periphery into mesh shapes are set. The warp W is a set of affine transformations for each of the plurality of triangular areas TA. That is, in the warp W, the image of triangle areas TA in a sample image SI is affine transformed into an image of corresponding triangle areas TA of the average shape s _0. The warp W generates a sample image SI (hereinafter referred to as “sample image SIw”) in which the set position of the feature point CP is equal to the set position of the feature point CP in the average shape s ₀ .

  Each sample image SIw has a rectangular frame containing an average shape area BSA (shown with hatching in FIG. 8) as an outer periphery, and an area other than the average shape area BSA (hereinafter also referred to as “mask area MA”). Generated as a masked image. An image area in which the average shape area BSA and the mask area MA are combined is referred to as a reference area BA. Each sample image SIw is normalized as an image having a size of 56 pixels × 56 pixels, for example.

  Next, a principal component analysis is performed on a luminance value vector composed of luminance values in each pixel group x of each sample image SIw, and a facial texture (also referred to as “appearance”) A (x) is expressed by the following formula ( 2). The pixel group x is a set of pixels located in the average shape area BSA.

In the above formula (2), A ₀ (x) is an average face image.
FIG. 9 is an explanatory diagram illustrating an example of the average face image A ₀ (x). The average face image A ₀ (x) is an image representing the average face of the sample image SIw (see FIG. 8) after the warp W. That is, the average face image A ₀ (x) is an image calculated by taking the average of the pixel values (luminance values) of the pixel group x in the average shape area BSA of the sample image SIw. Therefore, the average face image A ₀ (x) is a model representing the average face texture (appearance) in the average face shape. Note that the average face image A ₀ (x) is composed of the average shape area BSA and the mask area MA, similarly to the sample image SIw, and is calculated as an image having a size of 56 pixels × 56 pixels, for example.

In the above equation (2) representing the texture model, A _i (x) is a texture vector, and λ _i is a texture parameter representing the weight of the texture vector A _i (x). The texture vector A _i (x) is a vector representing the characteristics of the facial texture A (x), and is specifically an eigenvector corresponding to the i-th principal component obtained by principal component analysis. That is, the number m of eigenvectors set based on the cumulative contribution rate is adopted as the texture vector A _i (x) in order from the eigenvector corresponding to the principal component having the larger contribution rate. In the present embodiment, the first texture vector A ₁ (x) corresponding to the first principal component having the largest contribution rate is a vector that is substantially correlated with a change in face color (also regarded as a gender difference).

As shown in the above equation (2), in the texture model in the present embodiment, the facial texture A (x) representing the appearance of the face is the average face image A ₀ (x) and m texture vectors A _i (x ) And the linear combination. By appropriately setting the texture parameter λ _i in the texture model, it is possible to reproduce the facial texture A (x) in any image.
The computer 300 transmits the average face image A ₀ (x) and the texture vector A _i (x) set in the texture model setting step (step S140) to the printer 100. The printer 100 stores the transmitted information of the average face image A ₀ (x) and the texture vector A _i (x) in the internal memory 120 as AAM information AMI (FIG. 1).

By the AAM setting process described above (FIG. 2), a shape model for modeling the face shape and a texture model for modeling the face texture are set. By combining the set shape model and the texture model, that is, the synthesized texture A (x) is converted from the average shape s ₀ to the shape s (inverse conversion of the warp W shown in FIG. 8). Thus, it is possible to reproduce the shape and texture of any face image.

3. Face Feature Position Detection Processing FIG. 10 is a flowchart showing the flow of face feature position detection processing in the present embodiment. The face feature position detection process is a process of detecting the position of the feature part in the face image by determining the arrangement of the feature points CP in the face image included in the target image using the AAM information AMI. As described above, in the present embodiment, in the AAM setting process (FIG. 2), a total of 68 predetermined positions in the human facial organs (eyebrows, eyes, nose, mouth) and facial contour are set as the characteristic parts. (See FIG. 4). Therefore, in the face feature position detection process of the present embodiment, the arrangement of 68 feature points CP indicating predetermined positions in the organ and face contour of the person is determined.

Note that when the face characteristic position detecting process disposition of the characteristic points CP in the face image is determined, and the shape parameters p _i of the face image, the value of the texture parameter lambda _i is identified. Therefore, the result of the face feature position detection process is a facial expression determination for detecting a facial image of a specific facial expression (for example, a smile or a face with closed eyes) or a facial image in a specific direction (for example, rightward or downward). Therefore, it can be used for face orientation determination, face deformation for deforming the face shape, face shadow correction, and the like.

  First, the image processing unit 200 (FIG. 1) acquires image data representing an attention image that is a target of face feature position detection processing (step S210). In the printer 100 of the present embodiment, when the memory card MC is inserted into the card slot 172, thumbnail images of image files stored in the memory card MC are displayed on the display unit 150. One or more images to be processed are selected by the user via the operation unit 140. The image processing unit 200 acquires an image file including image data corresponding to one or more selected images from the memory card MC and stores it in a predetermined area of the internal memory 120. The acquired image data is called attention image data, and the image represented by the attention image data is called attention image OI.

  The face area detection unit 230 (FIG. 1) detects an image area including at least a part of the face image included in the target image OI as the face area FA (step S220). The detection of the face area FA can be performed using a known face detection method. Known face detection methods include, for example, pattern matching methods, skin color region extraction methods, learning using sample images (for example, learning using neural networks, learning using boosting, and support vector machines). For example, a method using learning data set by learning).

  FIG. 11 is an explanatory diagram illustrating an example of a detection result of the face area FA in the target image OI. FIG. 11 shows the face area FA detected in the target image OI. In the present embodiment, a face detection method is used in which a rectangular area including the vertical direction of the face from the forehead to the chin and the horizontal direction from the outside of both ears is detected as the face area FA.

  Next, the trimming unit 250 (FIG. 1) trims an image in a predetermined range including the face area FA from the target image OI (step S230). The trimming unit 250 first specifies a range to be trimmed (trimming range TI) based on the face area FA according to a predetermined global parameter. The global parameters are parameters that represent the size, inclination, vertical position, and horizontal position of the face image with respect to the face area FA, respectively. During the initial position setting process (step S250) of the feature point CP described later. It is a numerical value used for. The global parameter is stored in advance in a predetermined area of the internal memory 120. In this embodiment, the global parameter is also used when specifying the trimming range TI.

  FIG. 17 illustrates each range specified based on the face area FA according to each global parameter. In FIG. 17A, the size of a predetermined shape (for example, an ellipse) C assuming a face image is changed in accordance with three levels (large, standard, and small) global parameters representing the size of the face area FA. Shows the state. FIG. 17B shows each position determined by a global parameter of three stages (upper, middle, and lower) representing the vertical position with respect to the face area FA, and three stages (left and right positions with respect to the face area FA). The predetermined shape C (size is standard) is arranged at each position determined by the global parameters (left, center, right). FIG. 17C shows three steps (15 degrees for counterclockwise (−), 0 degree, and 15 degrees clockwise (+)) for the predetermined shape C (size is standard), representing the inclination with respect to the face area FA. ) Shows a state in which the slope is changed according to the global parameter.

  In this embodiment, the trimming unit 250 combines the above-described four types of global parameters (size, inclination, vertical position, horizontal position) as known three-stage values, thereby combining the face area FA. A rectangular range including all of the total 81 types of ranges specified on the basis of is specified as the trimming range TI. Assume that the upper, lower, left and right sides of the trimming range TI are parallel to the upper, lower, left and right sides of the face area FA, respectively. Then, the image in the trimming range TI specified in this way is trimmed from the target image OI.

Next, the conversion unit 260 (FIG. 1) affine-transforms the trimmed image so that the size of the image trimmed from the target image OI (referred to as a trimmed image) becomes a predetermined size, and after the affine transformation The image is stored in a predetermined area of the internal memory 120 (step S240). The image stored in this way is an image (referred to as a detection target image CI) that is a target of the position setting / correction processing (steps S250 and S260) described later. The predetermined size here refers to a size whose vertical and horizontal lengths are not less than 1 and not more than twice the vertical and horizontal lengths of the average face image A ₀ (x). As described above, when the average face image A ₀ (x) has a size of 56 pixels × 56 pixels vertically and horizontally, for example, 100 pixels × 100 pixels vertically and horizontally are set as the predetermined size.

  FIG. 18 shows a state in which a trimming image is affine-transformed from a certain target image OI to generate a detection target image CI having a predetermined size (vertical and horizontal 100 pixels × 100 pixels). In FIG. 18, the trimming range TI in the target image OI is indicated by a solid line. The trimming range TI includes the entire face area FA. Although depending on the size of the target image OI and the size of the face area FA, when the trimming range TI is larger than the predetermined size, the image is reduced in the affine transformation for the trimmed image. On the other hand, when the trimming range TI is smaller than the predetermined size, the image is enlarged in the affine transformation for the trimmed image. Further, the vertical direction of the face image included in the target image OI may be inclined with respect to the vertical direction of the target image OI. In this case, the face area FA and the trimming range TI are also included in the target image OI. It leans against it. Therefore, the conversion unit 260 performs size conversion with rotation processing in a direction to eliminate the inclination in the affine transformation for the trimmed image.

In this embodiment, when performing the affine transformation (size conversion and rotation processing) on the trimmed image, the conversion unit 260 also performs processing for noise reduction on the trimmed image.
When the trimming image is reduced to the predetermined size, the conversion unit 260 does not simply sample the pixels in accordance with the reduction ratio (the ratio between the image size of the trimming range TI and the predetermined size), but instead of the predetermined noise. Reduction is performed while performing reduction processing. The noise reduction process is, for example, a smoothing process. In the smoothing process, for example, when the detection target image CI having the predetermined size is generated by reducing the size of the trimmed image to 1 / n, each pixel value of a pixel group including n pixels is averaged. To generate one pixel value. And let the pixel produced | generated in this way be 1 pixel which comprises the detection target image CI. Such a smoothing process is realized by using a moving average filter.

  Alternatively, the conversion unit 260 may use a Gaussian filter as a kind of smoothing filter in the smoothing process. In this case, the conversion unit 260 uses different filters depending on the size of the trimmed image. For example, when the size of the trimming range TI is 300 pixels × 300 pixels in the vertical and horizontal directions, the conversion unit 260 is used based on the ratio (300/100) to the vertical and horizontal lengths (100 pixels × 100 pixels) of the predetermined size. The size of the Gaussian filter is 3 pixels × 3 pixels. Further, when the size of the trimming range TI is 1500 pixels × 1500 pixels in the vertical and horizontal directions, the conversion unit 260 is used based on the ratio (1500/100) to the vertical and horizontal lengths (100 pixels × 100 pixels) of the predetermined size. The size of the Gaussian filter is 15 pixels × 15 pixels. The conversion unit 260 applies the Gaussian filter determined in this way to the trimmed image. As a result, for each region to which the Gaussian filter is applied, one pixel of interest that has been smoothed by the weighting of the surrounding pixels is acquired, and thus the detection target image CI is acquired by using the pixel of interest acquired for each region to which the filter is applied. Constitute.

  The converter 260 performs pixel interpolation when enlarging the trimmed image to the predetermined size. That is, the detection target image CI in which noise is suppressed can be generated by enlarging the trimmed image to the predetermined size by linear interpolation or nonlinear interpolation.

The feature position detector 220 (FIG. 1) sets the initial position of the feature point CP in the detection target image CI (step S250).
FIG. 12 is a flowchart showing a flow of initial position setting processing of the feature point CP in the present embodiment. In this case, the setting unit 221 changes various values of the global parameter described above, and sets the feature point CP to a temporary setting position on the detection target image CI (step S310).

FIG. 13 is an explanatory diagram showing an example of a temporary setting position of the feature point CP by changing the value of the global parameter. FIGS. 13A and 13B show a mesh formed by connecting the feature points CP and the feature points CP in the detection target image CI. Note that the mesh formed by connecting the feature points CP is a set of a plurality of triangular areas TA having the feature points CP as vertices as described above (see FIG. 6A and the like). Setting unit 221, as shown in the center shown in FIG. 13 (a) and 13 (b), a mesh having the arrangement between placement equal characteristic point CP between the characteristic points CP of the average shape s _0, A mesh whose size, inclination, and position are adjusted for the face area FA (however, the face area FA after the size conversion is performed in accordance with the size conversion of the trimmed image) is set according to the combination of the most average global parameters. . In the example described so far, the most average combination of global parameters is “standard” for the size of the face area FA, and “center” for the vertical position and the horizontal position for the face area FA. In addition, the inclination with respect to the face area FA is a combination of global parameters indicating “0 degree”. In the present embodiment, the setting position (temporary setting position) of the feature point CP constituting the mesh as shown in the center of FIGS. 13A and 13B is also referred to as “reference temporary setting position”.

  The setting unit 221 also sets a plurality of temporary setting positions obtained by changing various global parameter values with respect to the reference temporary setting position. Changing global parameters (size, inclination, vertical position and horizontal position) means that the mesh formed by the feature points CP in the detection target image CI is enlarged / reduced, the inclination is changed, and the mesh is moved in parallel. It corresponds to. Accordingly, as shown in FIG. 13 (a), the setting unit 221 forms a temporary setting position (below the reference temporary setting position diagram and a temporary setting position that forms a mesh obtained by enlarging or reducing the mesh at the reference temporary setting position at a predetermined magnification. Or a temporary setting position (shown on the right and left in the drawing of the reference temporary setting position) that forms a mesh whose inclination is changed clockwise or counterclockwise by a predetermined angle. In addition, the setting unit 221 forms a temporary setting position (upper left of the figure of the reference temporary setting position, which forms a mesh obtained by performing a combination of enlargement / reduction and change of inclination with respect to the mesh of the reference temporary setting position. Also set in the lower left, upper right, and lower right).

  Also, as shown in FIG. 13B, the setting unit 221 forms a temporary setting position (a reference temporary setting position diagram) that forms a mesh that is translated up or down by a predetermined amount from the mesh at the reference temporary setting position. And a temporary setting position (shown on the left and right in the drawing of the reference temporary setting position) that forms a mesh moved in parallel to the left or right. In addition, the setting unit 221 temporarily sets positions (upper left and lower left in the drawing of the reference temporary setting positions) that form a mesh obtained by performing a combination of vertical and left and right parallel movements on the reference temporary setting position mesh. , Shown in the upper right and lower right).

  The setting unit 221 also has temporary setting positions at which the horizontal movement shown in FIG. 13B is executed with respect to the mesh at each of the eight temporary setting positions other than the reference temporary setting position shown in FIG. Set. Therefore, in the present embodiment, as described above, there are 80 patterns set by combining four global parameters (size, inclination, vertical position, horizontal position) each having three levels of values ( = 3 × 3 × 3 × 3-1) temporary setting positions and reference temporary setting positions, a total of 81 temporary setting positions are set.

The generation unit 222 (FIG. 1) generates an average shape image I (W (x; p)) corresponding to each set temporary setting position (step S320).
FIG. 14 is an explanatory diagram showing an example of the average shape image I (W (x; p)). The average shape image I (W (x; p)) is calculated by conversion such that the arrangement of the feature points CP in the detection target image CI is equal to the arrangement of the feature points CP in the average shape s ₀ .

The transformation for calculating the average shape image I (W (x; p)) is a warp W that is a set of affine transformations for each triangular area TA, similarly to the transformation for calculating the sample image SIw (see FIG. 8). Is done. Specifically, a conversion target region (region surrounded by the feature point CP located on the outer periphery of the mesh) is specified by the feature point CP (see FIG. 13) temporarily set in the detection target image CI, and this detection target image An average shape image I (W (x; p)) is calculated by performing affine transformation for each triangular area TA on the transformation target area in the CI. In this embodiment, the average shape image I (W (x; p) ) is composed of an average face image _A 0 (x) and an average shape area BSA and a mask area MA, average face image _A 0 (x) Is calculated as an image of the same size (for example, 56 pixels × 56 pixels).

As described above, the detection target image CI, the length and width is the size of the extent twice within the length of the vertical and horizontal respectively average face image A _{0 (x),} than the average face image A _{0 (x)} large. Therefore, when the deformation target area specified by the feature point CP at the temporarily set position in the detection target image CI is converted into the average shape image I (W (x; p)), an image reduction process is performed. . In such an average shape image I (W (x; p)), a part of an image (detection target image CI) that has been trimmed from the target image OI and subjected to size conversion has a predetermined arrangement of feature points CP. It is the image converted so that it becomes.

The pixel group x as described above, a set of pixels located in the average shape area BSA of the average shape s _0. In this embodiment, the average shape image is expressed as an average shape image I (W (x; p)) in the sense of an image (pixel group) I generated by the warp W (x; p) with the shape parameter p. ing. FIG. 14 shows nine average shape images I (W (x; p)) corresponding to the nine temporarily set positions shown in FIG.

The calculation unit 224 (FIG. 1) calculates a difference image Ie between the average shape image I (W (x; p)) corresponding to each temporarily set position and the average face image A ₀ (x) (step S330). Since 81 types of temporarily set positions of the feature points CP are set, the calculation unit 224 (FIG. 1) calculates 81 difference images Ie.

  The setting unit 221 calculates a norm (Euclidean distance) from the pixel value of each difference image Ie, and a temporary installation position corresponding to the difference image Ie having the smallest norm value (hereinafter also referred to as “norm minimum temporary setting position”). The initial position of the feature point CP in the detection target image CI is set (step S340). The pixel value of the difference image Ie for calculating the norm may be a luminance value or an RGB value. Thus, the initial position setting process for the feature point CP is completed.

The feature position detection unit 220 (FIG. 1) corrects the set position of the feature point CP set as the initial position in the detection target image CI in step S250 (step S260).
FIG. 15 is a flowchart showing the flow of the setting position correction process for the feature point CP executed in step S260.

  The generation unit 222 (FIG. 1) calculates an average shape image I (W (x; p)) from the detection target image CI (step S410). The calculation method of the average shape image I (W (x; p)) is the same as that in step S320 in the initial position setting process of the feature point CP.

The calculation unit 224 calculates a difference image Ie between the average shape image I (W (x; p)) generated in step S410 and the average face image A ₀ (x) (step S420). The feature position detector 220 determines whether or not the feature point CP setting position correction process has converged based on the difference image Ie calculated in step S420 (step S430). The feature position detection unit 220 calculates the norm of the difference image Ie, determines that the norm value has converged if it is smaller than a preset threshold value, and if the norm value is greater than or equal to the threshold value, Judge that it has not converged yet. Note that the feature position detection unit 220 determines that the calculated norm value of the difference image Ie has converged when the norm value is smaller than the norm value calculated in the previous step S430. May be determined as not yet converged. Alternatively, the feature position detection unit 220 may perform the convergence determination by combining the determination based on the threshold value and the determination based on the comparison with the previous value. For example, the feature position detection unit 220 determines that the calculated norm value has converged only when it is smaller than the threshold value and smaller than the previous value, and otherwise has not yet converged. It may be determined.

If it is determined in step S430 that the convergence has not yet been completed, the correction unit 226 (FIG. 1) calculates the parameter update amount ΔP (step S440). The parameter update amount ΔP includes four global parameters (total size, inclination, X direction position, Y direction position), and n shape parameters p _i (i = 1 to n) that are feature amounts. .) Means the change amount of the value. Immediately after setting the feature point CP to the initial position, the global parameter is set to the value determined in the initial position setting process (FIG. 12) of the feature point CP. Further, difference between the initial position of the characteristic point CP and sets the position of the characteristic points CP of the average shape s ₀ in this case, the overall size, the tilt, because it is limited to the difference in position, shape parameters in the shape model p The values of _i are all zero.

  The parameter update amount ΔP is calculated by the following equation (3). That is, the parameter update amount ΔP is a product of the update matrix R and the difference image Ie.

The update matrix R in the expression (3) is a matrix of M rows and N columns set in advance by learning in order to calculate the parameter update amount ΔP based on the difference image Ie, and the internal memory 120 as the AAM information AMI (FIG. 1). Stored in In this embodiment, the number of rows M of the update matrix R includes a number of global parameters (4), equal to the sum of the number of shape parameters p _i (n-number) ((4 + n) pieces), the number of columns N is , Equal to the number of pixels in the average shape area BSA of the average face image A ₀ (x) (56 pixels × 56 pixels−the number of pixels in the mask area MA). The update matrix R is calculated by the following formulas (4) and (5).

Function W in the formula (4) and (5), the warp W; refers to (x p), the variable P refers to the shape parameter _{p i,} T denotes a transposed matrix.

  The correcting unit 226 (FIG. 1) updates parameters (four global parameters and n shape parameters) based on the calculated parameter update amount ΔP (step S450). Thereby, the setting position of the feature point CP in the detection target image CI is corrected (updated). The correction unit 226 corrects the setting position of the feature point CP in the detection target image CI so that the norm of the difference image Ie is small. That is, by updating the n shape parameters with the parameter update amount ΔP, a new shape s is generated according to the above-described equation (1) of the shape model, and thus the correction unit 226 has a feature in the generated shape s. The point CP is set in the detection target image CI. At this time, the correction unit 226 adjusts the size, inclination, and position of the generated shape s according to the four global parameters updated by the parameter update amount ΔP, and then detects the feature point CP. Set to image CI. Thus, the position of the feature point CP set in the detection target image CI in accordance with the parameter update is the set position of the corrected feature point CP.

  After the parameter update, the average shape image I (W (x; p)) is calculated again from the detection target image CI in which the installation position of the feature point CP is corrected (step S410), and the difference image Ie is calculated ( In step S420), a convergence determination (step S430) based on the difference image Ie is performed. If it is determined that the convergence has not been made in the convergence determination again, the parameter update amount ΔP based on the difference image Ie is calculated (step S440), and the setting position correction of the feature point CP by the parameter update (step S450). ) Is performed.

  When the processing from step S410 to step S450 in FIG. 15 is repeatedly executed, the position of the feature point CP corresponding to each feature part in the detection target image CI approaches the position of the actual feature part as a whole, and at a certain point in time. In convergence determination (step S430), it is determined that convergence has occurred. If it determines with having converged in convergence determination, the feature position detection part 220 can detect the position of the feature point CP currently set to the detection target image CI as a coordinate position of a feature part. In this embodiment, when the feature position detection unit 220 determines that the convergence has been made in the convergence determination, the feature position detection unit 220 ends the flowchart of FIG. 15 and further performs a setting position correction process for the feature point CP in the target image OI (see FIG. 10). Step S270).

FIG. 19 is a flowchart showing the setting position correction process of the feature point CP executed in step S270.
The feature position detection unit 220 reproduces the arrangement of the feature points CP finally set in the detection target image CI in the process of step S260 (FIGS. 10 and 15) on the target image OI, thereby generating a feature on the target image OI. A point CP is set (step S510). Specifically, the feature position detection unit 220 uses the detection target image CI (image whose size has been changed by the changing unit 260) as the arrangement relationship (mesh) of the feature points CP that is finally set in the detection target image CI. The size is enlarged or reduced according to the ratio between the size of the image and the size of the trimmed image. That is, if the changing unit 260 has reduced the trimmed image, the image is enlarged in step S510. If the changing unit 260 has enlarged the trimmed image, the image is reduced in step S510. Then, the inclination and position of the enlarged or reduced mesh are adjusted according to the global parameter finally set in the process of step S260 with reference to the face area FA in the target image OI, and then the mesh The feature point CP is set to the attention image OI.

  Next, the number of times the feature position detection unit 220 corrects (updates) the setting position of the feature point CP by updating the parameters after setting the feature point CP in step S510 (the process of step S570) is defined in advance. It is determined whether or not the number of times (the upper limit number) has been reached (step S520). If it is determined that the upper limit number has been reached, the feature position detection unit 220 completes the face feature position detection process (step S580). The upper limit number is a small number, for example, about three times.

If it is determined in step S520 that the number of corrections (updates) of the setting position of the feature point CP by the parameter update has not reached the upper limit number, the generation unit 222 (FIG. 1) displays the attention image OI at that time. By performing affine transformation so that the arrangement of the set feature points CP is equal to the arrangement of the feature points CP in the average shape s ₀ , the average shape image I (W) having the same size as the average face image A ₀ (x) (X; p)) is calculated (step S530). In the description of FIG. 19, the pixel group in the conversion target region (region surrounded by the feature point CP located on the outer periphery of the mesh) specified by the feature point CP set in the target image OI is represented by W (x; p).

The calculating unit 224 calculates a difference image Ie between the average shape image I (W (x; p)) generated in step S530 and the average face image A ₀ (x) (step S540). The feature position detection unit 220 determines whether or not the feature point CP setting position correction processing has converged based on the difference image Ie calculated in step S540 (step S550). That is, the feature position detection unit 220 calculates the norm of the difference image Ie, and determines that it has converged when the norm value is smaller than a preset threshold value. When it is determined that the feature position detection unit 220 has converged, the facial feature position detection process is completed (step S580), as in the case of the “Yes” determination in step S520. Note that the threshold value used in the convergence determination in step S550 is set to be equal to or lower than the threshold value used in the convergence determination in step S430 (FIG. 15). In this case as well, similarly to step S430, the feature position detection unit 220 determines that convergence has occurred when the norm value of the calculated difference image Ie is smaller than the norm value calculated in the previous step S550. May be. Or you may determine with having converged only when the value of the calculated norm is smaller than a threshold value and smaller than the previous value.

  When it is determined in step S550 that the convergence has not been achieved, the correction unit 226 (FIG. 1) calculates the parameter update amount ΔP (step S560), similarly to steps S440 and S450, and the calculated parameter update. The parameters (four global parameters and n shape parameters) are updated based on the quantity ΔP (step S570). Note that immediately after the feature point CP is set as the target image OI in step S510, the global parameters and the n shape parameters are values obtained by the last parameter update in the process of step S260 (FIGS. 10 and 15). Is set.

  As a result of step S570, the setting position of the feature point CP in the target image OI is corrected (updated). The correcting unit 226 corrects the setting position of the feature point CP in the target image OI so that the norm of the difference image Ie is reduced. That is, by updating the n shape parameters with the parameter update amount ΔP obtained in step S560, a new shape s is generated according to the above equation (1) of the shape model, and the correction unit 226 generates the generated shape s. The feature point CP in the selected shape s is set as the target image OI. At this time, the correction unit 226 adjusts the size, inclination, and position of the generated shape s according to the four global parameters updated with the parameter update amount ΔP obtained in step S560, and then the feature. The point CP is set to the attention image OI.

  After the parameter update, the determination in step S520 is made again. If the determination is “No”, the average shape image I (W (x; p) from the target image OI in which the installation position of the feature point CP is corrected. )) After the calculation (step S530) is repeated. When the processing from step S520 to S570 in FIG. 19 is repeatedly executed, the position of the feature point CP corresponding to each feature part in the target image OI approaches the position of the actual feature part as a whole. The feature position detecting unit 220 determines the set position of the feature point CP specified by the values of the global parameter and the shape parameter set at the time of completing the face feature position detection process (step S580) as the final target image. It is specified as the setting position of the feature point CP in OI.

  In the present embodiment, no limitation is imposed on the number of times regarding the repetition of the setting position correction processing of the feature point CP on the detection target image CI (the processing of steps S410 to S450 in FIG. 15) (the norm is optimized). Can be repeated). On the other hand, as described above, the upper limit of the number of times is set for the repetition of the setting position correction process of the feature point CP on the target image OI. This is because the position of the feature point CP is already in the target image OI at the time when the feature point CP is set in the target image OI through the setting position correction processing of the target point CP on the detection target image CI (step S510). This is because the overall position of the actual feature portion is quite close. In other words, regarding the setting position correction processing of the feature point CP in the target image OI, the setting position of the feature point CP is updated with a small number of times, and the setting position is accurately set to the position of the actual feature portion of the target image OI. Can be matched to. For this reason, the upper limit number of times as described above is provided in consideration of the balance between the accuracy of position detection of the characteristic part and the processing speed.

  FIG. 16 is an explanatory diagram illustrating an example of a result of the face feature position detection process. FIG. 16 shows the setting positions of the feature points CP finally specified in the target image OI. The coordinate position of the facial feature part (predetermined position in the facial organs (eyebrows, eyes, nose, mouth) and facial contour) included in the target image OI is specified by the setting position of the feature point CP. It is possible to detect the shape and position of the facial organ of the person and the contour shape of the face in the target image OI. In addition, when performing facial expression determination and face orientation determination, the facial expression position and facial orientation are determined by comparing the value of n shape parameters when the facial feature position detection processing is completed with a threshold value. Can do. In addition, when the face image is deformed, the shape of the face can be favorably deformed by changing the values of the n shape parameters when the face feature position detection process is completed.

  As described above, according to the image processing apparatus of the present embodiment, the face area FA is included, and all the image areas that can be the position setting target of the feature point CP when the initial position of the feature point CP is set are included. The range is trimmed from the target image OI, and the trimmed image is subjected to affine transformation (size transformation and rotation) so that the image size becomes a predetermined size. Then, based on the image of the predetermined size (detection target image CI) obtained by the conversion, the feature point CP is set (optimization of the set position) for detecting the coordinate position of the facial feature part. . As a result, the affine transformation of the image area specified by the feature point CP is performed a plurality of times in the initial position setting process (step S250 in FIG. 10) of the feature point CP and the setting position correction process of the feature point CP (step S260). In the case of generation of an average shape image by warp W), the scanning range of pixels is limited to the detection target image CI. For this reason, the initial process for setting the initial position of the feature point CP (step S250 in FIG. 10), the process for correcting the setting position of the feature point CP (step S260), and the entire face feature position detection process are accelerated.

  The image processing apparatus according to the present embodiment also performs image noise reduction processing when the trimmed image is affine transformed so as to have a predetermined size. That is, the detection target image CI is improved by improving the image quality of the detection target image CI that is the target of the initial position setting process of the feature point CP (step S250 in FIG. 10) and the setting position correction process of the feature point CP (step S260). The average shape image generated based on the image is also an image with less noise. Therefore, the convergence determination based on the comparison between the average shape image and the average face image (difference image Ie) and the update processing of the parameter amount are also more accurate without being affected by noise, and as a result, the position of the feature part of the face Detection accuracy is improved.

  According to the image processing apparatus according to the present embodiment, the setting unit 221 uses the global parameters to set the feature points CP, and thus detects the position of the facial feature part included in the image efficiently and at high speed. Can do. Specifically, multiple global parameters (size, inclination, vertical position, horizontal position) are changed to prepare multiple temporary setting positions for feature points CP that form various meshes. The temporary setting position corresponding to the difference image Ie having the smallest norm value is set as the initial position. Thereby, the initial position of the feature point CP in the detection target image CI can be set closer to the position of the feature part of the face. Therefore, in the setting position correction process of the feature point CP, the correction by the correction unit 226 is facilitated, so that the process of detecting the position of the feature part of the face can be made more efficient and faster.

4). Modifications The present invention is not limited to the above-described examples and embodiments, and can be implemented in various modes without departing from the scope of the invention. For example, the present invention can be modified as follows.

The conversion unit 260 (FIG. 1) may perform a correction process on the trimmed image so that at least one element of brightness, contrast, and color approaches the average face image A ₀ (x). In this case, the printer 100 stores the brightness reference value, the luminance range (dynamic range), and the color reference value for the average face image A ₀ (x) in the internal memory 120 in advance. That is, the computer 300 that performs the AAM setting process generates a reference value of brightness, a luminance range, and a reference value of color based on the average face image A ₀ (x) in advance, and uses the generated values for the printer 100. To keep. The brightness reference value referred to here is, for example, the average of the luminance values in the average shape area BSA of the average face image A ₀ (x). The luminance range is, for example, a luminance range in the average shape area BSA of the average face image A ₀ (x). The color reference value is, for example, a ratio (color balance) of average values for R, G, and B in the average shape area BSA of the average face image A ₀ (x). Note that such a brightness reference value, luminance range, and color reference value may be generated by the printer 100 (image processing apparatus) itself based on the average face image A ₀ (x) as the AAM information AMI. Good.

Under such a premise, the conversion unit 260 calculates a luminance average value of the trimmed image, calculates a difference between the luminance average value and the brightness reference value of the average face image A ₀ (x), and calculates the difference. The brightness of the trimmed image is corrected according to (for example, by a γ curve having a correction degree corresponding to the difference). In addition, the conversion unit 260 expands the brightness range of the trimmed image so that the brightness range of the trimmed image is substantially equal to the brightness range of the average face image A ₀ (x) (contrast expansion processing is performed). In addition, the conversion unit 260 obtains the ratio of the average values of the trimmed images for each RGB, and the trimmed image has a ratio substantially equal to the reference color value of the average face image A ₀ (x). The distribution for each RGB is corrected. Then, the conversion unit 260 performs affine transformation on the trimmed image in which all or part of the brightness, contrast, and color are corrected, and generates the detection target image CI having the predetermined size.

As a result, the average shape image I (W (x; p)) generated by affine transformation of the detection target image CI is also similar in brightness, contrast, and color to the average face image A ₀ (x). Therefore, the comparison result (difference image Ie or norm) between the average shape image I (W (x; p)) and the average face image A ₀ (x) is purely set to the detection target image CI at that time. This represents the quality of the position of the feature point CP (the degree of coincidence with the position of the feature part), and thus the speed and accuracy of the position detection of the feature part can be further improved.

  In the above-described example, the processing based on the detection target image CI is performed for both the initial position setting of the feature point CP (step S250 in FIG. 10) and the setting position correction of the feature point CP (step S260). However, only one of the initial position setting of the feature point CP and the setting position correction of the feature point CP may be processed based on the detection target image CI.

  When the image processing apparatus performs only the initial position setting process of the feature point CP based on the detection target image CI, the feature point of the initial position set in the detection target image CI when step S250 is completed in the flowchart of FIG. The arrangement of the CP is reproduced on the target image OI based on the ratio between the size of the detection target image CI and the size of the trimmed image and the global parameter at that time. Then, for the feature point CP reproduced in the target image OI, a setting position correction process on the target image OI is performed. When the setting position correction processing after the initial position setting processing of the feature point CP is performed only on the target image OI, the setting position correction processing (generation of average shape image, calculation of difference image Ie, convergence determination, parameter update amount) Regarding the repetition of ΔP generation and parameter correction), there is no limit on the number of times. Thus, even when only the initial position setting process of the feature point CP is performed based on the detection target image CI, the time required for the entire face feature position detection process can be increased by speeding up the initial position setting process. Is shortened.

  In the case where only the setting position correction of the feature point CP is performed based on the detection target image CI, the image processing apparatus performs the initial operation of the feature point CP in the target image OI after step S220 (face area FA detection) in the flowchart of FIG. Perform position setting processing. Then, a range including the initial position of the feature point CP set in the face area FA and the target image OI is trimmed, and the trimmed image is affine transformed to generate a detection target image CI of a predetermined size. Then, a process for correcting the set position of the feature point CP (initial position) in the detection target image IC is performed. As described above, since the noise reduction process is performed on the detection target image CI, the setting position correction process of the feature point CP can be performed based on the high-quality detection target image CI. That is, since the setting position correction process that requires particularly high accuracy can be performed without the influence of noise, the setting position of the feature point CP that is finally specified as representing the coordinate position of the facial feature part Is a position corresponding to the position of the actual characteristic part very accurately.

In the initial position setting process and the setting position correction process of the feature point CP described above, the average shape image I (W (x; p)) is calculated based on the detection target image CI and the target image OI, thereby detecting the detection target image CI and The setting position of the feature point CP of the target image OI is matched with the setting position of the feature point CP of the average face image A ₀ (x). However Conversely, by performing the image conversion for the average face image A _{0 (x),} the arrangement of the characteristic points CP of the average feature point CP of the detection target image CI and the target image OI face image A _{0 (x)} May be matched.

In addition, in the feature point CP initial position setting process, temporary setting positions corresponding to combinations of three levels of values for each of the four global parameters (size, inclination, vertical position, and horizontal position) are set in advance. However, the type and number of parameters used for setting the temporary setting position and the number of parameter values can be changed.
Further, in the above, the texture model is set by principal component analysis on the luminance value vector formed by the luminance values in each of the pixel groups x of the sample image SIw, but other than the luminance values representing the texture (appearance) of the face image. A texture model may be set by principal component analysis with respect to an index value (for example, RGB value).

The size of the average face image A ₀ (x) is not limited to 56 pixels × 56 pixels, and may be other sizes. Further, the average face image A ₀ (x) does not need to include the mask area MA (FIG. 8), and may include only the average shape area BSA. Further, instead of the average face image A ₀ (x), another reference face image set based on the statistical analysis of the sample image SI may be used.
In the above, the shape model and the texture model are set using AAM. However, the shape model and the texture model can be set using another modeling method (for example, a method called Morphable Model or a method called Active Blob). Setting may be performed.

In the above description, the image stored in the memory card MC is set as the attention image OI. However, the attention image OI may be an image acquired via a network, for example. Also, the detection mode information may be acquired via a network.
In the above description, image processing by the printer 100 as the image processing apparatus has been described. However, part or all of the processing is executed by another type of image processing apparatus such as the computer 300, a digital still camera, or a digital video camera. It is good. The printer 100 is not limited to an ink jet printer, and may be another type of printer such as a laser printer or a sublimation printer.

DESCRIPTION OF SYMBOLS 100 ... Printer, 110 ... CPU, 120 ... Internal memory, 140 ... Operation part, 150 ... Display part, 160 ... Printing mechanism, 170 ... Card interface, 172 ... Card slot, 200 ... Image processing part, 220 ... Feature position detection part 221 ... setting unit, 222 ... generating unit, 224 ... calculating unit, 226 ... correcting unit, 230 ... face area detecting unit, 250 ... trimming unit, 260 ... converting unit, 300 ... computer, 310 ... display processing unit, 320 ... Print processing section

Claims (8)

An image processing apparatus for detecting a coordinate position of a feature part of a face included in an image,
A face area detection unit that detects an image area including at least a part of a face from a target image as a face area;
A trimming unit that trims an image of a predetermined range including the face area from the attention image;
A conversion unit that converts the size of the trimmed image into a predetermined size;
An image processing apparatus comprising: a feature position detection unit that detects a coordinate position of the feature part based on the image whose size has been converted.   The image processing apparatus according to claim 1, wherein the conversion unit performs the conversion of the size together with a process for reducing noise on the trimmed image.   The image processing apparatus according to claim 2, wherein the conversion unit executes a process for noise reduction using a noise reduction filter that varies depending on a size of the trimmed image.   The trimming unit trims a range including all of a range specified on the basis of the face area according to each parameter defined in advance representing the size, angle, and position of the face image with respect to the face area. The image processing apparatus according to claim 1. The feature position detector
Set a feature point for detecting the coordinate position of the feature part for the image whose size has been converted,
Generate an average shape image that is an image obtained by converting a part of the image whose size has been converted so that the arrangement of the feature points is a predetermined arrangement,
Calculating a difference value between the average shape image and an average face image that is an image generated by averaging a plurality of sample images;
Based on the calculated difference value, by updating the setting position of the feature point in the image whose size has been converted, the setting position is corrected to be closer to the coordinate position of the feature part,
The image processing apparatus according to claim 1, wherein the corrected set position is detected as a coordinate position of the characteristic part. The feature position detector
Until the difference value becomes a value smaller than a predetermined threshold, repeatedly update the setting position of the feature point in the image whose size has been converted,
When the difference value is smaller than the threshold value, the arrangement of the feature points in the image whose size has been converted depends on the ratio of the size of the image whose size has been changed and the trimmed image. By recreating the featured image in the featured image, set feature points on the featured image,
An average shape image that is an image obtained by converting a part of the target image is generated so that the arrangement of the feature points in the target image becomes the predetermined arrangement, and a difference value between the average shape image and the average face image is generated. Based on this, by updating the setting position of the feature point in the image of interest, the setting position of the feature point in the image of interest is corrected to be close to the coordinate position of the feature part,
6. The image processing apparatus according to claim 5, wherein a setting position of a feature point in the corrected attention image is detected as a coordinate position of the feature part.   The image processing apparatus according to claim 6, wherein the feature position detection unit executes a process of updating a setting position of a feature point in the target image with a predetermined number of times as an upper limit.   The conversion unit performs the conversion of the size together with a correction process for bringing at least one element of brightness, contrast, and color of the trimmed image closer to the average face image. The image processing apparatus according to claim 5.