JP2010244251A - Image processor for detecting coordinate position for characteristic site of face - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To achieve the efficiency and high speed operation of processing for detecting the position of the characteristic site of the face included in an image. <P>SOLUTION: An image processor for detecting the coordinate position of the characteristic site of the face included in an image under consideration includes: a face region detection part for detecting an image region including at least a portion of the face image from the image under consideration as a face region; an initial position setting part for setting the initial position of a characteristic point set in the image under consideration for detecting the coordinate position of the characteristic site from the candidates of a plurality of initial positions to be set based on the face region detection information as information related with the detection of the face image; and a characteristic position detection part for correcting the setting position of the characteristic point set at the initial position to be close to the position of the characteristic site, and for detecting the corrected setting position as the coordinate position of the characteristic site. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that detects a coordinate position of a feature portion of a face included in an attention image.

  As a visual event modeling method, an active appearance model (Active Appearance Model, also referred to as “AAM” for short) is known. In AAM, for example, through statistical analysis of the positions (coordinates) and pixel values (for example, luminance values) of facial features (for example, the corners of the eyes, the nose and the face lines) included in a plurality of sample images, A shape model representing the shape of the identified face and a texture model representing “appearance” in the average shape are set, and the face image is modeled using these models. According to AAM, it is possible to model (synthesize) an arbitrary face image and to detect the position of a facial feature part included in the image (Patent Document 1).

JP 2007-141107 A

  However, the above-described conventional technology has room for further efficiency and speed-up regarding the detection of the position of the facial feature part included in the image.

  Such a problem is not limited to the case of using AAM, but is a problem common to image processing for detecting the position of a facial feature part included in an image.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to improve the efficiency and speed of processing for detecting the position of a facial feature part included in an image.

  In order to solve at least a part of the above problems, the present invention employs the following aspects.

  A first aspect provides an image processing apparatus that detects a coordinate position of a feature portion of a face included in an attention image. An image processing apparatus according to the first aspect of the present invention detects a coordinate area of a feature area and a face area detection unit that detects an image area including at least a part of a face image from the target image as a face area. An initial position setting unit that sets an initial position of a feature point set in the target image from a plurality of candidates for the initial position set based on face area detection information that is information related to detection of the face area And a feature position detector that corrects the set position of the feature point set to the initial position so as to approach the position of the feature part, and detects the corrected set position as a coordinate position of the feature part; Is provided.

  According to the image processing apparatus according to the first aspect, the initial position of the feature point is set from a plurality of candidates for the initial position set based on the face area detection information, so the initial position is set to a good position. can do. Thereby, it is possible to increase the efficiency and speed of the process of detecting the position of the facial feature part included in the target image.

  In the image processing device according to the first aspect, the face area detection information is information specified along with the detection by the face area detection unit, and the initial position setting unit is the specified face area detection information. And an initial position candidate setting unit that sets the initial position candidate based on the acquired face area detection information. In this case, since the initial position of the feature point is set from one of the initial position candidates set by the initial position candidate setting unit, the position of the feature part of the face included in the target image is detected efficiently and at high speed. Can do.

  In the image processing apparatus according to the first aspect, the face area detection information includes a face area that represents a probability that the face image included in the face area detected by the face area detection unit is a true face image. Including the reliability, the initial position candidate setting unit may increase the number of candidates for the initial position to be set when the face area reliability is low compared to when the face area reliability is high. . In this case, when the face area reliability is low, by increasing the number of candidates for the initial position, it is possible to efficiently detect the position of the facial feature part included in the target image.

  In the image processing device according to the first aspect, the face area detection information includes angle information related to a rotation angle in the image plane of the face image included in the face area detected by the face area detection unit, and The position candidate setting unit may rotate and set the predetermined initial position candidates according to the rotation angle based on the angle information. In this case, the position of the facial feature part included in the image of interest can be detected efficiently and at high speed by setting the initial position candidates by rotating them according to the rotation angle.

  In the image processing device according to the first aspect, the face area detection information includes information on a rotation angle in the image plane of the face image that can be specified by the face area detection unit, and the plurality of initial position candidates are For each of the rotation angles that can be specified by the face area detection unit, from the intermediate value between the rotation angle and the one of the specified rotation angles that are adjacent to each other, the other specified rotation angle that has the adjacent value May be set in a range up to an intermediate value. In this case, by using the rotation angle of the face image included in the face area, a plurality of initial position candidates are set within a predetermined angle range centered on the specified rotation angle, whereby the face included in the target image It is possible to efficiently and rapidly detect the position of the characteristic part.

  In the image processing device according to the first aspect, the face area detection information includes information on a tendency of a relative position of a face image with respect to the face area detected by the face area detection unit, and the plurality of initial positions The relative position with respect to the face area may be determined according to the tendency. In this case, since the relative position with respect to the candidate face area of the initial position is determined according to the tendency of the relative position of the face image with respect to the face area, the position of the facial feature portion included in the target image is efficiently determined. And can be detected at high speed.

  In the image processing apparatus according to the first aspect, the initial position setting unit is an average that is an image obtained by converting a part of the target image based on the feature points set at positions that are candidates for the initial position. A difference value between a generation unit that generates a shape image, the average shape image, and an average face image that is an image generated based on a plurality of sample images including a face image in which the coordinate position of the characteristic part is known. A calculation unit for calculating, and among the plurality of initial position candidates, an initial position candidate having a minimum difference value may be set as the initial position. In this case, by setting the initial position candidate having the smallest difference value as the initial position, the position of the facial feature part included in the target image can be detected efficiently and at high speed.

  In the image processing device according to the first aspect, the feature position detection unit is configured to reduce the difference value based on a difference value between an average shape image corresponding to the initial position and the average face image. While providing the correction | amendment part which correct | amends the said setting position, you may detect the said setting position where the said difference value becomes predetermined as said coordinate position. In this case, since the coordinate position of the feature part is detected based on the difference value between the average shape image corresponding to the initial position and the average face image, the position of the feature part of the face included in the target image can be efficiently and It can be detected at high speed.

  In the image processing apparatus according to the first aspect, the characteristic part may be a part of eyebrows, eyes, nose, mouth, and face line. In this case, the coordinate positions can be satisfactorily detected for the eyebrows, eyes, nose, mouth, face line, and part.

  Note that the present invention can be realized in various modes, for example, a printer, a digital still camera, a personal computer, a digital video camera, and the like. Also, an image processing method and apparatus, a position detection method and apparatus for a characteristic part, a facial expression determination method and apparatus, a computer program for realizing the functions of these methods or apparatuses, a recording medium recording the computer program, and the computer program Including a data signal embodied in a carrier wave.

1 is an explanatory diagram schematically showing a configuration of a printer 100 as an image processing apparatus in a first embodiment of the present invention. FIG. It is a flowchart which shows the flow of the AAM setting process in 1st Example. It is explanatory drawing which shows an example of sample image SI. It is explanatory drawing which shows an example of the setting method of the feature point CP in the sample image SI. It is explanatory drawing which shows an example of the coordinate of the feature point CP set to sample image SI. It is explanatory drawing which shows an example of average shape s0. It is explanatory drawing which illustrated the relationship between shape vector si and shape parameter pi, and face shape s. It is explanatory drawing which shows an example of the method of the warp W of the sample image SI. It is explanatory drawing which shows an example of average face image A0 (x). It is a flowchart which shows the flow of the face feature position detection process in 1st Example. It is a flowchart which shows the flow of the detection process of face area FA. It is explanatory drawing for demonstrating the detection of the face area FA in the attention image OI. It is explanatory drawing for demonstrating the filter used for calculation of evaluation value Tv. It is explanatory drawing which illustrated the state which moved window SW. It is explanatory drawing which illustrated several window SW determined to be an image area | region corresponding to the image of a face. It is explanatory drawing which shows an example of the sample image used for learning. It is a flowchart which shows the flow of the initial position setting process of the feature point CP in 1st Example. It is explanatory drawing which illustrated the temporary setting position of the feature point CP by changing the value of a global parameter. It is explanatory drawing which illustrated the temporary setting position of the feature point CP in case the specific face inclination of the face area FA is 30 degree | times. It is explanatory drawing for demonstrating the step number of the change of a temporary setting position. It is explanatory drawing which shows an example of the average shape image I (W (x; p)). It is a flowchart which shows the flow of the feature point CP setting position correction process in 1st Example. It is explanatory drawing which shows an example of the result of a face feature position detection process. It is explanatory drawing which showed the 2nd example about the temporary setting position of face area | region detection information and the feature point CP. It is explanatory drawing which showed the 3rd example about the temporary setting position of face area | region detection information and the feature point CP.

  Hereinafter, a printer which is an aspect of an image processing apparatus according to the present invention will be described based on examples with reference to the drawings.

A. First embodiment:
A1. Configuration of image processing device:
FIG. 1 is an explanatory diagram schematically showing the configuration of a printer 100 as an image processing apparatus according to the first embodiment of the present invention. The printer 100 of this embodiment is an ink jet color printer that supports so-called direct printing, in which an image is printed based on image data acquired from a memory card MC or the like. The printer 100 includes a CPU 110 that controls each unit of the printer 100, an internal memory 120 configured by a ROM and a RAM, an operation unit 140 configured by buttons and a touch panel, a display unit 150 configured by a liquid crystal display, and printing. A mechanism 160 and a card interface (card I / F) 170 are provided. The printer 100 may further include an interface for performing data communication with other devices (for example, a digital still camera or a personal computer). Each component of the printer 100 is connected via a bus so that bidirectional communication is possible.

  The printing mechanism 160 performs printing based on the print data. The card interface 170 is an interface for exchanging data with the memory card MC inserted into the card slot 172. In this embodiment, an image file including image data is stored in the memory card MC.

  The internal memory 120 stores an image processing unit 200, a display processing unit 310, and a print processing unit 320. The image processing unit 200 is a computer program, and performs facial feature position detection processing by being executed by the CPU 110 under a predetermined operating system. The face feature position detection process is a process for detecting the position of a predetermined feature part (for example, the corner of the eye, the nose head, or the face line) in the face image. The face feature position detection process will be described in detail later. The display processing unit 310 and the print processing unit 320 are also executed by the CPU 110 to realize the respective functions.

  The image processing unit 200 includes a face area detection unit 210, a feature position detection unit 220, and an initial position setting unit 230 as program modules. The initial position setting unit 230 includes an acquisition unit 232, an initial position candidate setting unit 234, a generation unit 236, and a calculation unit 238. The functions of these units will be described in detail in the description of the face feature position detection process described later.

  The display processing unit 310 is a display driver that controls the display unit 150 to display processing menus, messages, images, and the like on the display unit 150. The print processing unit 320 is a computer program for generating print data from image data, controlling the printing mechanism 160, and printing an image based on the print data. The CPU 110 reads out and executes these programs (the image processing unit 200, the display processing unit 310, and the print processing unit 320) from the internal memory 120, thereby realizing the functions of these units.

  The internal memory 120 also stores AAM information AMI and face learning data FLD. The AAM information AMI is information set in advance by an AAM setting process described later, and is referred to in a face feature position detection process described later. The contents of the AAM information AMI will be described in detail in the description of AAM setting processing described later. The face learning data FLD is used for the detection of the face area FA by the face area detection unit 2110. The contents of the face learning data FLD will be described in detail in the description of the face area FA detection process described later.

A2. AAM setting process:
FIG. 2 is a flowchart showing the flow of AAM setting processing in the first embodiment. The AAM setting process is a process for setting a shape model and a texture model used for modeling an image called AAM (Active Appearance Model). In this embodiment, the AAM setting process is performed by the user.

  First, the user prepares a plurality of images including a person's face as sample images SI (step S110). FIG. 3 is an explanatory diagram showing an example of the sample image SI. As shown in FIG. 3, the sample image SI is related to various attributes such as personality, race / gender, facial expression (anger, laughter, trouble, surprise, etc.), direction (front direction, upward, downward, rightward, leftward, etc.). It is prepared to include different face images. If the sample image SI is prepared in this way, any face image can be accurately modeled by AAM, and accurate face feature position detection processing (described later) for any face image can be executed. It becomes. The sample image SI is also called a learning image.

  A feature point CP is set to the face image included in each sample image SI (step S120). FIG. 4 is an explanatory diagram showing an example of a method for setting the feature point CP in the sample image SI. The feature point CP is a point indicating the position of a predetermined feature part in the face image. In this embodiment, as predetermined feature parts, a predetermined position on the eyebrows in a person's face (for example, an end point or a four-divided point, the same applies hereinafter), a predetermined position on the eye contour, and a predetermined position on the nose and nose contours 68 parts such as a predetermined position on the contour of the upper and lower lips and a predetermined position on the contour of the face (face line) are set. That is, in the present embodiment, a predetermined position in the facial organs (eyebrows, eyes, nose, mouth) and facial contour that are commonly included in the human face are set as the characteristic parts. As shown in FIG. 4, the feature points CP are set (arranged) at positions representing 68 feature parts designated by the operator in each sample image SI. Since each feature point CP set in this way corresponds to each feature part, the arrangement of the feature points CP in the face image can be expressed as specifying the shape of the face.

  The position of the feature point CP in the sample image SI is specified by coordinates. FIG. 5 is an explanatory diagram showing an example of the coordinates of the feature point CP set in the sample image SI. In FIG. 5, SI (j) (j = 1, 2, 3,...) Indicates each sample image SI, and CP (k) (k = 0, 1,..., 67) indicates each feature. Point CP is shown. CP (k) -X represents the X coordinate of the feature point CP (k), and CP (k) -Y represents the Y coordinate of the feature point CP (k). The coordinates of the feature point CP include predetermined reference points (for example, in the sample image SI normalized with respect to the size of the face, the inclination of the face (inclination in the image plane), and the position of the face in the X direction and the Y direction) The coordinates with the origin at the lower left point of the image are used. Further, in the present embodiment, a case where a plurality of human face images are included in one sample image SI is allowed (for example, two sample face images are included in the sample image SI (2)). Each person in one sample image SI is specified by a person ID.

  Subsequently, the user sets an AAM shape model (step S130). Specifically, a principal component analysis is performed on a coordinate vector (see FIG. 5) composed of the coordinates (X coordinate and Y coordinate) of 68 feature points CP in each sample image SI, and specified by the position of the feature point CP. The face shape s to be modeled is expressed by the following equation (1). The shape model is also referred to as a feature point CP arrangement model.

In the above formula (1), s ₀ is an average shape. FIG. 6 is an explanatory diagram showing an example of the average shape s ₀ . As shown in FIGS. 6A and 6B, the average shape s ₀ is a model representing the average face shape specified by the average position (average coordinate) for each feature point CP of the sample image SI. is there. In the present embodiment, in the average shape s ₀ , a region surrounded by a straight line connecting feature points CP (face lines, eyebrows, feature points CP corresponding to the eyebrows, see FIG. 4) located on the outer periphery (FIG. 6 ( b) is indicated by hatching) and is referred to as “average shape region BSA”. In the average shape s ₀ , as shown in FIG. 6A, a plurality of triangular areas TA having the feature points CP as vertices are set so as to divide the average shape area BSA into a mesh shape.

In the above equation (1) representing the shape model, s _i is a shape vector, and p _i is a shape parameter representing the weight of the shape vector s _i . The shape vector s _i is a vector representing the characteristics of the face shape s and is an eigenvector corresponding to the i-th principal component obtained by principal component analysis. As shown in the above equation (1), in the shape model in the present embodiment, the face shape s representing the arrangement of the feature points CP is modeled as the sum of the average shape s ₀ and the linear combination of the n shape vectors s _i. It becomes. By appropriately setting the shape parameter p _i in the shape model, the face shape s in any image can be reproduced.

FIG. 7 is an explanatory diagram illustrating the relationship between the shape vector s _{i, the} shape parameter p _i, and the face shape s. As shown in FIG. 7A, in order to specify the face shape s, the number n set based on the cumulative contribution rate in order from the eigenvector corresponding to the principal component having the larger contribution rate (in FIG. 7, n = The eigenvector of 4) is adopted as the shape vector s _i . Each of the shape vectors s _i corresponds to the movement direction / movement amount of each feature point CP, as indicated by the arrows in FIG. In this embodiment, most first shape vector s ₁ corresponding to the first principal component having a large contribution rate is a vector that is approximately correlated with the left and right appearance of a face, by the magnitude of the shape parameter p _1, 7 As shown in (b), the horizontal face direction of the face shape s changes. Second shape vector s ₂ corresponding to the large second principal component of the contribution rate to the second is a vector that is approximately correlated with the vertical appearance of a face, by the magnitude of the shape parameter p _2, FIG. 7 (c) As shown in FIG. 4, the vertical face direction of the face shape s changes. The third shape vector s ₃ corresponding to the third principal component having the third largest contribution ratio is a vector that is substantially correlated with the aspect ratio of the face shape, and the fourth principal component having the fourth largest contribution ratio. The fourth shape vector s ₄ corresponding to is a vector that is substantially correlated with the degree of mouth opening. As described above, the value of the shape parameter represents the feature of the face image such as facial expression and face orientation.

The average shape s ₀ and shape vector s _i set in the shape model setting step (step S130) are stored in the internal memory 120 as AAM information AMI (FIG. 1).

Subsequently, an AAM texture model is set (step S140). Specifically, first, for each sample image SI, image conversion (hereinafter referred to as “warp”) is performed so that the setting position of the feature point CP in the sample image SI is equal to the setting position of the feature point CP in the average shape s ₀ . W ”).

FIG. 8 is an explanatory diagram showing an example of a method of warping W of the sample image SI. In each sample image SI, similarly to the average shape s ₀ , a plurality of triangular regions TA that divide the region surrounded by the feature points CP located on the outer periphery into mesh shapes are set. The warp W is a set of affine transformations for each of the plurality of triangular areas TA. That is, in the warp W, an image of a certain triangular area TA in the sample image SI is affine transformed into an image of the corresponding triangular area TA in the average shape s ₀ . With the warp W, a sample image SI (hereinafter referred to as “sample image SIw”) in which the set position of the feature point CP is equal to the set position of the feature point CP in the average shape s ₀ is generated.

  Each sample image SIw has a rectangular frame containing an average shape area BSA (shown with hatching in FIG. 8) as an outer periphery, and an area other than the average shape area BSA (hereinafter also referred to as “mask area MA”). Generated as a masked image. An image area in which the average shape area BSA and the mask area MA are combined is referred to as a reference area BA. Each sample image SIw is normalized as an image having a size of 56 pixels × 56 pixels, for example.

  Next, a principal component analysis is performed on a luminance value vector composed of luminance values in each pixel group x of each sample image SIw, and a facial texture (also referred to as “appearance”) A (x) is expressed by the following formula ( 2). The pixel group x is a set of pixels located in the average shape area BSA.

In the above equation (2), A ₀ (x) is an average face image. FIG. 9 is an explanatory diagram showing an example of the average face image A ₀ (x). The average face image A ₀ (x) is an image representing the average face of the sample image SIw (see FIG. 8) after the warp W. That is, the average face image A ₀ (x) is an image calculated by taking the average of the pixel values (luminance values) of the pixel group x in the average shape area BSA of the sample image SIw. Therefore, the average face image A ₀ (x) is a model representing the average face texture (appearance) in the average face shape. Note that the average face image A ₀ (x) is composed of the average shape area BSA and the mask area MA, similarly to the sample image SIw, and is calculated as an image having a size of 56 pixels × 56 pixels, for example.

In the above equation (2) representing the texture model, A _i (x) is a texture vector, and λ _i is a texture parameter representing the weight of the texture vector A _i (x). The texture vector A _i (x) is a vector representing the characteristics of the facial texture A (x), and specifically, is an eigenvector corresponding to the i-th principal component obtained by principal component analysis. That is, the number m of eigenvectors set based on the cumulative contribution rate is adopted as the texture vector A _i (x) in order from the eigenvector corresponding to the principal component having the larger contribution rate. In the present embodiment, the first texture vector A ₁ (x) corresponding to the first principal component having the largest contribution rate is a vector that is substantially correlated with a change in face color (also regarded as a gender difference).

As shown in the above equation (2), in the texture model in the present embodiment, the facial texture A (x) representing the appearance of the face is the average face image A ₀ (x) and m texture vectors A _i (x ) And the linear combination. By appropriately setting the texture parameter λ _i in the texture model, it is possible to reproduce the facial texture A (x) in any image. Note that the average face image A ₀ (x) and the texture vector A _i (x) set in the texture model setting step (step S140 in FIG. 2) are stored in the internal memory 120 as AAM information AMI (FIG. 1). .

By the AAM setting process described above (FIG. 2), a shape model for modeling the face shape and a texture model for modeling the face texture are set. By combining the set shape model and the texture model, that is, the synthesized texture A (x) is converted from the average shape s ₀ to the shape s (inverse conversion of the warp W shown in FIG. 8). Thus, it is possible to reproduce the shape and texture of any face image.

A3. Face feature position detection processing:
FIG. 10 is a flowchart showing the flow of face feature position detection processing in the first embodiment. The face feature position detection process in the present embodiment is a process for detecting the position of the feature part in the face image by determining the arrangement of the feature points CP in the face image included in the target image using AAM. As described above, in the present embodiment, in the AAM setting process (FIG. 2), a total of 68 predetermined positions in the human facial organs (eyebrows, eyes, nose, mouth) and facial contour are set as the characteristic parts. (See FIG. 4). Therefore, in the face feature position detection process of the present embodiment, the positions of the feature parts are detected by specifying the positions of 68 feature points CP representing predetermined positions in the organ and face contour of the person.

Note that when the arrangement of the feature points CP in the face image is determined by the face feature position detection process, the shape parameter p _i and the value of the texture parameter λ _i for the face image are specified. Therefore, the processing result of the face feature position detection process is a facial expression determination for detecting a facial image of a specific facial expression (for example, a smile or a face with closed eyes), or a facial image in a specific direction (for example, rightward or downward). It can be used for face orientation determination for detection, face deformation for deforming the face shape, face shadow correction, and the like.

  First, the image processing unit 200 (FIG. 1) acquires image data representing an attention image that is a target of face feature position detection processing (step S210). In the printer 100 of the present embodiment, when the memory card MC is inserted into the card slot 172, thumbnail images of image files stored in the memory card MC are displayed on the display unit 150. One or more images to be processed are selected by the user via the operation unit 140. The image processing unit 200 acquires an image file including image data corresponding to one or more selected images from the memory card MC and stores it in a predetermined area of the internal memory 120. The acquired image data is called attention image data, and an image represented by the attention image data is called attention image OI.

  The face area detector 210 (FIG. 1) detects an image area including at least a part of the face image included in the target image OI as the face area FA (step S220). FIG. 11 is a flowchart showing a flow of processing for detecting the face area FA. FIG. 12 is an explanatory diagram for explaining the detection of the face area FA in the target image OI. As shown in FIG. 12, the face area detection unit 2 10 sets one of a plurality of windows SW having square shapes of various sizes defined in advance as the attention image OI (step S300). Specifically, the face area detection unit 210 first sets a window SW having a size set as an initial value at a position on the target image OI set as an initial position.

  The face area detecting unit 2110 calculates an evaluation value Tv to be used for face determination from the image area defined by the window SW (step S310). Here, the face determination means determination whether or not the image area defined by the window SW is a face area corresponding to the face image. In the present embodiment, face determination is executed for each specific face inclination set in advance. That is, for each specific face inclination, it is determined whether or not the image area defined by the window SW is an image area corresponding to a face image inclined by the specific face inclination. Therefore, the evaluation value Tv is also calculated for each specific face inclination. Here, the specific face inclination means the rotation angle of the face image in the image plane (in-plane) as shown in the lower part of FIG. In this embodiment, the specific face inclination is based on the state where the face image is positioned along the vertical direction of the image (the state where the head faces upward and the jaw faces downward) (specific face inclination = 0). 12 degrees (0 degrees, 30 degrees, 60 degrees,..., 330 degrees) are set by increasing the inclination of the face image by 30 degrees clockwise.

  The calculation method of the evaluation value Tv is not particularly limited, but in this embodiment, N filters (filter 1 to filter N) are used for calculating the evaluation value Tv. FIG. 13 is an explanatory diagram for explaining a filter used for calculating the evaluation value Tv. The external shape of each filter (filter 1 to filter N) has the same aspect ratio as that of the window SW (that is, has a square shape), and a positive region pa and a negative region ma are set for each filter. The face area detection unit 210 applies the filter X (X = 1, 2,..., N) in order to the image area defined by the window SW, and the basic evaluation value vX ( v1, v2,..., vN) are calculated. Specifically, the basic evaluation value vX is the sum of the luminance values of the pixels included in the image area corresponding to the minus area ma from the sum of the luminance values of the pixels included in the image area corresponding to the plus area pa of the filter X. Is the value obtained by subtracting.

  The face area detection unit 2 10 calculates the calculated basic evaluation value vX and the threshold thX (th1, th2,..., ThN) set in correspondence with each basic evaluation value vX (v1.v2..., VN). ) And each. In the present embodiment, the face area detection unit 210 determines that the image area defined by the window SW is an image area corresponding to the face image in the filter X in which the basic evaluation value vX is greater than or equal to the threshold thX. The value “1” is set as the output value of X. On the other hand, in the filter X whose basic evaluation value vX is smaller than the threshold thX, it is determined that the image area defined by the window SW is an image area that is not considered to correspond to the face image, and the value “ Set to “0”. Weight coefficients WeX (We1, We2,..., WeN) are set for each filter X, and the sum of products of output values and weighting coefficients WeX for all filters X is set as an evaluation value Tv. Note that the aspect of the filter X used for face determination, the threshold thX, the weighting coefficient WeX, and the threshold TH described below are set in advance for each of the 12 specific face inclinations, and are used as face learning data FLD (FIG. 1). Stored in the internal memory 120.

  The face area detection unit 210 compares the calculated evaluation value Tv with the threshold value TH (step S320). When the evaluation value Tv is greater than or equal to the threshold value TH for a specific face inclination (step S320: YES), the face area detection unit 2110 determines that the image area defined by the window SW is a face inclined by the specific face inclination. As the image area corresponding to the image, the position of the image area defined by the window SW, that is, the coordinates of the currently set window SW and the specific face inclination are stored (step S330). On the other hand, if the evaluation value Tv is smaller than the threshold value TH for any specific face inclination, the process of step S330 is skipped.

  FIG. 14 is an explanatory diagram illustrating a state in which the window SW is moved. The face area detection unit 210 determines whether or not the entire image of interest OI has been scanned with the window SW having the currently set size (step S340). If the entire image of interest OI has not been scanned yet (step S340: NO), as shown in FIG. 14, the window SW is moved in a predetermined direction by a predetermined movement amount (step S350). In this embodiment, the face area detection unit 210 moves the window SW to the right by a movement amount corresponding to 20% of the horizontal size of the window SW. In addition, when the window SW is arranged at a position where it cannot be moved further to the right, the window SW is returned to the left end of the target image OI and is moved by 20% of the vertical size of the window SW. It is supposed to move downward. When the window SW is arranged at a position where it cannot be moved further downward, the entire target image OI is scanned. After moving the window SW, the face area detection unit 210 performs the processing after step S310 described above for the moved window SW.

  If the face area detection unit 210 determines that the entire attention image OI has been scanned by the window SW having the currently set size (step S340: YES), the attention image is displayed by the windows SW having all the sizes set (prepared). It is determined whether the OI has been scanned (step S360). When there is an unused window SW size (step S360: NO), the face area detection unit 210 changes the size of the window SW used for scanning to the next smaller size than the currently set size ( Step S370). In other words, in the present embodiment, the face area detection unit 210 first scans with the maximum size window SW, and then uses the smaller size windows SW in order. After the size of the window SW is changed, the face area detection unit 210 executes the processes after step S300 described above for the window SW having the changed size.

  When the face area detection unit 2110 performs the scan using the windows SW of all sizes (step S360: YES), the face area detection unit 2110 executes a face area setting process (step S380). The face area detection unit 210 sets a face area FA as an image area corresponding to the face image based on the coordinates of the window SW and the specific face inclination which are determined and evaluated that the evaluation value Tv is equal to or greater than the threshold value TH. . Specifically, when the specific face inclination is 0 degree, the image area defined by the window SW is set as the face area FA as it is. On the other hand, when the specific face inclination is other than 0 degrees, the image area defined by the window SW is rotated clockwise by the specific face inclination around a predetermined point (for example, the center of gravity of the window SW). Is set as the face area FA.

  When there are a plurality of window SWs determined that the image area defined by the window SW is an image area corresponding to the face image, the face area detection unit 210 has a predetermined point (for example, the window SW) in each window SW. The average coordinate of the coordinates of the center of the window SW is used as the center of gravity, and one new window having the average size of the windows SW is detected as the face area FA. FIG. 15 is an explanatory diagram illustrating a plurality of windows SW that are determined to be image regions corresponding to a face image. As shown in FIG. 15A, for example, when it is determined that an image area defined by four windows SW (SW1 to SW4) partially overlapping each other is an image area corresponding to a face image, As shown in FIG. 15B, the average coordinate of the coordinates of the center of gravity of each of the four windows SW is taken as the center of gravity, and one window having the average size of the respective sizes of the four windows SW is taken as the face area FA. Set.

  After setting the face area FA, the face area detection unit 210 calculates the face area reliability (step S390). The face area reliability is an index that is calculated based on the detection process of the face area FA, and is an index that represents the probability that the detected face area FA is an image area that truly corresponds to a face image. . In the detection process of the face area FA, an image area that does not correspond to the face image, that is, an image area that does not include the face image at all or a part of the face image, but an image area that truly corresponds to the face image is included. There is a possibility that a missing image area is erroneously detected as the face area FA. The face area reliability represents the certainty that the detection of the face area FA is not a false detection but a correct detection.

  In this embodiment, a value obtained by dividing the number of overlapping windows by the maximum number of overlapping windows is used as the face area reliability. Here, the number of overlapping windows is the number of windows SW referred to when setting the face area FA, that is, the window SW determined that the image area defined by the window SW is an image area corresponding to the face image. Is the number of For example, when the face area FA shown in FIG. 15B is set, the four windows SW (SW1 to SW4) shown in FIG. Become. The maximum number of overlapping windows is the number of windows SW that overlap at least a part of the face area FA among all the windows SW arranged on the target image OI when the face area FA is detected. The maximum number of overlapping windows is uniquely determined by the movement pitch of the window SW and the size change pitch. Both the number of overlapping windows and the maximum number of overlapping windows can be calculated in the process of detecting the face area FA.

  If the detected face area FA is an image area that truly corresponds to a face image, the image area defined by the window SW corresponds to the face image for a plurality of windows SW whose positions and sizes approximate each other. There is a high possibility that the face area is determined to be a face area. On the other hand, when the detected face area FA is not an image area corresponding to a face image but a false detection, for a certain window SW, the image area defined by the window SW is a face area corresponding to the face image. Even if it is determined that there is another window SW whose position and size approximate that window SW, there is a possibility that the image area defined by the window SW is not determined to be a face area corresponding to the face image. high. Therefore, in this embodiment, a value obtained by dividing the number of overlapping windows by the maximum number of overlapping windows is used as the face area reliability. The face area FA detection process is thus completed.

  Here, the aspect of the filter X used for face determination, the threshold value thX, the weighting coefficient WeX, and face learning data FLD including a threshold value TH described later will be described. The face learning data FLD is set by learning using a sample image. FIG. 16 is an explanatory diagram illustrating an example of a sample image used for learning. For learning, a face sample image group composed of a plurality of face sample images that are known in advance to be images corresponding to face images, and a plurality of information that is known in advance to be images that do not correspond to face images. A non-face sample image group composed of non-face sample images. Since the setting of the face learning data FLD by learning is executed for each specific face inclination, as shown in FIG. 16, a face sample image group corresponding to each of 12 specific face inclinations is prepared. The face sample image group corresponding to each specific face inclination includes a plurality of basic face samples in which the ratio of the face image size to the image size is within a predetermined value range and the face image inclination is equal to the specific face inclination. An image and an image obtained by enlarging and reducing the basic face sample image at a predetermined magnification ranging from 1.2 times to 0.8 times (for example, images FIa and FIb in FIG. 16), and the basic face sample image clockwise And images rotated counterclockwise by a predetermined angle within a range of, for example, 15 degrees (for example, images FIc and FId in FIG. 16). Learning using a sample image is executed by, for example, a method using a neural network, a method using boosting (for example, adaboost), a method using a support vector machine, or the like. For example, when learning is performed by a method using a neural network, each filter X (filter 1 to filter N) is included in a face sample image group and a non-face sample image group corresponding to a specific face inclination. A basic evaluation value vX (v1 to vN) is calculated using all the sample images, and a threshold thX (th1 to thN) for achieving a predetermined face detection rate is set. Also, initial values of the weighting factors WeX (We1 to WeN) set for each filter X are set, and an evaluation value Tv for one sample image selected from the face sample image group and the non-face sample image group is set. Calculated. In learning, the value of the weighting coefficient WeX set for each filter X is corrected based on the correctness / incorrectness of the result when face determination described later is performed based on the calculated evaluation value Tv. By executing the above process for each specific face inclination, face learning data FLD for each specific face inclination is set.

  After the face area FA detection process, the initial position setting unit 230 (FIG. 1) sets the initial position of the feature point CP with respect to the target image OI (step S230). FIG. 17 is a flowchart showing the flow of the initial position setting process of the feature point CP in the first embodiment. First, the acquisition unit 232 of the initial position setting unit 230 acquires face area detection information (step S400). The face area detection information refers to information related to the detection of the face area FA by the face area detection unit 210. For example, the number of preset specific face inclinations (12), the setting angle (0 degrees, 30 degrees, 60 degrees,..., 330 degrees), the setting interval (30 degrees), the face area FA In addition to the tendency of the position of the face image in the face area FA based on the statistical data of the detection process, it is specified when the face area detection unit 210 detects the face area FA from the target image OI in the face area FA detection process. Information is also included. Specifically, the specific face inclination of the detected face area FA, the face area reliability, and the like. In the present embodiment, an example will be described in which the specific face tilt setting interval, the specific face tilt of the detected face area FA, and the face area reliability are used as the face area detection information. When the detection process of the face area FA is executed, the acquisition unit 232 acquires the specific face inclination and the face area reliability for the detected face area FA.

  The initial position candidate setting unit 234 sets the feature point CP as a temporary setting position on the target image OI based on the face area detection information (step S410). Here, the specific face tilt setting interval is 30 degrees, the specific face tilt of the detected face area FA is 0 degrees, and the face area reliability is more than a predetermined value, an example will be specifically described. The initial position candidate setting unit 234 pays attention to the feature point CP by changing various values of global parameters representing the size, inclination, and position (vertical position and horizontal position) of the face image with respect to the face area FA. Set on image OI. The temporarily set position corresponds to an “initial position candidate” in the claims.

FIG. 18 is an explanatory view exemplifying a temporary setting position of the feature point CP by changing the value of the global parameter. 18A and 18B show a mesh formed by connecting feature points CP and feature points CP in the target image OI. The initial position candidate setting unit 234 temporarily sets the feature point CP such that the average shape s ₀ is formed at the center of the face area FA, as shown in the center of FIGS. 18 (a) and 18 (b). (Hereinafter also referred to as “reference temporary setting position”).

  The initial position candidate setting unit 234 also sets a plurality of temporary setting positions obtained by changing various global parameter values with respect to the reference temporary setting position. Changing the global parameters (size, inclination, vertical position and horizontal position) means that the mesh formed by the feature point CP in the target image OI is enlarged / reduced, the inclination is changed, and the parallel movement is performed. Equivalent to. Accordingly, as shown in FIG. 18A, the initial position candidate setting unit 234 forms a temporary setting position (reference temporary setting position diagram) that forms a mesh obtained by enlarging or reducing the reference temporary setting position mesh at a predetermined magnification. And a temporary setting position (shown on the right and left in the drawing of the reference temporary setting position) that forms a mesh whose inclination is changed clockwise or counterclockwise by a predetermined angle. In addition, the initial position candidate setting unit 234 forms a temporary setting position (a figure of the reference temporary setting position) that forms a mesh obtained by performing a combination of enlargement / reduction and inclination change on the reference temporary setting position mesh. (Shown in the upper left, lower left, upper right, and lower right).

  Further, as shown in FIG. 18B, the initial position candidate setting unit 234 forms a temporary setting position (reference temporary setting) that forms a mesh that is translated upward or downward by a predetermined amount from the mesh at the reference temporary setting position. A temporary setting position (shown on the left and right in the drawing of the reference temporary setting position) that forms a mesh moved in parallel to the left or right. In addition, the initial position candidate setting unit 234 forms a temporary setting position (a reference temporary setting position in the diagram of the reference temporary setting position) that forms a mesh obtained by performing a combination of vertical and left and right parallel movements on the mesh of the reference temporary setting position. Also set (upper left, lower left, upper right, lower right).

  Here, when the specific face inclination of the detected face area FA is 0 degree and the setting interval of the specific face inclination is 30 degrees, the rotation angle of the mesh is set in a range from −15 degrees to 15 degrees. That is, the initial position candidate setting unit 234 determines the specific face inclination (0 degree) of the detected face area FA among the specific face inclination setting angles (0 degrees, 30 degrees, 60 degrees,..., 330 degrees). A range from one intermediate value (-15 degrees) to the adjacent setting angle (-30 degrees and 30 degrees) to the other intermediate value (15 degrees) is set as a range of mesh rotation angles.

  FIG. 19 is an explanatory diagram illustrating the temporarily set position of the feature point CP when the specific face inclination of the face area FA is 30 degrees. As shown in FIG. 19, when the specific face inclination of the detected face area FA is 30 degrees and the specific face inclination setting interval is 30 degrees, the mesh rotation angle is set in the range of 15 degrees to 45 degrees. The That is, the initial position candidate setting unit 234 determines the specific face inclination (30 degrees) of the detected face area FA from the specific face inclination setting angles (0 degrees, 30 degrees, 60 degrees,..., 330 degrees). A range from one intermediate value (15 degrees) to the adjacent setting angle (0 degrees and 60 degrees) to the other intermediate value (45 degrees) is set as a mesh rotation angle range. In other words, the initial position candidate setting unit 234 sets each of the detected specific inclination angles of the face area FA with respect to the provisional setting position when the specific inclination is 0 degree.

  The initial position candidate setting unit 234 temporarily performs vertical and horizontal parallel movements shown in FIG. 18B on the mesh at each of the eight temporary setting positions other than the reference temporary setting positions shown in FIG. The setting position is also set. Therefore, in the present embodiment, the four global parameters (size, inclination, vertical position, horizontal position) are set in 80 ways (= 3 × 3 × 3) that are set as known three-level values. A total of 81 temporary setting positions of (3-1) temporary setting positions and reference temporary setting positions are set.

  The initial position candidate setting unit 234 sets 81 temporarily set positions as described above when the face area reliability is greater than or equal to the threshold, but when the face area reliability is smaller than the threshold, the face area reliability is the threshold. More temporary setting positions are set than in the above case. Specifically, in the above description, the temporary setting position is set by changing the rotation and enlargement / reduction of the mesh, the vertical movement and the horizontal movement of the mesh in three stages, but the initial position candidate setting unit 234 Based on the comparison result between the face area reliability and the threshold value, the number of stages of change (rotation, enlargement / reduction, vertical movement, horizontal movement) of each temporary setting position to be set is determined. FIG. 20 is an explanatory diagram for explaining the number of stages of change of the temporarily set position. When the face area reliability is greater than or equal to the threshold value, the initial position candidate setting unit 234 changes in three stages for mesh rotation and enlargement / reduction, mesh vertical movement, and horizontal movement as shown in FIG. To set the temporary setting position. On the other hand, when the face area reliability is smaller than the threshold value, the initial position candidate setting unit 234 sets 5 for each of mesh rotation and enlargement / reduction, mesh vertical movement, and horizontal movement, as shown in FIG. Change the stage to set the temporary setting position.

  The range for setting the mesh may be the same range or a different range in the case of three stages and in the case of five stages. That is, as shown in FIG. 20, even in the case of 5 steps, the same range (−15 to 15 degrees) as in the case of 3 steps shown in FIG. 18, and the rotation angle of the mesh is −15 degrees, 0 Degrees, 15 degrees, and 3 levels may be subdivided into 5 levels of -15 degrees, -7.5 degrees, 0 degrees, 7.5 degrees, and 15 degrees. The rotation angle may be set to a wide range as compared with the case of three stages, with five stages of −30 degrees, −15 degrees, 0 degrees, 15 degrees, and 30 degrees. Further, it is not necessary to set all of the mesh rotation, enlargement / reduction, mesh up / down movement, and left / right movement in five stages, some of which may be set in five stages, and others may be set in three stages.

The generation unit 236 generates an average shape image I (W (x; p)) corresponding to each set temporary setting position (step S420). FIG. 21 is an explanatory diagram showing an example of the average shape image I (W (x; p)). The average shape image I (W (x; p)) is calculated by conversion such that the arrangement of the feature points CP in the input image is equal to the arrangement of the feature points CP in the average shape s ₀ .

The transformation for calculating the average shape image I (W (x; p)) is a warp W that is a set of affine transformations for each triangular area TA in the AAM setting process, similarly to the transformation for calculating the sample image SIw. Is done. Specifically, an average shape that is an area surrounded by a straight line connecting feature points CP (feature points CP corresponding to face lines, eyebrows, and eyebrows) located on the outer periphery by feature points CP arranged in the target image OI The area BSA is specified, and the average shape image I (W (x; p)) is calculated by performing affine transformation for each triangular area TA on the average shape area BSA in the target image OI. In this embodiment, the average shape image I (W (x; p) ) is composed of an average face image A ₀ (x) and an average shape area BSA and a mask area MA, average face image A ₀ (x) And is calculated as an image of the same size.

Here, a set of pixels located in the average shape area BSA in the average shape s ₀ is represented as a pixel group x. In addition, the pixel group in the image before the warp W execution (the average shape area BSA of the target image OI) corresponding to the pixel group x in the image after the warp W execution (face image having the average shape s ₀ ) is represented by W (x; p ). Since the average shape image is an image composed of luminance values in each of the pixel groups W (x; p) in the average shape region BSA of the target image OI, it is represented as I (W (x; p)). FIG. 21 shows nine average shape images I (W (x; p)) corresponding to the nine temporarily set positions shown in FIG.

The calculation unit 238 calculates a difference image Ie between the average shape image I (W (x; p)) corresponding to each temporarily set position and the average face image A ₀ (x) (step S430). The difference image Ie is a difference between pixel values of the average shape image I (W (x; p)) and the average face image A ₀ (x), and is also referred to as a difference value in this embodiment. Since the difference image Ie does not appear when the setting position of the feature point CP matches the position of the feature part, the difference image Ie represents the difference between the setting position of the feature point CP and the position of the feature part. In the present embodiment, since 81 types of temporarily set positions of feature points CP are set, the calculation unit 238 calculates 81 difference images Ie.

  The initial position setting unit 230 calculates a norm from the pixel values of each difference image Ie, and pays attention to a temporary installation position corresponding to the difference image Ie having the smallest norm value (hereinafter also referred to as “norm minimum temporary setting position”). The initial position of the feature point CP in the image OI is set (step S440). In this embodiment, the pixel value for calculating the norm may be a luminance value or an RGB value. Thus, the feature point CP initial position setting process is completed.

  When the feature point CP initial position setting process is completed, the feature position detection unit 220 corrects the setting position of the feature point CP in the target image OI (step S240). FIG. 22 is a flowchart showing the flow of the feature point CP setting position correction process in the first embodiment.

  The feature position detection unit 220 calculates the average shape image I (W (x; p)) from the attention image OI (step S510). The calculation method of the average shape image I (W (x; p)) is the same as that in step S420 in the feature point CP initial position setting process.

The feature position detection unit 220 calculates a difference image Ie between the average shape image I (W (x; p)) and the average face image A ₀ (x) (step S520). The feature position detection unit 220 determines whether or not the feature point CP setting position correction process has converged based on the difference image Ie (step S530). The feature position detection unit 220 calculates the norm of the difference image Ie, determines that it has converged if the norm value is smaller than a preset threshold value, and still converges if the norm value is greater than or equal to the threshold value. Judge that there is no.

  The feature position detection unit 220 determines that the calculated norm value of the difference image Ie has converged when the value is smaller than the value calculated in the previous step S520, and still determines that the value is equal to or greater than the previous value. It is good also as what determines with not having converged. Alternatively, the feature position detection unit 220 may perform the convergence determination by combining the determination based on the threshold and the determination based on comparison with the previous value. For example, the feature position detection unit 220 determines that the calculated norm value has converged only when the calculated norm value is smaller than the threshold value and smaller than the previous value, and otherwise determines that it has not yet converged. It may be a thing.

When it is determined in the convergence determination in step S530 that the convergence has not yet occurred, the correction unit 222 calculates the parameter update amount ΔP (step S540). The parameter update amount ΔP includes four global parameters (total size, inclination, X direction position, Y direction position), and n shape parameters p _i calculated by the AAM setting process (formula (1)) This means the amount of change in the value of (see). Note that immediately after the feature point CP is set to the initial position, the value determined in the feature point CP initial position setting process is set as the global parameter. Further, difference between the initial position of the characteristic point CP and sets the position of the characteristic points CP of the average shape s ₀ in this case, the overall size, the tilt, because it is limited to the difference in position, shape parameters in the shape model p The values of _i are all zero.

  The parameter update amount ΔP is calculated by the following equation (3). That is, the parameter update amount ΔP is a product of the update matrix R and the difference image Ie.

The update matrix R in the expression (3) is a matrix of M rows and N columns set in advance by learning in order to calculate the parameter update amount ΔP based on the difference image Ie, and the internal memory 120 as the AAM information AMI (FIG. 1). Stored in In this embodiment, the number of rows M in the update matrix R is equal to the sum ((4 + n)) of the number of global parameters (4) and the number of shape parameters p _i (n), and the number of columns N is , average face image a ₀ average number of pixels in the shape area BSA of (x) - is equal to (56 pixels × 56 pixels the number of pixels in the mask area MA). The update matrix R is calculated by the following formulas (4) and (5).

The correction unit 222 updates the parameters (four global parameters and n shape parameters p _i ) based on the calculated parameter update amount ΔP (step S550). Thereby, the setting position of the feature point CP in the target image OI is updated. The correction unit 222 updates the difference image Ie so that the norm of the difference image Ie becomes small. After the parameter update, the average shape image I (W (x; p)) is calculated again from the target image OI in which the installation position of the feature point CP is corrected (step S510), and the difference image Ie is calculated (step S520), convergence determination based on the difference image Ie (step S530) is performed. If it is determined that the convergence has not occurred in the convergence determination again, the parameter update amount ΔP based on the difference image Ie is calculated (step S540), and the setting position correction of the feature point CP by the parameter update (step S550). ) Is performed.

  When the processing from step S510 to step S550 in FIG. 22 is repeatedly executed, the position of the feature point CP corresponding to each feature part in the target image OI approaches the position of the actual feature part as a whole, and converges at a certain time. In the determination (step S530), it is determined that convergence has occurred. If it is determined in the convergence determination that convergence has been achieved, the face feature position detection process is completed (step S560). The setting position of the feature point CP specified by the values of the global parameter and the shape parameter set at this time is specified as the setting position of the feature point CP in the final target image OI. By repeating the processing from step S510 to S550, the position of the feature point CP corresponding to each feature part in the target image OI may coincide with the position of the actual feature part.

  FIG. 23 is an explanatory diagram illustrating an example of a result of the face feature position detection process. FIG. 23 shows the setting positions of the feature points CP finally specified in the target image OI. Since the position of the feature point CP specifies the position of the facial feature part (predetermined position in the facial organs (eyebrows, eyes, nose, mouth) and facial contour) included in the attention image OI. It is possible to detect the shape and position of the human face organ and the face contour shape in the image OI. Thus, the face feature position detection process is completed.

  The print processing unit 320 generates print data for the attention image OI in which the shape / position of the facial organ and the face contour shape are detected. Specifically, the print processing unit 320 performs color conversion processing for matching the pixel value of each pixel with the ink used by the printer 100 for the target image OI, and the gradation of the pixel after color conversion processing according to the distribution of dots. Print data is generated by performing halftone processing for representing, rasterizing processing for rearranging the data arrangement of the image data subjected to the halftone processing in an order to be transferred to the printer 100, and the like. Based on the print data generated by the print processing unit 320, the printing mechanism 160 prints the attention image OI in which the shape and position of the facial organ and the contour shape of the face are detected. Note that the print processing unit 320 is not limited to print data for the target image OI, and predetermined processing such as face deformation and face shading correction based on the detected shape and position of the facial organ and the face contour shape. It is also possible to generate print data of images subjected to. The printing mechanism 160 can also print an image that has undergone processing such as face deformation and face shading correction based on the print data generated by the print processing unit 320.

  As described above, according to the image processing apparatus of the first embodiment, the initial position of the feature point CP is set from a plurality of initial position candidates set based on the face area detection information. The position can be set closer to the characteristic part. Thereby, it is possible to increase the efficiency and speed of the process of detecting the position of the facial feature part included in the target image. That is, the position and range of the face image in the face area FA are not always constant, and the center of the face area FA does not always coincide with the center of the face image. In addition to the orientation and clarity of the face image included in the target image OI, The position and range of the face image vary depending on the detection characteristics of the face area detection unit 210 and the like. Therefore, when the feature point CP is set to a temporary setting position on the target image OI that is a candidate for the initial position, information related to the detection of the face area FA by the face area detection unit 2110 is used, so that the feature point CP is determined. It can be set at a more appropriate position with respect to the characteristic part to be detected.

  Specifically, the initial position candidate setting unit 234 has a low possibility that a face image is included in the face area FA when the face area reliability is low, that is, when the face area reliability is smaller than a threshold value. Therefore, the possibility that the feature point CP can be arranged at an appropriate position corresponding to the feature part is increased by increasing the number of the temporarily set positions of the feature point CP, compared with the case where the face area reliability is high. it can. On the other hand, when the face area reliability is high, there is a high possibility that a face image is included in the face area FA. Therefore, the number of temporarily set positions of feature points CP is higher than when the face area reliability is low. Therefore, the position of the facial feature portion can be detected efficiently and at high speed.

  Further, the initial position candidate setting unit 234 is set so as to be inclined by the angle of the specific inclination of the detected face area FA with respect to the provisional setting position when the specific inclination is set to 0 degree. The feature point CP can be arranged at a more appropriate position corresponding to the part. Therefore, the position of the facial feature part included in the target image can be detected efficiently and at high speed.

  In addition, the initial position candidate setting unit 234 sets the specific face inclination (for example, the detected face area FA) (for example, 0 degrees, 30 degrees, 60 degrees,..., 330 degrees) of the specific face inclination setting angles (for example, 0 degrees, 30 degrees, 60 degrees,. Rotate the mesh within a range from one intermediate value (for example, -15 degrees) between the adjacent setting angle (for example, -30 degrees and 30 degrees) to the other intermediate value (for example, 15 degrees). Processing to set a temporary setting position by rotating the mesh to an angle range where it is unlikely that the face inclination of the actual face image is included as a result of the detection processing of the face area FA to set as an angle range The temporary installation position can be set by rotating the mesh within a range where the face inclination of the actual face image is likely to be included. Therefore, the position of the facial feature part included in the target image can be detected efficiently and at high speed.

  According to the image processing apparatus according to the first embodiment, in the feature point CP initial position setting process, the initial position of the feature point CP is set using a global parameter. It is possible to increase the efficiency and speed of the process for detecting the. Specifically, multiple global parameters (size, inclination, vertical position, horizontal position) are changed to prepare multiple temporary setting positions for feature points CP that form various meshes. The temporary setting position corresponding to the difference image Ie having the smallest norm value is set as the initial position. Thereby, the initial position of the feature point CP in the target image OI can be set closer to the position of the facial feature part. Accordingly, in the feature point CP setting position correction process, correction by the correction unit 222 is facilitated, so that the process of detecting the position of the facial feature part can be made more efficient and faster.

  According to the printer 100 according to the first embodiment, it is possible to print the attention image OI in which the shape / position of the facial organ and the contour shape of the face are detected. Thus, facial expression determination for detecting a facial image with a specific facial expression (for example, a face with a smile or eyes closed) and facial orientation determination for detecting a facial image with a specific orientation (for example, rightward or downward) are performed. After that, any image can be selected and printed based on the determination result. Further, based on the detected shape and position of the organ of the face and the contour shape of the face, it is possible to print an image subjected to predetermined processing such as face deformation and face shading correction. Thereby, it is possible to print a specific face image after performing face deformation, face shading correction, and the like.

B. Second embodiment:
In the first example, the acquisition unit 232 acquires face area detection information (step S400), and the initial position candidate setting unit 234 focuses on the feature point CP based on the face area detection information acquired by the acquisition unit 232. Although the temporary setting position on the image OI has been set (step S410), the initial position candidate setting unit 234 does not acquire the face area detection information every time the facial feature position detection processing is performed, but the initial position candidate setting unit 234 previously stores the face area detection information. The feature point CP may be set at the temporary setting position set based on the above.

  FIG. 24 is an explanatory diagram showing a second example of the temporary setting positions of the face area detection information and the feature points CP. FIG. 25 is an explanatory diagram showing a third example of the temporary setting positions of the face area detection information and the feature points CP. The initial position candidate setting unit 234 in the second embodiment uses a temporary setting position of the feature point CP set in advance based on the tendency of the position of the face image in the face area FA for setting the initial position of the feature point CP. Specifically, as shown in FIG. 24A, when the statistical data of the face area FA detection process determines that the face image is located below the face area FA. As shown in FIG. 24B, global parameters are set in advance so that the mesh is positioned below the face area FA. In addition, when the face image is located above the face area FA, or when the face image is biased to the left or right, a global parameter is set in advance to place a mesh closer to the face image. can do. The global parameter setting may be performed by the user or the image processing unit 200.

  In addition, as illustrated in FIG. 25A, the initial position candidate setting unit 234 determines that the face area FA is smaller than the face image by statistical data of the face area FA detection process. As shown in FIG. 25B, global parameters are set in advance so that the mesh becomes larger with respect to the face area FA. Similarly, when the face area FA is larger than the face image, or when the face image is biased vertically and horizontally with respect to the face area FA, and the face area FA is larger or smaller than the face image, the same applies in advance. By setting global parameters, the mesh can be placed closer to the face image.

  According to the image processing apparatus according to the second embodiment, the image processing unit 200 does not include the acquisition unit 232, and improves the efficiency and speed of the process of detecting the position of the facial feature part included in the target image. Can be achieved. Specifically, without acquiring the face area detection information every time the facial feature position detection process is performed by the acquisition unit 232, for example, by the face area FA detection process such as the tendency of the position of the face image in the face area FA. If the face area detection information is other than the specified information, the temporary setting position of the feature point CP can be set in advance based on the face area detection information. Thereby, the feature point CP can be set at a more appropriate position with respect to the feature part. Further, the initial position candidate setting unit 234 does not need to set the feature point CP as a temporary setting position on the attention image OI based on the face area detection information acquired by the acquisition unit 232, but based on the face area detection information. The feature point CP may be set at a preset temporary setting position.

C. Variation:
The present invention is not limited to the above-described examples and embodiments, and can be implemented in various modes without departing from the gist thereof. For example, the following modifications are possible.

C1. Modification 1:
In the first example, the initial position candidate setting unit 234 performs mesh rotation and enlargement / reduction, mesh vertical movement, and horizontal movement as shown in FIG. If the temporary setting position is set in three stages for each, and the face area reliability is smaller than the threshold, the mesh is rotated and enlarged / reduced, the mesh is moved up and down, and left and right as shown in FIG. Although it has been described that the temporary setting position is set by changing the movement to 5 stages, the above-described 3 stages and 5 stages are merely examples. For example, the number of stages is set to other stages such as 4 stages and 6 stages. May be.

C2. Modification 2:
In the first embodiment, as the face area detection information, the number of specific face inclinations, the setting angle, the setting interval, and the tendency of the position of the face image in the face area FA based on statistical data of the face area FA detection process, In the detection process of the face area FA, the information specified when the face area detection unit 2110 detects the face area FA from the target image OI is shown. The face area detection information is the face area detection unit 2110. The information is not limited to the above as long as it is information related to FA detection, and other information is also included. For example, the result of detection processing of the previous face area FA is also included. Also, the information related to the detection of the face area FA by the face area detection unit 2110 is not limited to the specific face inclination and the face area reliability of the detected face area FA, but includes information other than these. For example, the amount of movement of the window SW in the horizontal and vertical directions, the number of overlapping windows, and the like are also included.

C3. Modification 3:
In the first embodiment, as the face area detection information, the specific face inclination setting interval, the specific face inclination of the detected face area FA, and the face area reliability are used, but it is not always necessary to use all of them. Even when the temporary setting position of the feature point CP is set based on these parts, the feature point CP can be arranged at a more appropriate position corresponding to the feature part, and the feature of the face included in the attention image The position of the part can be detected efficiently and at high speed. Further, the first embodiment and the second embodiment can be realized by appropriately combining them.

C4. Modification 4:
In this embodiment, the number of temporary setting positions set is changed using the face area reliability. However, the number of temporary setting positions may be determined based on face area detection information other than the face area reliability. Good. For example, the set number of temporarily set positions may be determined based on the degree of dispersion of the normal distribution calculated using statistical data of the face area FA detection process. Specifically, if the statistical data of the face area FA detection process causes a large variation in the position and rotation of the face image with respect to the face area FA, the range of the face image in the face area FA, etc. It may be set so that the number of stages of change for enlargement / reduction, vertical movement of the mesh, and horizontal movement increases. For example, the number of temporarily set positions may be determined so that the interval between the temporarily set positions is within a predetermined range in a range where the deviation from the average μ is ± 3σ.

C5. Modification 5:
The specific face tilt setting number (12), setting angles (0 degrees, 30 degrees, 60 degrees,..., 330 degrees), and setting intervals (30 degrees) shown in the first embodiment are examples. Other set numbers, set angles, and set intervals may be used. The sample image used for setting the face learning data FLD and the sample image SI used for the AAM setting process may be partially overlapping images or may be different images.

C6. Modification 6:
The sample image SI in this embodiment is merely an example, and the number and type of images employed as the sample image SI can be arbitrarily set. In the present embodiment, the predetermined feature portion of the face indicated by the position of the feature point CP is merely an example, and a part of the feature portion set in the embodiment may be omitted or another portion may be used as the feature portion. May be adopted.

  In this embodiment, the texture model is set by principal component analysis with respect to the luminance value vector formed by the luminance values in each of the pixel groups x of the sample image SIw, but the luminance representing the texture (appearance) of the face image. The texture model may be set by principal component analysis for index values other than the values (for example, RGB values).

In this embodiment, the size of the average face image A ₀ (x) is not limited to 56 pixels × 56 pixels, and may be other sizes. Further, the average face image A ₀ (x) does not need to include the mask area MA (FIG. 8), and may be configured only by the average shape area BSA. Further, instead of the average face image A ₀ (x), another reference face image set based on the statistical analysis of the sample image SI may be used.

  In this embodiment, the shape model and the texture model are set using AAM. However, the shape model and the texture model using other modeling methods (for example, a method called Morphable Model or a method called Active Blob) are used. A texture model may be set.

  In this embodiment, the image stored in the memory card MC is set as the attention image OI. However, the attention image OI may be an image acquired via a network, for example. Also, the detection mode information may be acquired via a network.

  In this embodiment, image processing by the printer 100 as the image processing apparatus has been described. However, part or all of the processing is executed by another type of image processing apparatus such as a personal computer, a digital still camera, or a digital video camera. It is good also as what is done. The printer 100 is not limited to an ink jet printer, and may be another type of printer such as a laser printer or a sublimation printer.

  In this embodiment, a part of the configuration realized by hardware may be replaced with software, and conversely, a part of the configuration realized by software may be replaced by hardware.

  In addition, when part or all of the functions of the present invention are realized by software, the software (computer program) can be provided in a form stored in a computer-readable recording medium. In the present invention, the “computer-readable recording medium” is not limited to a portable recording medium such as a flexible disk or a CD-ROM, but an internal storage device in a computer such as various RAMs and ROMs, a hard disk, etc. It also includes an external storage device fixed to the computer.

100 ... Printer 110 ... CPU
DESCRIPTION OF SYMBOLS 120 ... Internal memory 140 ... Operation part 150 ... Display part 160 ... Printing mechanism 170 ... Card interface 172 ... Card slot 200 ... Image processing part 210 ... Face area detection part 220 ... Feature position detection part 222 ... Correction part 230 ... Initial position setting Unit 232 ... acquisition unit 234 ... initial position candidate setting unit 236 ... generation unit 238 ... calculation unit 310 ... display processing unit 320 ... print processing unit

Claims (12)

An image processing apparatus for detecting a coordinate position of a feature part of a face included in an attention image,
A face area detection unit that detects an image area including at least a part of a face image from the attention image as a face area;
A plurality of the initial positions set based on face area detection information, which is information related to the detection of the face area, is an initial position of a feature point set in the image of interest for detecting the coordinate position of the feature part An initial position setting unit set from position candidates;
A feature position detector configured to correct the set position of the feature point set to the initial position so as to approach the position of the feature part and detect the corrected set position as the coordinate position of the feature part; Image processing device. The image processing apparatus according to claim 1.
The face area detection information is information specified along with the detection by the face area detection unit,
The initial position setting unit includes:
An acquisition unit for acquiring the identified face area detection information;
An initial position candidate setting unit that sets the initial position candidate based on the acquired face area detection information. The image processing apparatus according to claim 2,
The face area detection information includes a face area reliability indicating the certainty that the face image included in the face area detected by the face area detection unit is a true face image,
The initial position candidate setting unit increases the number of candidates for the initial position to be set when the face area reliability is low compared to when the face area reliability is high. The image processing apparatus according to claim 2,
The face area detection information includes angle information related to a rotation angle in an image plane of a face image included in the face area detected by the face area detection unit,
The initial position candidate setting unit is an image processing apparatus configured to rotate and set the predetermined initial position candidates according to the rotation angle based on the angle information. The image processing apparatus according to claim 1.
The face area detection information includes information related to a rotation angle in the image plane of the face image that can be specified by the face area detection unit,
The candidates for the plurality of initial positions are adjacent to each other for each rotation angle that can be specified by the face area detection unit, based on an intermediate value between the rotation angle and one of the rotation angles that can be specified. The image processing apparatus respectively set to the range to the intermediate value with the other said identifiable rotation angle. The image processing apparatus according to claim 1.
The face area detection information includes information regarding a tendency of a relative position of a face image with respect to the face area detected by the face area detection unit,
The plurality of initial position candidates are image processing devices in which relative positions with respect to the face region are determined according to the tendency. The image processing apparatus according to any one of claims 1 to 6,
The initial position setting unit includes:
A generating unit that generates an average shape image that is an image obtained by converting a part of the image of interest based on the feature point set at a position that is a candidate for the initial position;
A calculation unit that calculates a difference value between the average shape image and an average face image that is an image generated based on a plurality of sample images including a face image in which the coordinate position of the feature part is known. ,
An image processing apparatus configured to set, as the initial position, an initial position candidate having a minimum difference value among the plurality of initial position candidates. The image processing apparatus according to any one of claims 1 to 7,
The feature position detector
Based on the difference value between the average shape image corresponding to the initial position and the average face image, a correction unit that corrects the set position so that the difference value is small,
An image processing apparatus that detects the set position where the difference value is predetermined as the coordinate position. The image processing apparatus according to any one of claims 1 to 8,
The image processing apparatus, wherein the characteristic part is a part of eyebrows, eyes, nose, mouth, and face line. A printer that detects a coordinate position of a feature part of a face included in an attention image,
A face area detection unit that detects an image area including at least a part of a face image from the attention image as a face area;
A plurality of the initial positions set based on face area detection information, which is information related to the detection of the face area, is an initial position of a feature point set in the image of interest for detecting the coordinate position of the feature part An initial position setting unit set from position candidates;
A feature position detector that corrects the setting position of the feature point set to the initial position so as to approach the position of the feature part, and detects the corrected setting position as a coordinate position of the feature part;
And a printing unit for printing the target image from which the coordinate position is detected. An image processing method for detecting a coordinate position of a feature part of a face included in an image of interest,
Detecting an image area including at least a part of a face image from the noted image as a face area;
A plurality of the initial positions set based on face area detection information, which is information related to the detection of the face area, is an initial position of a feature point set in the image of interest for detecting the coordinate position of the feature part A step of setting from position candidates;
Correcting the setting position of the feature point set to the initial position so as to approach the position of the feature part, and detecting the corrected setting position as the coordinate position of the feature part. . A computer program for image processing that detects a coordinate position of a feature part of a face included in an image of interest,
A face area detection function for detecting, as a face area, an image area including at least a part of a face image from the attention image;
A plurality of the initial positions set based on face area detection information, which is information related to the detection of the face area, is an initial position of a feature point set in the image of interest for detecting the coordinate position of the feature part Initial position setting function to set from position candidates,
A feature position detection function for correcting the set position of the feature point set to the initial position so as to approach the position of the feature part, and detecting the corrected set position as the coordinate position of the feature part; Computer program to be realized.

Images