JP2008003749A - Feature point detection device, method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To quickly precisely and stably detect a plurality of kinds of feature points of a prescribed object in a picture. <P>SOLUTION: With respect to the plurality of kinds of feature points of the prescribed object in a detection object picture, the plurality of kinds of feature points restricted by position relations between the plurality of kinds of feature points are temporarily determined by using a first feature point detector group which is generated by machine learning and has a large tolerance, and a first position relation model which is generated by statistical learning and prescribes the position relations and has a large tolerance. Then, the plurality of kinds of final feature points restricted by the position relations are determined by using a second feature point detector group which is generated by machine learning in vicinity of respective temporal feature points and has a small tolerance, and a second position relation model which is generated by statistical learning and has a small tolerance. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a feature point detection apparatus and method for detecting feature points of a predetermined object included in an image, and a program therefor.

  A method for detecting the position, posture, form, and the like of a predetermined object (object) included in an image has been studied in various fields. As one of the methods, a plurality of characteristic parts of the predetermined object are represented. Various methods have been proposed in which feature points are defined, a plurality of feature points are detected from the detection target image, and the position, posture, form, etc. of the predetermined object are detected based on the positional relationship of the detected feature points. ing.

  For example, in Patent Document 1, a plurality of reference feature points (for example, eyes, nose, face contour, etc. when the target object is a person's face) constituting the target object are defined and specific feature points are specified. Learn the response when applying the filter and learn the standard positional relationship between feature points, that is, the existence probability distribution on the image of the center point of the object for each feature point. And applying the same filter as used during learning to the input image to detect a plurality of feature point candidates from the response, and a standard between the detected candidate and the learned feature points Compared with the positional relationship, the presence probability distribution on the image of the center point of the face is added, and as a result, there is an object position detection method in which the point at the position with the highest existence probability is the center point of the target object. Proposed. The probability distribution here is approximated by a Gaussian function.

  Further, Non-Patent Document 1 proposes an object position detection method similar to the method of Patent Document 1. This method does not detect only one point, such as the center point of an object, but detects a plurality of feature points as a set. In addition, the “standard positional relationship between feature points” is determined. In order to make a decision according to actual data, the existence probability distribution is statistically generated from a large number of learning samples. Hereinafter, this method will be described more specifically.

(Learning step)
In this method, an existence probability distribution of points that are correct answers of other feature points with respect to the position of a feature point detected by a feature point detector (including a classifier generated by an AdaBoost learning algorithm) is Two sets of feature points and one other feature point are prepared for each set, and the positional relationship between the feature points is expressed using the plurality of sets of existence probability distributions. Here, the existence probability distribution of a point that is a correct answer of the feature point X _i (coordinate x _i ) with respect to the output coordinate x _j of the detector of the feature point X _j is defined as P _ij (x _i | x _j ). Note that P _ij is represented by a two-dimensional histogram in terms of implementation.

In order to obtain the existence probability distribution P _ij , first, a target object is detected for a training image set (thousands of images in which correct coordinates of feature points of the target object are input), and the target object is set at a reference position. Normalize the image to be located. FIG. 7 shows an example in which when a target is a human face, the face is detected from the image and the image is normalized so that the face is positioned at a predetermined size in the center of the image. is there.

Next, the feature point detector D _i detects the feature point X _i from the standardized image, and the difference between the coordinate x _i of the feature point and the correct coordinate x _j of the other feature point X _j is determined as one The feature points X _i and one other feature point X _j are compared and tabulated in units of two sets. An example of the existence probability distribution P _ij obtained by such learning is shown in FIG. FIG. 8 shows an example in which the object is a human face, the position of the feature point detected by the feature point detector is represented by x, and the existence probability distribution of the target feature point is represented by shading on the image. ing. Here, the higher the probability of existence of the target feature point, the higher the density at that position. FIG. 8A is a diagram showing the existence probability distribution of the left eye corner point with respect to the position of the left eye head point detected by the left eye head detector, and FIG. 8B is the position of the left eye head point. FIG. 8C is a diagram showing the existence probability distribution of the right mouth corner point with respect to the position of the left eye point.

(Detection step)
A target is detected from the detection target image, a standardized image including the target is obtained, a feature point candidate is detected for the standardized image, and another feature point is detected for each feature point. The sum of the existence probability distribution on the image of the feature point estimated from the feature point candidates is calculated, and the point at the position having the highest existence probability is estimated and selected as the true point of the feature point. The point estimated as the true point of the feature point is expressed by the following equation (1).

Here, x ^ _i (left side) is the position coordinate of the point estimated as the true point of the feature point, and P _ij (x _i | q _jt ) is the position coordinate q _jt of the t th candidate of the feature point X _j. Is the existence probability distribution of the feature point X _i (positional coordinate x _i ), where k is the number of candidate feature points X _j , and n is the number of defined feature points.

  In this method, when detecting the position of a certain feature point, rather than relying on the output of a single feature point detector that detects the feature point, a plurality of feature point detectors determine the position of another feature point from each other. By inferring each other, there is an advantage that a superior detection ability that exceeds the performance of the detector alone can be obtained.

On the other hand, Patent Document 2 is a method for detecting the position of the eyes constituting the face, and after detecting the approximate position of the eyes using a so-called coarse detector with relatively low detection accuracy and high robustness, In the vicinity of the approximate position, a two-stage eye detection method has been proposed in which an accurate eye position is detected using a so-called fine detector with relatively high detection accuracy and low robustness.
JP-A-6-348851 JP 2005-108197 A David Cristinacce “A Multi-Stage Approach to Facial Feature Detection”, In Proc. Of BMVC, Pages 231-240, 2004

  By the way, in the object position detection method of Non-Patent Document 1, the detection target image is usually normalized geometrically at the positions of a plurality of predetermined characteristic points of the target object, and the target is compared with the normalized image. Detect multiple types of feature points of objects. For example, when the object is a human face, the detection target image is geometrically normalized at the positions of both eyes of the face, and feature points such as the face of the face, the corners of the eyes, the nose, and the mouth corners are detected.

  However, in this method, the characteristic position that is used as a reference when performing geometric normalization, and taking the above face as an example, the characteristic point that is located far from the position of both eyes of the face is the position of that position. The variation is wide and the search range of the feature points is widened. Here, it is necessary to use a feature point detector with relatively high robustness in order to cover this wide search range while emphasizing speeding up of the search process. If such a feature point detector is used, Generally, the position accuracy of the detected feature point is deteriorated.

  Further, the eye detection method of Patent Document 2 is a method with high detection accuracy because it adopts a two-stage configuration, but if this method is applied as it is to a plurality of types of feature point detection, the positional relationship between the feature points. Therefore, if an image having a feature similar to the image representing the feature point is accidentally present at a location shifted from the position of the feature point that is the correct answer, a feature point candidate for the incorrect answer is detected. Therefore, stable feature point detection cannot be expected.

  The present invention has been made in view of the above circumstances, and a feature point detection apparatus and method capable of detecting a plurality of types of feature points of a predetermined object included in an image at high speed with high accuracy and stability, and therefore The purpose is to provide a program.

  The feature point detection apparatus of the present invention provides a first feature inspection having a first detection accuracy and a first robustness generated by machine learning for a plurality of types of feature point candidates of a predetermined object in a detection target image. First feature point candidate detection means for detecting at least one for each type using an output device group, positions of feature point candidates detected by the first feature point candidate detection means, and the plurality of types The plurality of types of provisional feature points having a positional relationship constrained by the first positional relationship model based on a first positional relationship model that defines the positional relationship between the feature points of the first positional relationship with a first tolerance. First feature point determination means to be determined, and feature point candidates generated by machine learning in the vicinity of each of the plurality of types of provisional feature points determined by the first feature point determination means. Higher than 1 detection accuracy Second feature point candidate detecting means for detecting at least one for each type using a second feature point detector group having second detection accuracy and second robustness lower than the first robustness. And the position of the feature point candidates detected by the second feature point candidate detecting means and the positional relationship between the plurality of types of feature points are defined with a second tolerance smaller than the first tolerance. And a second feature point determining means for determining the plurality of types of final feature points having a positional relationship constrained by the second positional relationship model based on the second positional relationship model. It is what.

In the feature point detection apparatus of the present invention, the aspect ratio is maintained, in which the plurality of types of temporary feature points determined by the first feature point determination unit are brought close to a predetermined reference position for the detection target image, respectively. Normalization means for performing geometric normalization processing may be further provided, and the second feature point candidate detection means may detect the candidate on the detection target image normalized by the normalization means. In the feature point detection apparatus of the present invention, the first feature point determination means is one feature point statistically obtained for each combination of two different feature points of the plurality of feature points. When using the existence probability distribution on the image of the other feature point when the position of the other is used as a reference, for one type of feature point, the position of the detected candidate of the other type of feature point is used as a reference The one of First existence probability distribution synthesis for obtaining the existence probability distribution of feature points of each type for each of the other feature point candidates, and performing the process of synthesizing the obtained existence probability distribution for each type of feature points Means, for one type of feature point, the position of the detected candidate of the one type of feature point, and the magnitude of the existence probability in the synthesized existence probability distribution of the one type of feature point Based on the first feature point candidate selection means for selecting and narrowing down the one type of feature point candidates from the detected candidates, and the feature point candidates For each type, there may be provided first feature point position determining means for determining the position of the temporary feature point of the type based on the position of the selected candidate of the type of feature point. .

  In the feature point detection apparatus according to the present invention, the first feature point candidate selection unit may, for each type of feature point, out of the detected candidates, the synthesized existence probability of the feature point of the type. You may select the candidate which exists in the predetermined area | region represented by the point of the position with the highest existence probability in distribution.

  Further, in the feature point detection apparatus of the present invention, the first feature point candidate selecting means, for each type of feature point, the feature of the type corresponding to the position of the candidate among the detected candidates. You may select the candidate whose existence probability in the said synthetic | combination existence probability distribution of the point is more than a predetermined threshold value.

  Further, in the feature point detection apparatus of the present invention, the first feature point candidate detection means is based on a threshold determination of certainty indicating the probability that the identification target image on the detection target image is an image including a feature point. , Detecting a target in the identification target image as a candidate for the feature point, and the first feature point position determining means, for each of the types of feature points, for all the selected feature points of the type The weighted average of the position coordinates when the candidate position coordinates are weighted with the certainty factor calculated for the candidate may be determined as the position coordinates of the temporary feature point of the type.

  In the feature point detection apparatus of the present invention, the first feature point position determining means may include, for each type of feature point, the position coordinates of all candidates selected for the feature point of the type. The weighted average of the position coordinates when weighted by the existence probability in the combined existence probability distribution of the type of feature points corresponding to the position is determined as the position coordinates of the type of temporary feature points. May be.

  The feature point detection method of the present invention provides a first feature check having first detection accuracy and first robustness generated by machine learning for a plurality of types of feature point candidates of a predetermined object in a detection target image. A first feature point candidate detection step for detecting at least one for each type using an output device group; positions of feature point candidates detected by the first feature point candidate detection step; and the plurality of types The plurality of types of provisional feature points having a positional relationship constrained by the first positional relationship model based on a first positional relationship model that defines the positional relationship between the feature points of the first positional relationship with a first tolerance. A first feature point determination step to be determined; and feature point candidates generated by machine learning in the vicinity of each of the plurality of types of provisional feature points determined by the first feature point determination step. A second feature point detector group having a second detection accuracy higher than the detection accuracy of the second and a second robustness lower than the first robustness, and detecting at least one for each type A feature point candidate detection step, a position of a feature point candidate detected by the second feature point candidate detection step, and a positional relationship between the plurality of types of feature points are smaller than the first tolerance. A second feature point determining step for determining the plurality of types of final feature points having a positional relationship constrained by the second positional relationship model based on a second positional relationship model defined by tolerance. It is characterized by having.

  A program according to the present invention is a first feature check having a first detection accuracy and a first robustness generated by machine learning, using a computer, a plurality of types of feature point candidates of a predetermined object in a detection target image. First feature point candidate detection means for detecting at least one for each type using an output device group, positions of feature point candidates detected by the first feature point candidate detection means, and the plurality of types The plurality of types of provisional feature points having a positional relationship constrained by the first positional relationship model based on a first positional relationship model that defines the positional relationship between the feature points of the first positional relationship with a first tolerance. First feature point determination means to be determined, and feature point candidates generated by machine learning in the vicinity of each of the plurality of types of provisional feature points determined by the first feature point determination means. 1's A second feature of detecting at least one for each type using a second feature point detector group having a second detection accuracy higher than the output accuracy and a second robustness lower than the first robustness. A second tolerance that is smaller than the first tolerance in the point candidate detection means, the position of the feature point candidate detected by the second feature point candidate detection means, and the positional relationship between the plurality of types of feature points; Based on a second positional relationship model defined in degrees, functioning as second feature point determining means for determining the plurality of types of final feature points having positional relationships constrained by the second positional relationship model It is characterized by this.

  In the present invention, “a first positional relationship model that defines a positional relationship between a plurality of types of feature points with a first tolerance” means that the positional relationship between feature points is not a positional relationship between points, but between regions This is a model that is defined as the positional relationship of the above, and the average width of this region is defined by the first tolerance. Similarly, “a second positional relationship model that defines a positional relationship between a plurality of types of feature points with a second tolerance smaller than the first tolerance” refers to a positional relationship between feature points between regions. The model is defined as a relationship, and the average width of this region is smaller than the width defined by the first tolerance.

  Note that these positional relationship models are obtained by applying a feature point detector group to a sample image including a predetermined object and the positions of a plurality of types of feature point candidates and a plurality of types of features in the sample image. It can be obtained statistically based on the correct position of the point.

  In the present invention, the feature point existence probability distribution means a distribution in which a probability that a feature point exists at a certain position on the image is expressed for a plurality of positions on the image.

  In the present invention, “synthesize the existence probability distribution” means to obtain a new existence probability distribution by synthesizing the existence probabilities at the positions corresponding to each other on the image. Here, the composition may be addition or integration.

  Further, in the present invention, the “predetermined region represented by the point with the highest probability” means a region having a predetermined size range with the point as the approximate center, approximate center of gravity, or approximate center. To do. This region may be a circle or a polygon such as a rectangle.

  In the present invention, examples of the predetermined object include a human face, a car, and an animal.

  Note that the feature point of the predetermined object detected by the method and apparatus of the present invention is not only the posture recognition of the predetermined object, but also an authentication process for determining whether or not the specific object is a specific object. When the face is a person, it can be used for facial expression discrimination or the like.

  According to the feature point detection apparatus and method and the program therefor according to the present invention, a plurality of types of feature point candidates of the predetermined object in the detection target image are first detected with the first detection accuracy and the first robustness. A feature point detector group is used for detection, and the detected feature point candidate positions and a first positional relationship model that defines the positional relationship among the plurality of types of feature points with a first tolerance. And determining the plurality of types of provisional feature points having the positional relationship constrained by the first positional relationship model, and then, in the vicinity of each of the determined types of the provisional feature points, candidate feature points Is detected using a second feature point detector group having a second detection accuracy higher than the first detection accuracy and a second robustness lower than the first robustness, and the detected feature points Candidate positions and the above types The plurality of types having positional relationships constrained by the second positional relationship model based on the second positional relationship model that defines the positional relationship between the scoring points with a second tolerance smaller than the first tolerance Therefore, it is possible to perform feature point detection at high speed and with high detection accuracy by adopting a two-stage configuration in which relatively fine detection is performed after relatively coarse detection. Incorrect feature point candidates can be eliminated by restricting the positional relationship between them, and a plurality of types of feature points of a predetermined object included in the image can be detected at high speed with high accuracy and stability.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of a face feature point detection system according to an embodiment of the present invention. This face feature point detection system is a system that detects a human face from an image, and further detects the detected face feature points. The processing program read into the auxiliary storage device is stored in a computer (for example, a personal computer). This is realized by executing on a computer or the like. Further, this processing program is stored in an information storage medium such as a CD-ROM, or distributed via a network such as the Internet and installed in a computer. The image data represents an image, and the following description will be given without particularly distinguishing the image from the image data.

  As shown in FIG. 1, the face feature point detection system according to the present embodiment includes an image input unit 10, a face detection unit 20, a first face normalization unit 30, and a first feature point candidate detection unit 40. The first feature point determination unit 50, the second face normalization unit 60, the second feature point candidate detection unit 70, and the second feature point determination unit 80 are provided.

  The image input unit 10 receives an input of an image S0 input from the user and is a face detection target, and stores the input image S0 in a memory (not shown). For example, an image of a snap photograph acquired by a digital camera or the like is input.

  The face detection unit 20 reads the input image S0 stored in the memory and detects a face included in the image. Specifically, the face detection unit 20 detects the positions of both eyes of the face S1 included in the input image S0. In addition, the orientation f of the face is also detected.

  Here, as a method for detecting the positions of both eyes of the face S1, the object identification method disclosed in Japanese Patent Laid-Open No. 2005-108197 is applied. In this method, the feature amount is calculated for the image of the identification target region in the input image S0, and the feature amounts of the plurality of face sample images and the plurality of non-face sample images obtained by normalizing the eye positions with a predetermined tolerance. A case where a face is included by identifying whether or not a face is included in the image of the identification target region based on the feature amount calculated from the image of the identification target region with reference to the first reference data that has been learned Further, referring to the second reference data obtained by learning the feature amounts of the plurality of face sample images and the plurality of non-face sample images obtained by normalizing the eye positions with a tolerance smaller than a predetermined tolerance. The method of identifying the position of the eyes contained in the face, using a discriminator learned by a so-called machine learning method called AdaBoost to detect the face and accurately detecting the position of the eyes contained in the face Is

  Since this discriminator discriminates faces having substantially the same orientation as the face sample image used for learning, it is obtained by learning using a plurality of types of face sample images prepared for each face orientation. If a plurality of types of classifiers are prepared and a face is detected using the plurality of types of classifiers, a face S1 in a multi-directional direction is detected, and the positions of both eyes of the face and the direction of the face are detected. f can be known at the same time.

  As a method for detecting the face S1, in addition to the above, a template matching method or the like may be used. In this case, the positions of both eyes of the detected face S1 may be obtained by previously defining positions corresponding to the positions of both eyes of the face on a plurality of templates having different face orientations.

  The first face normalization unit 30 uses the input image S0 (Gray image) and information on the positions of both eyes of the detected face S1 to position both eyes at a predetermined reference position from the input image S0. Thus, a normalized face image S1 ′ is obtained by cutting out an image of a predetermined size including the normalized face S1. FIG. 7 is a diagram illustrating a state in which the normalized face image S1 ′ is obtained by performing appropriate trimming after the input image S0 is enlarged / reduced or rotated as necessary. In this embodiment, the image size is 200 × 200 pixels, and the center positions of both eyes of the face S1 are respectively the upper left pixel of the image with coordinates (0, 0) and the lower right pixel with coordinates (199, 199). ), The face is normalized so as to be located at A (70, 70) and B (130, 70). When the face direction f is oblique, for example, 45 degrees, the center positions of both eyes of the face S1 are A (70, 50) and B (130, 50), respectively. The face position is shifted upward by 20 pixels and normalized.

The first feature point candidate detection unit 40 has a detection score (certainty factor) indicating a probability that a feature point is included in the approximate center of the image of the identification target region designated on the normalized face image S1 ′. By calculating the SCD and identifying the object represented by the image of the identification target area as the feature point when the detection score SCD is equal to or greater than a predetermined threshold, the feature is determined for each feature point X _i of the face. The point candidate Q1 _it is detected.

  FIG. 2 is a diagram illustrating a configuration of the first feature point candidate detection unit 40. As illustrated, the first feature point candidate detection unit 40 includes a multi-resolution image generation unit 41, an illumination normalization unit 42, a first feature point search range setting unit 43, and a first feature point detector. A first detection processing unit 44 including a group 45 and a first feature point detector selection unit 46 are provided.

The first feature point detector group 45 includes a plurality of types of feature point detectors D1 _i (i = 1, 2,..., N _D ) prepared for each type of feature point X _i to be detected. Each feature point detector D1 _i uses a plurality of discriminators learned by a machine learning method called AdaBoost, which is disclosed in Japanese Patent Laid-Open No. 2005-108197, etc., to accurately determine the position of the feature point. It is something that is often detected. That is, this first feature point detector group represents the feature point detector group having the first detection accuracy and the first robustness in the present invention.

  This discriminator learns the feature quantities of a plurality of feature point sample images and a plurality of non-feature point sample images normalized with a predetermined tolerance so that the position of a specific feature point is substantially centered. Based on the feature amount calculated from the image of the identification target region with reference to the reference data, the identification point indicating the probability that the image includes the feature point is calculated, and by setting an appropriate threshold value, This threshold value determination of the identification point makes it possible to identify whether or not the image to be identified includes a specific feature point at the approximate center.

  Here, the creation of a discriminator for a specific feature point will be briefly described with reference to FIG. First, with respect to all feature point sample images including a specific feature point substantially at the center, a plurality of types of feature amounts are calculated to obtain a combination of the feature amounts, and a histogram is created. Similarly, for all non-feature point sample images, the same combination of a plurality of types of feature amounts is obtained, and a histogram thereof is created. The histogram used as a discriminator shown on the rightmost side of FIG. 13 is a new histogram obtained by taking the logarithmic value of the ratio of the frequency values indicated by these two histograms. The value of each vertical axis (bin) indicated by the histogram of this discriminator is hereinafter referred to as an identification point. According to this discriminator, an image showing the distribution of the feature amount corresponding to the positive discrimination point is likely to be an image including the specific feature point at the center, and the possibility is higher as the absolute value of the discrimination point is larger. Can be said to increase. Conversely, an image showing the distribution of feature amounts corresponding to negative identification points is likely not to be a specific feature point, and the possibility increases as the absolute value of the identification point increases.

  A plurality of such discriminators are prepared and prepared for each combination of feature amounts, and it is effective for identifying whether or not the image includes the specific feature point at the approximate center among the created discriminators. The correct discriminator is selected by a predetermined algorithm. As the predetermined algorithm, for example, the following can be considered. First, each sample image of feature points and non-feature points is assigned a weight of 1 in advance, and each classifier is made to identify whether each sample image is an image centered on a specific feature point. The weighted correct answer rate of the discriminator is obtained, and the weighted correct answer rate of each discriminator is compared, and the discriminator having the relatively highest weighted correct answer rate is selected. Then, the weight of the sample image with a poor correct answer rate is set to be larger than the current time, and each classifier is again identified with each sample image to obtain the weighted correct answer rate of each classifier. Select the highest classifier. By repeating such processing, classifiers effective for identification are sequentially selected. For details of this algorithm, refer to JP-A-2005-108197.

The feature point detector D1 _i designates an identification target region on a normalized resolution image S1 ″ _k described later, and selects the selected valid discriminator for the image of the designated identification target region. Each of the identification points is calculated and the sum of all the calculated identification points is output as a detection score SCD, and it is determined whether or not the detection score SCD is equal to or greater than a predetermined threshold. When the threshold value is equal to or greater than the threshold value, it is identified that the specific feature point is included in the center of the image of the identification target region, that is, the center position is determined as the specific feature point candidate Q1 _it (t = 1,. _Detected as the position of _NQ1i ).

  As the feature point sample images used for learning of the classifiers, for example, tens of thousands of samples are prepared by adding variations of scaling, rotation, and aspect conversion based on thousands of different feature point images. Further, the resolution of each patch is 24 × 24 pixels, and learning is performed by the AdaBoost learning algorithm using the output value of the Haar-like filter as a feature amount.

This discriminator identifies feature points of the same type as the feature points of the image including the feature points used for learning, so learning is performed using a plurality of types of feature point sample images with different types of feature points. If a plurality of types of classifiers are prepared and feature points are detected using the plurality of types of classifiers, the position q1 _it of the feature point candidate Q1 _it is determined for each type of feature point X _i. Can be detected.

In this embodiment, when the direction f of the detected face S1 is closer to the front, as the feature points X _i of the face, the left eye area (X _1), the left eye head (X _2), the right eye head (X _3), the right eye area _(X 4), the left nostril _(X 5), the right nostril _(X 6), left corner of the mouth _(X 7), right corner of the mouth _(X 8), the midpoint of the upper lip _(X 9), the lower lip midpoint and the use of 10 kinds of (X _10), whereas, if the direction f of the detected face S1 is a face close to the diagonal 45 degrees, as each feature point xi face, left eye area (X ₁ ), 7 types of left eye (X ₂ ), right eye (X ₃ ), right eye corner (X ₄ ), nose tip (X ₁₁ ), left mouth corner (X ₇ ), right mouth corner (X ₈ ) are used. To do. Therefore, the feature point detector, the left eye area, eye head, right head, right eye area, left nostril, right nostril, the left corner of the mouth, right corner of the mouth, the midpoint of the upper lip, the midpoint of the lower lip, 11 types of nose (N _D = 10) is prepared.

FIG. 10 is a diagram illustrating an example of a feature point sample image used for learning of a discriminator included in each feature point detector. For the front face, the left eye corner (X ₁ ), the left eye head (X ₂ ), and the right eye head. (X ₃ ), right eye corner (X ₄ ), left nose (X ₅ ), right nose (X ₆ ), left mouth corner (X ₇ ), right mouth corner (X ₈ ), middle point of upper lip (X ₉ ), bottom 10 types of the midpoint (X ₁₀ ) of the lips are shown.

As a method for detecting the feature point X _i , a template matching method or the like may be used in addition to the above. In this case, the position of the feature point may be obtained by previously defining a position corresponding to the position of the feature point on a plurality of templates having different types of feature points.

As shown in FIG. 9, the multi-resolution image generation unit 41 uses a normalized face image S1 ′ that is an image having a size of 200 × 200 pixels as a reference, and a reduced image (hereinafter referred to as resolution) in increments of −1/5. S1 ′ _k (k = 1, 2,..., N _S1 ) is generated. The reason why the normalized face image S1 ′ is multi-resolution in this way is that the feature point candidates are detected for a plurality of generated images having different resolutions, thereby improving the accuracy of the eye position during face normalization. Even if the size of the facial parts that make up the face varies depending on the quality of the face, the orientation of the face, and individual differences, the feature point candidates can be created with multiple images that gradually change the size of the facial parts. This is because it is possible to absorb the variation of the facial parts and to stably detect the feature point candidates.

The illumination normalization unit 42 performs normalization processing on each of the resolution images so that the contrast of the resolution image is suitable for detection of feature point candidates, and generates the normalized resolution image S1 ″ _k . To get.

  In this normalization process, in order to bring the contrast of the resolution image close to a predetermined level suitable for detection of feature point candidates, that is, a level suitable for extracting the performance of the feature point detector, the pixel value of the entire resolution image is set. In this image, conversion is performed according to a conversion curve that approaches a value representing the logarithm of the luminance of the subject.

  FIG. 14 is a diagram illustrating an example of a conversion curve used for the illumination normalization process. As the illumination normalization processing, as shown in the figure, an image is obtained in accordance with a conversion curve (look-up table) that further takes a logarithm after performing so-called inverse gamma conversion (= 2.2) in the sRGB space. A process for converting the pixel values in the whole can be considered. This is due to the following reason.

  The light intensity I observed as an image is usually expressed as the product of the reflectance R of the subject and the intensity L of the light source (I = R × L). Therefore, when the intensity L of the light source changes, the light intensity I observed as an image also changes. However, if only the reflectance R of the subject can be evaluated, it does not depend on the intensity L of the light source. The feature point candidates can be detected with high accuracy without being affected by the brightness of the image, that is, the brightness of the illumination at the time of shooting.

Here, when the intensity of the light source is L, the light intensity observed from the portion with the reflectance R1 on the subject is I1, and the light intensity observed from the portion with the reflectance R2 on the subject is I2. In the space where each logarithm is taken, the following equation holds.

  In other words, logarithmic conversion of pixel values in an image results in conversion into a space where the reflectance ratio is expressed as a difference. In such a space, only the reflectance of the subject that does not depend on the intensity L of the light source is evaluated. It becomes possible to do. In other words, it is possible to align different contrasts (here, the pixel value difference itself) depending on the brightness in the image.

  On the other hand, the color space of an image acquired by a device such as a general digital camera is sRGB. sRGB is an international standard color space that defines and unifies color, saturation, etc., in order to unify the differences in color reproduction between devices. In this color space, the gamma value (γout) is 2. The image pixel value is a value obtained by raising the input luminance to 1 / γout (= 0.45) in order to enable proper color reproduction in the .2 image output device.

  Therefore, the pixel value in the entire image is converted according to a so-called inverse gamma conversion, that is, according to a conversion curve that takes a logarithm after being raised to the power of 2.2, thereby evaluating only by the reflectance of the subject independent of the intensity of the light source. Can be performed properly.

  In other words, the normalization process can be said to be a process of converting pixel values in the entire image according to a conversion curve for converting a specific color space into a color space having different characteristics.

  By applying such processing to the image, different contrasts can be made according to the brightness in the image, and the accuracy of feature point candidate detection is improved.

  Note that the classifier included in the feature point detector is learned using the sample image that has been subjected to such normalization processing.

The first feature point search range setting unit 43 is configured to search each feature point on the resolution image so that the feature point is searched only in an area where each feature point can exist in order to reduce the time required for detecting the feature point candidate. it is to set the search range SR1 _i of the feature point _{X i.} This search range is determined based on the existence probability distribution of feature points statistically obtained from a training image set including a face, which will be described later, and the search range is set to be wider for feature points with large positional fluctuations. Like that. The feature point existence probability distribution is stored in a first database 58 to be described later, and the first feature point search range setting unit 43 reads the existence probability distribution from the first database 58 to obtain a resolution. A search range SR _1i is set for each image S1 ″ _{k and for} each feature point X _i .

FIG. 11 is a diagram showing search ranges SR1 ₁₁ , SR1 ₁₆ and SR1 ₁₇ for each feature point of the left eye corner (X ₁ ), right nose (X ₆ ), and left mouth corner (X ₇ ) as an example of the search range. It is. Since the mouth angle is a feature point that is subject to individual differences and is relatively easy to change, the search range is set wider than the corners of the eyes and the nose as shown in the figure.

When the face direction f of the detected face S1 is close to the front, the first feature point detector selection unit 46 selects the feature point to be detected as the type of the feature point detector to be used. ₁ ), left eye (X ₂ ), right eye (X ₃ ), right eye corner (X ₄ ), left nose (X ₅ ), right nose (X ₆ ), left mouth corner (X ₇ ), right mouth corner (X ₈₎ ), 10 types of feature point detectors, which are the middle point of the upper lip (X ₉ ) and the middle point of the lower lip (X ₁₀ ), are selected, while the face orientation f of the detected face S1 is 45 degrees oblique When close, the detected feature points are left eye corner (X ₁ ), left eye head (X ₂ ), right eye head (X ₃ ), right eye corner (X ₄ ), nose tip (X ₁₁ ), left mouth corner (X ₇ ). ), And seven types of feature point detectors for the right mouth corner (X ₈ ) are selected.

The first detection processing unit 44 determines a feature point X _i for each normalized resolution image S1 ″ _k using a feature point detector selected from the feature point detector group D1 _i. A feature point candidate is detected within a set search range for each type of feature point.

  As shown in FIG. 3, the first feature point determination unit 50 includes a first feature point existence probability distribution synthesis unit 51, a first feature point candidate selection unit 52, and a first feature point position determination unit 53. And a first database 58.

The first database 58 has other features different from the feature point X _i when statistically determined in advance, for each feature point X _{i of the} face, based on the position coordinates x _i of the feature point. Each of the existence probability distributions P1 _ij of the point X _j is stored.

The existence probability distribution P1 _ij is an existence probability distribution of points that are correct answers of other feature points with respect to the position coordinates of the feature points detected by a certain feature point detector, one feature point X _i and one other feature. This is prepared for each set in units of two sets of points _Xj, and the positional relationship between the feature points can be expressed using a plurality of sets of probability distributions. . That is, this existence probability distribution represents the first positional relationship model that defines the positional relationship of the feature points in the present invention with the first tolerance. Here, the existence probability distribution of the feature point X _i with respect to the output coordinate x1 _j of the feature point detector D1 _j is defined as P1 _ij (x _i | x1 _j ), and P1 _ij is represented by a two-dimensional histogram. The

In order to obtain the existence probability distribution P1 _ij , first, a face is detected from a training image set (thousands of images in which correct coordinates of facial feature points are input) so that the face is positioned at the reference position. Standardize images. FIG. 7 shows an example in which a face is detected from an image and the image is normalized so that the face is positioned at a predetermined reference position with a predetermined size in the center of the image.

Next, the feature point detector D1 _i detects the feature point X _i from the normalized image, and the difference between the coordinate x1 _i of the feature point and the correct coordinate x _j of the other feature point X _j is Comparison and totalization are performed in units of two sets of one kind of feature point X _i and one other kind of feature point X _j . An example of the existence probability distribution P1 _ij obtained by such learning is shown in FIG. Here, the position of the feature point detected by the feature point detector is represented by x, the existence probability distribution of the target feature point is represented by shading on the image, and the higher the existence probability of the target feature point, the more It is expressed so that the density at the position is high. 12 (a) is a diagram showing an existence probability distribution P1 ₂₁ for the left eye area of the point with respect to the position coordinate x ₂ points of the left eye head detected by the left eye head detector D1 ₂ (position coordinate x _1), FIG. 12 (b) is a diagram showing the existence probability distribution P1 ₂₅ of the point of the left nose (position coordinate x ₅ ) with respect to the position coordinate x ₂ of the point of the left eye, and FIG. it is a graph showing a presence probability distribution _{P1 28} point in the right mouth corner (coordinates _{x 8)} with respect to the position coordinate _{x 2} of the point. Note that the resolution of the two-dimensional histogram represented by the existence probability distribution P1 _ij is 100 × 100 pixel size, which is 1/4 of the image size when the normalized face image S1 is 200 × 200 pixel size, The learning efficiency is improved and the amount of information stored in the database is reduced.

The first feature point existence probability distribution synthesizing unit 51 uses the existence probability distribution P1 _ij stored in the first database 58, for each feature point X _i, is different from the one type of feature points X _i Existence probability distribution P1 _ij (x _i | q1 _{jt) of the} one type of feature point X _i on the input image S 0 when the position of each candidate Q 1 _jt of another type of feature point X _j is used as a reference. ) And the probability distribution is synthesized according to the following equation.

Here, P1 _i is the combined existence probability distribution of the feature point X _i , and P1 _ij (x _i | q1 _jt ) is the position coordinate q1 _jt of the t-th candidate Q1 _jt of the feature point X _j as a reference. The existence probability distribution of the feature point X _i (position coordinates x _i ), N _Q1j is the number of feature point X _j candidates, and N _D is the number of types of feature points defined in advance.

If a large number of candidates are detected for one type of feature point, the amount of calculation according to the above formula increases and the time required for the calculation becomes long. For example, when the number of candidates N _Q1j is 6 or more Alternatively, the calculation may be made only for the top five candidates with the highest detection score SCD.

The existence probability distribution P1 _i synthesized in this way can be considered as an estimation result when a position where a certain type of feature point exists is viewed from the position of another type of feature point different from the feature point. it can.

For each feature point X _{i of the} face, the first feature point candidate selection unit 52 uses the probability of the existence probability distribution P1 _i of the feature points X _i combined, and the likelihood of the shape indicating the positional relationship of the facial parts. calculated as the shape score SCG ₁ representing the magnitude of this shape score SCG _1, on the basis of the candidate _{Q1 it} the position of the characteristic point _{X i,} candidate _{Q1 'iu} (u from the candidates _{Q1 it} of the feature point = N _Q1′i ) is selected and narrowed down.

In this embodiment, the shape score SCG ₁ , that is, the point having the highest existence probability on the existence probability distribution P1 _i synthesized with the feature point X _i is set as the representative point C1 _i, and among the feature point candidates Q1 _it Candidates Q1 ′ _{iu to} be narrowed down are selected as candidates to be narrowed down within a predetermined range having the representative point C1 _i substantially at the center.

Here, the representative point C1 _i is defined according to the following equation.

As a method for narrowing down candidates, in addition to the above, among the candidates Q1 _it , the shape score SCG ₁ corresponding to this candidate is a predetermined threshold, for example, on the existence probability distribution P1 _i in which the feature points X _i are synthesized. A method of selecting and narrowing down only candidates having a shape score equal to or greater than a value obtained by subtracting a predetermined value from the maximum value of the existence probability may be used.

The first feature point position determination unit 53 determines the position coordinates of each candidate with respect to the candidates narrowed down by the first feature point candidate selection unit 52 for each facial feature point X _i . A weighted average of the position coordinates when weighted by the detection score is calculated, and the calculated position coordinates are determined as the position of the feature point X _i .

In this embodiment, the weight coefficient Weight1 _iu is calculated according to the following equation, and the position coordinate represented by ΣWeight1 _iu × q1 ′ _iu is determined as the position of the feature point X _i .

Here, the position coordinates _{x i} are the characteristic points _{X i,} q1 _'iu candidate _Q1' coordinates of _iu, weight1 _iu weighting _{factor, SCD 1iu} the _'detection scores _{iu, SCD 1imax} candidate _Q1' candidate Q1 _iu The maximum value of the detection score, N _1nor, is a normalization constant defined so that Σweight1 _iu = 1.

Note that the method for determining feature points is not limited to the present embodiment. For example, for the candidates that have been narrowed down, the position coordinates when the position coordinates of each candidate are weighted by the shape score corresponding to the candidate are shown. The weighted average may be determined as the position of the feature point X _i , and the weighted average of the position coordinates when the position coordinates of each candidate are respectively weighted by the sum of the detection score and the shape score corresponding to the candidate May be determined as the position of the feature point X _i . Alternatively, among the narrowed down candidates, the position coordinates of the candidate with the maximum detection score, the position coordinates of the candidate with the maximum shape score, or the position coordinates of the candidate with the maximum sum of the detection score and the shape score are feature points. it may be determined as the position of the X _i.

The second face normalization unit 60 performs geometric normalization processing again on the input image S0 so that each feature point determined by the first feature point determination unit 50 approaches a predetermined reference position. The image including the face is cut out to obtain a new normalized face image. However, in actuality, considering the efficiency of the process, the geometrical normalization process is performed on the resolution image S1 ″ _k from which the feature point of the face S1 has already been detected, and the normalized face image S2 ″ _k is obtained. .

The geometric normalization process at this time uses affine transformation that considers only scaling, rotation, and translation while maintaining the aspect ratio. This affine transformation is expressed by the following equation.

Here, (x, y) represents the coordinates of the pixel before conversion, (x ′, y ′) represents the coordinates of the pixel after conversion, and a, b, c, and d represent conversion parameters, respectively.

The second feature point candidate detection unit 70 calculates a detection score SCD indicating the probability that the feature point is included in the approximate center of the image of the determination target region designated on the normalized face image S2 ″ _k. When the detection score SCD is equal to or greater than a predetermined threshold, by determining that the target represented by the image of the determination target region is the feature point, the feature point candidate for each feature point X _i of the face Q _2it is detected.

  As shown in FIG. 4, the second feature point candidate detection unit 70 includes a second feature point search range setting unit 73, a second detection processing unit 74 including a second feature point detector group 75, And a second feature point detector selection unit 76.

Similar to the second feature point detector group 45, the second feature point detector group 75 is composed of a plurality of types of feature point detectors D2 _i prepared for each type of feature point X _i to be detected. Each feature point detector D2 _i detects the position of a feature point with high accuracy using a plurality of discriminators learned by a so-called machine learning method called AdaBoost. That is, this second feature point detector group represents the feature point detector group having the second detection accuracy and the second robustness in the present invention.

However, this classifier includes a plurality of feature point sample images and a plurality of normalized feature point sample images having a smaller tolerance than the case of the feature point detector D1 _i so that the position of a specific feature point is substantially the center. Based on the feature amount calculated from the image of the region to be identified with reference to the reference data learned about the feature amount of the non-feature point sample image, an identification point indicating the probability that the image includes the feature point is calculated The identification point threshold determination determines whether or not the image to be identified includes a specific feature point substantially at the center. Therefore, the feature point detector D2 _i is higher in detection accuracy than the feature point detector D1 _i , but has a feature of low robustness.

Similar to the first feature point search range setting unit 43, the second feature point search range setting unit 73 performs normalization face image S2 so as to search for feature points only in an area where each feature point can exist. ″ Sets the search range SR2 _i of each feature point X _i on _k . This search range is determined based on the probability distribution of feature points stored in the second database 88 described later. A feature point having a larger position variation is set with a wider search range.

Similar to the first feature point detector selection unit 46, the second feature point detector selection unit 76 selects feature points from the feature point detector D2 _i according to the face orientation f of the detected face S1. A feature point detector to be used for detection is selected.

Similar to the first detection processing unit 44, the second detection processing unit 74 uses the feature point detector selected by the second feature point detector selection unit 76 for the normalized face image S2 ″ _k. Thus, feature point candidates are detected within the set search range for each type of feature point determined among the feature points X _i .

  As shown in FIG. 5, the second feature point determination unit 80 includes a second feature point existence probability distribution synthesis unit 81, a second feature point candidate selection unit 82, and a second feature point position determination unit 83. And a second database 88.

The second database 88, like the first database 58 in advance statistically determined, for each feature point X _i of the face, when the reference position coordinates x _i of the feature point, the feature Each of the existence probability distributions P2 _ij of other feature points X _j different from the point X _i is stored.

The existence probability distribution P2 _ij is an existence probability distribution of points that are correct answers of other feature points with respect to the position coordinates of the feature points detected by a certain feature point detector, one type of feature point X _i and one other type of feature point X _i. Are prepared for each set in units of two sets of types of feature points _Xj, and the positional relationship between the feature points can be expressed using a plurality of sets of probability distributions. It can be done. That is, this existence probability distribution represents the second positional relationship model that defines the positional relationship of the feature points in the present invention with the second tolerance.

Here, the existence probability distribution of the feature point X _i with respect to the output coordinate x2 _j of the feature point detector D2 _j is defined as P2 _ij (x _i | x2 _j ), and P2 _ij is represented by a two-dimensional histogram. Is done.

The procedure for _{obtaining the} existence probability distribution _P2ij is as follows. First, with respect to a training image in which the correct coordinates of feature points are known, the face detection unit 20, the first face normalization unit 30, the first feature point candidate detection unit 40, and the first feature point determination unit 50 are used. To determine the position of the feature point for each type of feature point X _i . Next, a normalized face image normalized by the second face normalization unit 69 is obtained based on the determined position of the feature point, and the second feature point detector group 75 is obtained on the normalized face image. detecting a characteristic point X _i by the feature point detector D2 _i constituting the, the deviation between the correct coordinates x _j of the coordinate x2 _i and other characteristic point X _j of the feature point, and one characteristic point X _i 1 Comparison and aggregation are performed in units of two sets of two other feature points _Xj . Such processing is performed on a large number of training images and statistically learned to obtain the aforementioned existence probability distribution _P2ij .

Second characteristic point existence probability distribution synthesizing unit 81 uses the existence probability distribution P2 _ij stored in the second database 88, for each feature point X _i, different other than the one feature point X _i The existence probability distribution P2 _ij (x _i | q2 _jt ) of the one feature point X _i on the input image S0 when the position of each candidate Q2 _jt of the feature point X _j is used as a reference is obtained. The distribution is synthesized according to the following equation.

Here, P2 _i is the combined existence probability distribution of the feature point X _i , and P2 _ij (x _i | q2 _jt ) is the position when the position coordinate q2 _jt of the t-th candidate Q2 _jt of the feature point X _j is _used as a reference. The existence probability distribution of the feature point X _i (position coordinates x _i ), N _Q2j is the number of feature point X _j candidates, and N _D is the number of types of feature points defined in advance.

For each facial feature point X _i , the second feature point candidate selection unit 82 sets the probability in the combined existence probability distribution P2 _i of the feature point X _i to a shape score SCG representing the likelihood of the shape of the face part. ₂ , and based on the size of the shape score SCG ₂ and the position of the candidate Q2 _it of the feature point X _i , the candidate Q2 ′ _iu (u = N _{Q2 ′) is selected} from the feature point candidates Q2 _{it. i} ) Select and narrow down.

In the present embodiment, the shape score SCG ₂ , that is, the point having the highest existence probability on the combined existence probability distribution P2 _{i of} the feature point X _i is set as the representative point C2 _i, and among the feature point candidates Q2 _it Candidates that fall within a predetermined range of the region having the representative point C2 _i substantially in the center are selected as candidates Q2 ′ _{iu to} be narrowed down.

Here, the representative point C1 _i is defined according to the following equation.

As a method for narrowing down candidates, as in the first feature point candidate selection unit 52, the shape score SCG ₂ corresponding to this candidate is combined with a predetermined threshold, for example, the feature point X _i in the candidate Q2 _it. A method may be used in which only candidates having a shape score equal to or greater than a value obtained by subtracting a predetermined value from the maximum value of the existence probability on the existence probability distribution P2 _i are selected and narrowed down.

The second feature point position determining unit 83 determines the position coordinates of each candidate for the candidates narrowed down by the second feature point candidate selecting unit 82 for each feature point X _{i of the} face. A weighted average of the position coordinates when weighted by the detection score is calculated, and the calculated position coordinates are determined as the position of the feature point X _i .

In the present embodiment, the weight coefficient Weight2 _iu is calculated according to the following equation, and the position coordinate represented by ΣWeight2 _iu × q2 ′ _iu is determined as the position of the feature point X _i .

Here, the position coordinates _{x i} are characteristic point _{X i,} q2 _'iu candidate _Q2' coordinates of _iu, weight2 _iu weighting _{factor, SCD 2 IU} is _'detection scores _{iu, SCD 2imax} candidate _Q2' candidate Q2 _iu N _2nor is a normalization constant defined so that Σweight2 _iu = 1.

Note that the feature point determination method is not limited to the present embodiment, similar to the first feature point position determination unit 53. For example, for the narrowed candidates, the position coordinates of each candidate correspond to the candidates. The weighted average of the position coordinates when weighted by the shape score to be determined may be determined as the position of the feature point X _i , or the position coordinate of each candidate is the sum of the detection score and the shape score corresponding to the candidate The weighted average of the position coordinates when weighted respectively with may be determined as the position of the feature point X _i . Alternatively, among the narrowed down candidates, the position coordinates of the candidate with the maximum detection score, the position coordinates of the candidate with the maximum shape score, or the position coordinates of the candidate with the maximum sum of the detection score and the shape score are feature points. it may be determined as the position of the X _i.

  Next, processing performed in the present embodiment will be described. FIG. 6 is a flowchart showing processing performed in the present embodiment.

  First, the image input unit 10 receives an input of an image S0 that is a detection target of a face and its feature points (step ST1). Next, the face detection unit 20 detects the face S1 included in the input image S0, and acquires information on the positions of both eyes and the face orientation f of the face S1 (step ST2). For the detected face S1, the first face normalization unit 30 cuts out an image of a predetermined size including the face S1 whose relative size and position are normalized from the input image S0, and obtains a normalized face image S1 ′. Obtain (step ST3).

'When is obtained, the multi-resolution image generation unit 41, the normalized face image S1' normalized face image S1 to generate a plurality of resolution images S1, different resolutions on the basis of the _'k, illumination normalization unit 42, illumination so as to detect the brightness feature point candidate without being affected by the performs normalization of contrast of the image with respect to resolution images S1 _'k. Then, the first feature point search range setting unit 43 sets the search range SR1 _i of each feature point X _i on the resolution image based on the existence probability distribution P1 _ij stored in the first database 58. The first feature point detector selection unit 46 selects the type of feature point detector to be used in accordance with the detected face orientation f of the face S1, and sets the feature point detector in the first feature point detector group 45. select from the first detection processing section 44 applies the feature point detector selected for resolution images S1 _'k. That is, for the image of the determination target region designated on the normalized face image S1 ′, the detection score SCD ₁ indicating the probability that the feature point is included at the approximate center of the image is calculated, and the threshold value determination of the detection score SCD ₁ is performed. Accordingly, at least one detection candidate _{Q1 it} of its feature points in each feature point _{X i} of the face (step ST4).

When a feature point candidate is detected, the first feature point existence probability synthesis unit 51 uses the existence probability distribution P1 _ij stored in the first database 58, for each feature point X _i . The existence probability distribution P1 _ij (x _{i) of the} one feature point X _i on the input image S 0 when the position of each candidate Q 1 _jt of another feature point X _j different from the feature point X _i is used as a reference. | Q1 _jt ), and the probability distribution is synthesized according to the above equation (3). The first feature point candidate selection unit 52 sets the point having the highest existence probability on the shape score SCG ₁ , that is, the existence probability distribution P1 _i in which the feature points X _i are combined, as the representative point C1 _i. Of the candidates Q1 _it are selected as candidates Q1 ′ _{iu to} be narrowed down, which are candidates within a predetermined range having the representative point C1 _i as the center. Then, the first feature point position determining unit 53 detects the position coordinates of each candidate for the candidates narrowed down by the first feature point candidate selecting unit 52 for each facial feature point X _i . The weighted average of the position coordinates when weighted with the detection score at the time is calculated, and the calculated position coordinates are determined as the position of the feature point X _i (step ST5).

When the position of each feature point X _i is determined by the first feature point position determination unit 53, the second face normalization unit 60 causes each of the determined feature points to approach a predetermined reference position. Then, affine transformation is performed on the resolution image S1 ″ _k in which the feature point of the face S1 has already been detected, taking into account only the scaling, rotation, and translation while maintaining the aspect ratio, and an image including the face is cut out to create a new one. A normalized face image S2 ″ _k is obtained (step ST6).

When the normalized face image S2 ″ _k is obtained, the second feature point search range setting unit 73 performs the operation on the normalized face image S2 ″ _k based on the existence probability distribution P2 _ij stored in the second database 88. To set the search range SR2 _i of each feature point X _i . The second feature point detector selecting unit 76 selects the type of feature point detector to be used according to the detected face orientation f of the face S1, and sets the feature point detector in the second feature point detector group 75. select from the second detection processing section 74, the normalized face image S2 "with respect to _k, applying the feature point detector selected. that is, the normalized face image S2" determination target specified by the _k A detection score SCD ₂ indicating the probability that a feature point is included in the approximate center of the image is calculated for the image of the region, and the threshold value of the detection score SCD ₂ is used to determine the feature score for each facial feature point X _i . at least one detecting candidate _{Q2 it} (step ST7).

When a feature point candidate is detected, the second feature point existence probability synthesis unit 81 uses the existence probability distribution P2 _ij stored in the second database 88 to _generate one feature point X _{i for} each feature point X _i . The existence probability distribution P2 _ij (x _{i) of the} one feature point X _i on the input image S 0 when the position of each candidate Q 2 _jt of another feature point X _j different from the feature point X _i is used as a reference. | Q2 _jt ), and the probability distribution is synthesized according to the above equation (7). The second feature point candidate selecting unit 82 sets the shape score SCG ₂ , that is, the point having the highest existence probability on the existence probability distribution P2 _i synthesized with the feature point X _i as the representative point C2 _i , and the feature point Of the candidates Q2 _it are selected as candidates Q2 ′ _{iu to} be narrowed down in the range of a predetermined range whose center is the representative point C2 _i . Then, the second feature point position determining unit 83 detects the position coordinates of each candidate for the candidates narrowed down by the second feature point candidate selecting unit 82 for each facial feature point X _i . The weighted average of the position coordinates when weighted with the detection score of each time is calculated, and the calculated position coordinates are determined as the position of the feature point X _i . Then, the position information of the feature point X _i finally determined in this way is output as the position of the true feature point (step ST8).

  According to such a face feature point detection system according to the present embodiment, a plurality of types of feature point candidates of the face S1 in the input image S0 are detected by the first feature inspection having the first detection accuracy and the first robustness. Based on a first positional relationship model that uses a first tolerance to detect the position of the candidate feature points detected using the output device group 45 and the positional relationship among the plurality of types of feature points. Then, after determining the plurality of types of feature points having a positional relationship constrained by the first positional relationship model, a feature point candidate is determined in the vicinity of each of the determined plurality of types of feature points. A second feature point detector group 75 having a second detection accuracy higher than the detection accuracy of 1 and a second robustness lower than the first robustness, and the detected feature point candidates Position and position between the above-mentioned multiple types of feature points The plurality of types of feature points having a positional relationship constrained by the second positional relationship model are determined based on a second positional relationship model that defines the relationship with a second tolerance smaller than the first tolerance. Therefore, by adopting a two-stage configuration in which relatively fine detection is performed after relatively coarse detection, feature points can be detected at high speed with good detection accuracy, and the positional relationship between the feature points can be restricted. Thus, incorrect feature point candidates can be eliminated, and a plurality of types of feature points of the face S1 included in the image can be detected quickly and accurately.

  Further, according to the face feature point detection system according to the present embodiment, after the rough feature point detection, the second face normalization process, that is, the position of each detected feature point approaches a predetermined reference position, Fine feature point detection is performed after geometric normalization processing that maintains the aspect ratio, so the geometric shift of the face that could not be normalized before the coarse feature point detection was performed. Can be obtained, and the detection accuracy of feature points can be further improved.

  Further, according to the face feature point detection system according to the present embodiment, for each type of feature point, the candidate detects the position coordinates of each candidate feature point narrowed down by the first and second feature point candidate selection units. Since the weighted average when weighted with the detection score at the time of each is calculated, and the position coordinates represented by this weighted average are determined as the position of the feature point, there is a large deviation from the correct position of the feature point It is possible to suppress the influence of incorrect answer candidates that exist independently at the position.

  In this case, the weighted average is obtained by using exponentially expressed weighting coefficients such as Expressions (5-1), (5-2), Expressions (9-1), and (9-2). It is good to do. According to the experiment of the present applicant, it has been clarified that according to the method of determining the position of the feature point from such weighted average, the accuracy of the position of the feature point is further improved.

  The preferred embodiments of the present invention have been described above. However, the apparatus and method of the present invention and the program therefor are not limited to the above-described embodiments, and various increases / decreases and changes can be made without departing from the gist of the present invention. Can be added.

The block diagram which shows the structure of the face feature point detection system which is embodiment of this invention The block diagram which shows the structure of the 1st feature point candidate detection part 40. The block diagram which shows the structure of the 1st feature point determination part 50. The block diagram which shows the structure of the 2nd feature point candidate detection part 70. The block diagram which shows the structure of the 2nd feature point determination part 80. The flowchart which shows the process in the face feature point detection system which is embodiment of this invention The figure which shows the mode of face normalization processing The figure which shows the example of the probability distribution of the position of other feature points when one feature point is made into a reference | standard Diagram showing the state of multi-resolution processing The figure which shows the example of the sample image of the feature point used for the learning of the discriminator with which a feature point detector is equipped The figure which shows the example of the search range set according to the feature point on the image Figure showing an example of statistical distribution of feature points Diagram showing how to derive a classifier The figure which shows an example of the conversion curve of the pixel value used for an illumination normalization process

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image input part 20 Face detection part 30 1st face normalization part 40 1st feature point candidate detection part 41 Multi-resolution image generation part 42 Illumination normalization part 43 1st feature point search range setting part 44 1st Feature point detector selection unit 45 First feature point detector group 50 First feature point determination unit 51 First feature point existence probability distribution synthesis unit 52 First feature point candidate selection unit 53 First feature point position Determination unit 58 First database 60 Second face normalization unit 70 Second feature point candidate detection unit 73 Second feature point search range setting unit 74 Second feature point detector selection unit 75 Second feature inspection 80 second feature point determination unit 81 second feature point existence probability distribution synthesis unit 82 second feature point candidate selection unit 83 second feature point position determination unit 88 second database

Claims (10)

Using the first feature point detector group having the first detection accuracy and the first robustness generated by machine learning, the types of feature point candidates of the predetermined object in the detection target image First feature point candidate detecting means for detecting at least one for each;
Based on the position of candidate feature points detected by the first feature point candidate detection means and a first positional relationship model that defines the positional relationship among the plurality of types of feature points with a first tolerance. First feature point determining means for determining the plurality of types of provisional feature points having a positional relationship constrained by the first positional relationship model;
A second detection accuracy that is higher than the first detection accuracy generated by machine learning in the vicinity of each of the plurality of types of provisional feature points determined by the first feature point determination means. And second feature point candidate detecting means for detecting at least one for each type using a second feature point detector group having a second robustness lower than the first robustness,
A second feature that defines the position of the feature point candidates detected by the second feature point candidate detection means and the positional relationship between the plurality of types of feature points with a second tolerance smaller than the first tolerance; And a second feature point determining means for determining the plurality of types of final feature points having a positional relationship constrained by the second positional relationship model. Feature point detection device. A normality that performs geometric normalization processing that maintains the aspect ratio that brings the plurality of types of provisional feature points determined by the first feature point determination unit closer to a predetermined reference position on the detection target image. And further comprising
The feature point detection apparatus according to claim 1, wherein the second feature point candidate detection unit detects the candidate on a detection target image normalized by the normalization unit. The first feature point determining means is
The existence probability distribution on the image of the other feature point when the position of one feature point is used as a reference, which is statistically obtained for each combination of two different feature points among the plurality of types of feature points. And using the existence probability distribution of the one type of feature point with respect to the position of the detected candidate of the other type of feature point for one type of feature point. First existence probability distribution synthesizing means for obtaining each candidate and synthesizing the obtained existence probability distribution for each type of feature points;
For one type of feature point, based on the position of the detected candidate for the one type of feature point and the magnitude of the existence probability in the combined existence probability distribution of the one type of feature point, First feature point candidate selecting means for performing processing for selecting and narrowing down the one type of feature point candidates from the detected candidates for each type of feature points;
For each of the types of feature points, there is provided first feature point position determining means for determining the position of the temporary feature point of the type based on the position of the selected candidate of the type of feature point. The feature point detection apparatus according to claim 1, wherein the feature point detection apparatus is provided.   For each type of feature point, the first feature point candidate selecting means selects a point at a position having the highest existence probability in the synthesized existence probability distribution of the feature points of the type among the detected candidates. 4. The feature point detecting apparatus according to claim 3, wherein a candidate existing in a predetermined predetermined region is selected.   The first feature point candidate selecting means, for each type of feature point, out of the detected candidates, the presence of the type of feature points in the combined existence probability distribution corresponding to the position of the candidate 4. The feature point detection apparatus according to claim 3, wherein a candidate having a probability equal to or greater than a predetermined threshold is selected. The first feature point candidate detecting means detects the target in the identification target image based on a threshold value determination of certainty indicating the probability that the identification target image on the detection target image is an image including a feature point. Are detected as candidates for
The first feature point position determining means weights, for each type of feature point, the position coordinates of all candidates selected for the type of feature point with the certainty factor calculated for the candidate. 6. The feature point detection apparatus according to claim 3, wherein the weighted average of the position coordinates is determined as the position coordinates of the temporary feature point of the type.   For each of the types of feature points, the first feature point position determining means sets the position coordinates of all candidates selected for the type of feature points to the feature points of the type corresponding to the positions of the candidates. The weighted average of the position coordinates when weighted with the existence probability in the synthesized existence probability distribution is determined as the position coordinates of the temporary feature point of the type, 5. The feature point detection apparatus according to 5.   The feature point detection apparatus according to claim 1, wherein the predetermined object is a human face. Using the first feature point detector group having the first detection accuracy and the first robustness generated by machine learning, the types of feature point candidates of the predetermined object in the detection target image A first feature point candidate detection step of detecting at least one for each;
Based on the position of the feature point candidate detected by the first feature point candidate detection step and the first positional relationship model that defines the positional relationship among the plurality of types of feature points with a first tolerance. A first feature point determining step for determining the plurality of types of provisional feature points having a positional relationship constrained by the first positional relationship model;
A second detection accuracy higher than the first detection accuracy generated by machine learning in the vicinity of each of the plurality of types of provisional feature points determined by the first feature point determination step. And a second feature point candidate detecting step of detecting at least one for each type using a second feature point detector group having a second robustness lower than the first robustness;
A second feature that defines the position of the feature point candidates detected by the second feature point candidate detection step and the positional relationship between the plurality of types of feature points with a second tolerance smaller than the first tolerance; And a second feature point determining step for determining the plurality of types of final feature points having a positional relationship constrained by the second positional relationship model. Point detection method. Computer
Using the first feature point detector group having the first detection accuracy and the first robustness generated by machine learning, the types of feature point candidates of the predetermined object in the detection target image First feature point candidate detecting means for detecting at least one for each;
Based on the position of candidate feature points detected by the first feature point candidate detection means and a first positional relationship model that defines the positional relationship among the plurality of types of feature points with a first tolerance. First feature point determining means for determining the plurality of types of provisional feature points having a positional relationship constrained by the first positional relationship model;
A second detection accuracy that is higher than the first detection accuracy generated by machine learning in the vicinity of each of the plurality of types of provisional feature points determined by the first feature point determination means. And second feature point candidate detecting means for detecting at least one for each type using a second feature point detector group having a second robustness lower than the first robustness,
A second feature that defines the position of the feature point candidates detected by the second feature point candidate detection means and the positional relationship between the plurality of types of feature points with a second tolerance smaller than the first tolerance; A program for functioning as second feature point determining means for determining the plurality of types of final feature points having a positional relationship constrained by the second positional relationship model based on the positional relationship model.