JP2010055395A - Image processing apparatus and method, program, and storage medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve the accuracy of segmenting a face region in an image and the accuracy of detecting the position of a feature part. <P>SOLUTION: An image processing apparatus includes: a detection parameter setting part 105 for respectively setting parameters to be used to detect face feature parts in a setting mode for setting reference data about a face and in a determination mode for determining the face by using the reference data; a face feature part position detecting part 104 for detecting the positions of face feature parts by using the parameters set in the respective modes; a feature value calculating part 110 for calculating feature values on the basis of the positions of the feature parts detected in the respective modes; a reference data setting part 113 for setting the reference data on the basis of the feature value calculated in the setting mode; and a facial expression determining part 112 for determining the face by using the feature value calculated in the determination mode and the reference data. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a technique suitable for determination of a face in an input image, particularly determination of facial expression.

  Conventionally, in the field of image recognition and voice recognition, a recognition processing algorithm specialized for a specific recognition target is executed by software or hardware using a dedicated processor, so that the recognition target can be detected from the image including the recognition target and the background. What detects is known.

  In particular, many techniques for detecting a face as a specific recognition target have been conventionally disclosed as introduced in the survey of Non-Patent Document 1, for example.

  In recent years, the technique introduced in Non-Patent Document 2 has attracted attention. Further, as shown in Non-Patent Document 3, a technique for detecting not only a face facing front but also a face facing sideways has been developed.

  Non-patent document 4 and non-patent document 5 disclose techniques related to detection of facial feature parts such as eyes and mouths in facial images.

  As techniques for recognizing facial expressions in images, there are techniques introduced in Non-Patent Documents 6 and 7.

  In addition, a technique for recognizing a facial expression by detecting a change corresponding to an action unit of FACS (Facial Action Coding System), which is known as a method for objectively describing facial expression behavior, has been developed.

Patent Document 1 discloses a technique for facial expression recognition.
JP 2005-56388 A “Face detection and recognition by image processing”, Iwai et al., Information Processing Society of Japan Research Report 2005-CVIM-149 pp343-368, May 2005 Rapid Object Detection using a Boosted Cascade of Simple Features ", Paul Viola, Michael Jones, CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION 200 Learning Sparse Features in Granular Space for Multi-View Face Detection (C. Huang, H. Ai, Y. Li, * S. Lao; 401-406 7th Int'l Conf. Automatic Face & Gesture Recognition A land mark paper in face recognition "GM Beumer, Q. Tao, AM Basen, RNJ Veldhuis", 7th Int'l Conf. Automatic Face & Gesture Recognition T. T. F. Cootes, G .; J. et al. Edwards and C.C. J. et al. Taylor: “Active appearance models”, Proc. of the 5th European Conference on Computer Vision (Ed. by H. B. Neumann), Vol. 2, Springer, pp. 484498 (1998). G. Donate, T.W. J. et al. Sejnowski, et. al, “Classifying Facial Actions”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 21, no. 10, Oct, 1999 Y. Tian, T .; Kanade, and J.H. F. Cohn "Recognizing Action Units for Facial Expression Analysis" IEEE Transactions on Pattern Analysis and Vol. 23, no. 2, Feb. 2001

  As described in Non-Patent Document 2 and Non-Patent Document 3, the face detection accuracy in terms of whether or not the face in the image has been successfully detected is improved. However, there has not been much development in terms of the positional accuracy of the face detection result, that is, whether the coordinates of the detected face position are always at the same position (for example, the face center) in the face region.

  Even in the detection of facial feature parts, a certain amount of error occurs in the detection position.

  In the technique for recognizing facial expressions in an image, the technique of Non-Patent Document 6 is based on the premise that a partial region of a face is accurately cut out from a frame image by visual observation. In the method of Non-Patent Document 7, rough positioning of the face pattern is automated, but fine positioning by human visual inspection is required for positioning of the characteristic part.

  As described above, the processing for extracting a face region from an input image or the detection accuracy of the position of a characteristic part used for facial expression recognition affects the accuracy of facial expression recognition.

  An object of the present invention is to provide an image processing apparatus and method which are not easily affected by the accuracy of clipping a face region in an image and the detection accuracy of a characteristic part position.

  In order to achieve the above object, the image processing apparatus of the present invention is used for detection of a facial feature portion in a setting mode for setting reference data related to a face and a determination mode for determining a face using the reference data. Parameter setting means for setting parameters based on the respective modes, and in the setting mode and the determination mode, the positions of facial feature portions existing in the images input in the respective modes are set in the respective modes. In the setting mode, the detection means for detecting using the parameter, the calculation means for calculating the feature amount based on the position of the feature portion detected in each mode in the setting mode and the determination mode, Reference data setting means for setting reference data based on the feature amount calculated in the setting mode; In de, and a judging means for judging for the face by using the reference data set feature amount calculated by the determination mode in the setting mode, present in the input image by the determination mode.

  According to another aspect of the present invention, in the setting mode for setting reference data related to a face in the image processing method, the step of setting a first parameter used for detection of a facial feature part, and the setting mode, Detecting a position of a feature part of a face existing in an image input in the setting mode using the first parameter; and a position of the feature part detected in the setting mode in the setting mode; In the setting mode, in the setting mode, in the step of setting reference data based on the feature amount calculated in the setting mode, and in the determination mode in which determination is performed on the face using the reference data A step of setting a second parameter independent of the first parameter as a parameter used for detection of a facial feature part; Detecting the position of the feature portion of the face present in the image input in the determination mode using the second parameter, and the feature portion detected in the determination mode in the determination mode. An image input in the determination mode using the feature amount calculated in the determination mode and the reference data set in the setting mode in the determination mode. And a step of determining a face existing inside.

  According to the present invention, it is possible to improve the detection accuracy of a facial feature part by setting a parameter used when detecting a facial feature part, for example, a detection area, for each operation mode.

  In addition, in the aspect in which the face feature part position detected in the setting mode is used as the initial value of the correction position when the face feature part position detected in the determination mode is corrected, the region extraction accuracy and the face feature part detection are detected. Facial expression recognition that is less affected by accuracy can be realized.

  Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.

  In the present embodiment, a facial expression recognition process for determining whether a facial expression in an input image is a predetermined facial expression will be described. Further, an explanation will be given using an example in which a predetermined facial expression is a smile.

  FIG. 1 is a diagram illustrating a functional configuration of the image processing apparatus according to the present embodiment. The image processing apparatus according to the present embodiment includes a control unit 100, a face detection unit 101, a target face selection unit 102, a normalized face image creation unit 103, and a facial feature part position detection unit 104. Further, it includes a detection parameter setting unit 105, a detection parameter holding unit 106, an operation mode holding unit 107, an operation mode setting unit 108, a feature amount evaluation unit 109, a feature amount calculation unit 110, an expression degree calculation unit 111, and an expression determination unit 112. Further, a reference data setting unit 113, a reference data holding unit 114, a facial feature part position correcting unit 115, a switching unit 116, and a correction result holding unit 117 are provided. Hereinafter, each part will be described.

  The control unit 100 performs processing for controlling the entire image processing apparatus according to the present embodiment. The control unit 100 includes a face detection unit 101, a target face selection unit 102, a normalized face image creation unit 103, a face feature part position detection unit 104, a detection parameter setting unit 105, a detection parameter holding unit 106, an operation mode holding unit 107, The operation mode setting unit 108 is connected. Further, the control unit 100 includes a feature amount evaluation unit 109, a feature amount calculation unit 110, a facial expression degree calculation unit 111, a facial expression determination unit 112, a reference data setting unit 113, a reference data holding unit 114, a facial feature part position correction unit 115, and a switching. Unit 116 and correction result holding unit 117. Then, the control unit 100 controls each unit so that each unit operates at an appropriate timing.

  The face detection unit 101 detects a face area (an area of a face image included in the input image) in the input image. Specifically, processing for obtaining the number of face areas, the coordinate position of the face area, and the size of the face area in the input image is performed.

  The target face selection unit 102 selects a face to be subjected to facial expression recognition from the information on the face area detected by the face detection unit 101. Although the selection method is not limited, for example, it can be determined from the size of the face or the position of the face in the input image.

  The normalized face image creation unit 103 creates a normalized face image in which the size and rotation (orientation) of the face are normalized using the face area information for the face selected by the target face selection unit 102. The creation method will be described later.

  The face feature part position detection unit 104 detects a face feature part used for facial expression recognition from the normalized image created by the normalized face image creation unit 103. This detection method will be described later.

  The detection parameter setting unit 105 sets parameters used when the facial feature part position detection unit 104 detects a facial feature part, for example, a detection area. This setting is performed by selecting from a plurality of parameters held by the detection parameter holding unit 106 in accordance with an operation mode held by an operation mode holding unit 107 described later. The detection parameter holding unit 106 holds detection parameters set for each operation mode.

  The operation mode holding unit 107 holds information on the operation mode set by the operation mode setting unit 108. The operation mode setting unit 108 sets a new operation mode in the operation mode holding unit 107 in order to change the operation mode when the evaluation result of the feature amount in the feature amount evaluation unit 109 described later is a predetermined result. .

  The feature amount evaluation unit 109 evaluates the feature amount calculated by the feature amount calculation unit 110 described later. The evaluation method will be described later. If the evaluation result is a predetermined result, a reference data setting unit 113 (to be described later) sets the feature amount as reference data via the control unit 100, and the operation mode setting unit 108 uses a predetermined operation mode. To set.

  The feature amount calculation unit 110 calculates a feature amount from the position of the face feature part detected by the face feature part position detection unit 104 or the correction position of the face feature part corrected by the face feature part position correction unit 115 described later. . A feature amount calculation method will be described later.

  The expression level calculation unit 111 calculates the expression level based on the feature amount calculated by the feature amount calculation unit 110. A method of calculating the expression level will be described later.

  The facial expression determination unit 112 evaluates the facial expression degree calculated by the facial expression degree calculation unit 111 and determines the facial expression in the input image selected by the target face selection unit 102. The determination method will be described later.

  The reference data setting unit 113 sets the feature amount calculated by the feature amount calculation unit 110 in the reference data holding unit 114 as reference data when the evaluation result of the feature amount by the feature amount evaluation unit 109 is a predetermined result. .

  The face feature part position correcting unit 115 corrects the position of the face feature part detected by the face feature part position detecting unit 104. The correction method will be described later.

  Based on the operation mode information held by the operation mode holding unit 107, the switching unit 116 outputs the position result of the face feature part detected by the face feature part position detection unit 104 based on the operation mode information held by the operation mode holding unit 107. Switch to the characteristic part position correction unit 115.

  The correction result holding unit 117 holds the correction position and correction amount of the facial feature part obtained by the facial feature part position correction unit 115.

  Next, the operation mode will be described. There are two types of operation modes of the present embodiment: a reference data search mode and a facial expression determination mode. An overview of processing in these modes will be described. Detailed description will be given later.

  The reference data search mode is a setting mode for searching and setting reference data used in facial expression determination processing in the facial expression determination mode. More specifically, when the feature amount calculated by the feature amount calculation unit 110 satisfies a predetermined condition, the feature amount or a value obtained from several past feature amounts including the feature amount is referred to as reference data. To do.

  In the facial expression determination mode, a facial expression indicating the degree of facial expression is calculated based on the change in the reference data set in the reference data search mode and the feature amount obtained from the current processing target image. This is a determination mode for determining facial expressions.

  FIG. 2 is a flowchart showing the procedure for setting the operation mode. First, as an operation mode at the start of processing, a reference data search mode is initially set in step S201. In step S202, the feature amount evaluation unit 109 evaluates whether the feature amount satisfies a predetermined condition. If it is evaluated that the feature amount satisfies a predetermined condition, the operation mode is changed to the facial expression determination mode in step S203, and facial expression determination is performed.

  Next, main processing for discriminating facial expressions in the input image, which is executed by the operations of the above-described units, will be described. The processing in the reference data search mode will be described with reference to FIG. 3, and the processing in the facial expression determination mode will be described with reference to FIG.

  FIG. 3 is a flowchart showing a processing procedure in the reference data search mode. In step S301, an image to be processed is input. The image may be a single image cut out from a moving image file as a still image, a frame image captured by an imaging system (not shown), or the like, but is not particularly limited.

  In step S302, the face detection unit 101 detects a face in the input image. Various face detection processing methods have been proposed as described in the background art. For example, a method using a rectangular filter introduced in Non-Patent Document 2 is used.

  In step S303, the target face selection unit 102 selects a face to be subjected to facial expression recognition from the faces detected in step S302. If only one face is detected, that face is selected. When there are a plurality of detected faces, for example, a face close to the center of the input image is selected from faces of a predetermined size or larger.

  In step S304, the normalized face image creation unit 103 creates a normalized face image from the image of the face area selected in the target face selection step S303. This creation method is schematically shown in FIG.

  FIG. 5A shows the face area in the input image before normalization, and FIG. 5B shows the face area after normalization. In FIG. 5A, 501 indicates a face area, 502 indicates the center coordinates of the detected face, 503 indicates the right eye coordinates of the detected face, and 504 indicates the left eye coordinates of the detected face. In FIG. 5B, reference numeral 511 denotes a normalized face area. That is, the face area 501 in the input image is rotated in the plane, and the face size is also different from the normalized face area 511 shown in FIG.

  In FIG. 5B, 512 indicates the coordinates on the normalized image of the position 502 used as the center coordinates of the face area when creating the normalized face image. Reference numerals 513 and 514 denote the coordinates on the normalized image of the right eye and left eye positions 503 and 504 used for obtaining the size of the face area, respectively.

  The normalization processing corrects the in-plane rotation amount so that the straight line connecting the right eye coordinate 503 and the left eye coordinate 504 is horizontal with the center coordinate 502 of the face as the center, and the right eye coordinate 503 and the left eye coordinate 504 are corrected. So that the line connecting the two has a predetermined length. This normalization process is performed by a geometric transformation process called affine transformation.

  For the detection of the right eye coordinates 503 and the left eye coordinates 504 used in this normalization process, for example, the techniques of Non-Patent Document 4 and Non-Patent Document 5 described above can be used. In this normalization process, the luminance value can be normalized. For normalization of the luminance value, for example, a histogram flattening process is used. This histogram flattening process is described in Non-Patent Document 7.

  In step S305, it is determined whether or not a detection parameter used by the face feature part position detection unit 104, for example, a face detection area, is set. If it is not set, the process proceeds to step S306, and if it is set, the process proceeds to step S307.

  In step S <b> 306, the detection parameter setting unit 105 sets face detection parameters used by the face feature part position detection unit 104. The detection parameter setting unit 105 selects and sets a detection parameter from a plurality of detection parameters held by the detection parameter holding unit 106 according to the current operation mode held by the operation mode holding unit 107.

  In this embodiment, information for specifying a detection area is used as a detection parameter. This detection area will be described. The detection area of this embodiment is specified by a mask. Hereinafter, this mask is referred to as a detection area mask.

  FIG. 6 is a diagram illustrating an example of a detection area mask used when detecting an end point of a mouth as a facial feature part. In FIG. 6A, reference numeral 601 denotes a detection area mask used in the reference data search mode. In FIG. 6B, reference numeral 602 denotes a detection area mask used in the facial expression determination mode. In FIG. 6, reference numerals 603 and 604 denote detection areas, and reference numerals 605 and 606 denote non-detection areas.

  These detection area masks are set at predetermined positions in the normalized image shown in FIG. 5B for each feature part of the face to be detected. In the face feature part position detection process described later, face feature parts are detected in the detection areas 603 and 604 in the detection area masks 601 and 602.

  FIG. 8 shows an example of a region for setting a detection region mask in the normalized image. Reference numeral 801 denotes an area for setting a right eye corner detection mask, and reference numeral 802 denotes an area for setting a left eye corner detection mask. Reference numeral 803 denotes an area for setting a right mouth end point detection mask, and reference numeral 804 denotes an area for setting a left mouth end point detection mask.

  For example, the detection area mask for detecting the mouth end point shown in FIG. 6 will be described. In the areas 803 and 804 for setting the right mouth end point detection mask and the left mouth end point detection mask, the detection areas shown in FIG. 6 are set independently for each operation mode. That is, in the reference data search mode, the detection area mask 601 is selected and the detection area 603 is set as the first parameter. In the facial expression determination mode, the detection area mask 602 is selected and the detection area 604 is set as the second parameter.

  As described above, the normalized image is generated based on the detection position of the face and the detection position of both eyes. Therefore, the positions where the mouth end points exist in the normalized image are distributed in a certain range due to individual differences and facial expressions. Further, the distribution is made in a certain range due to the error of the detection position of the face and the detection position of both eyes. Therefore, the detection area of the end point of the mouth becomes a wide area like the detection area 603 of the detection area mask 601 and the detection area 604 of the detection area mask 602 shown in FIG.

  Further, in the facial expression recognition process of the present embodiment, the facial expression degree indicating the degree of facial expression is calculated based on the change between the reference data set in the reference data search mode and the feature amount obtained from the current image, The facial expression is determined based on the expression level.

  In other words, in order to determine the facial expression, a change from a state where there is no change in the expression or the expression is investigated. Therefore, in the reference data search mode, the feature value to be set as the reference data is a feature value obtained from a face with no expression or little change in expression. Also, in the facial expression determination mode, the change from a state where there is no change in facial expression or facial expression is investigated. For this reason, the detection area 603 of the detection area mask 601 used in the reference data search mode shown in FIG. 6 is narrower than the detection area 604 of the detection area mask 602 used in the facial expression determination mode.

  In order to create the detection areas 603 and 604, for example, an image database A that collects faces with no expression or little change in expression and an image database B that collects faces with detected facial expressions are prepared. When the detection area 602 is determined, the frequency distribution of the position of the target detection portion in the normalized face image obtained by performing the face detection process and the normalization process described above on each image in the database A. And a region where the frequency at each position is equal to or higher than a threshold value is set as a detection region 602. When the detection area 604 is determined, the position of the target detection site in the normalized face image obtained by performing the above-described face detection process and normalization process on each of the images in the databases A and B is determined. The frequency distribution is examined, and an area where the frequency at each position is equal to or higher than a threshold is set as a detection area 604. In this way, the detection area mask is obtained and prepared in advance.

  Note that the detection area mask is prepared for each detection site detected by the facial feature site position detection process described later. That is, when detecting the corners of the eyes in addition to the end points of the mouth, detection area masks adapted to the respective characteristic parts are used.

  In addition to the detection area mask, parameters used for detecting the position of the facial feature part described later are set. For example, when the method used for the detection is based on Non-Patent Document 2, a rectangular filter group called a weak classifier corresponds to the parameter when a template matching method is used.

  The weak classifier and the template are also prepared in advance from a reference image search mode and a facial expression, which are prepared in advance from an image database A that collects faces without changes in expression or expression and an image database B that collects faces with facial expressions to be detected. Find what to use in each of the judgment modes.

  That is, the weak classifier obtained by learning to detect the facial feature portion in the image of the image database A is used in the reference data search mode, and the facial feature portion in the images of the image databases A and B is used. The weak classifier obtained by learning to detect is used in the facial expression determination mode.

  As for the template, a template generated for each feature part of the face to be detected from the average image of the images in the image database A is used in the reference data search mode. Further, a template generated for each feature part of the face to be detected from the average image of the images in the image databases A and B is used in the facial expression determination mode.

  In step S307, the position of the facial feature part is detected. This detection is performed by the face feature part position detection unit 104. For the detection of the face feature part position, the method described in Non-Patent Document 4 described as the eye detection method in step S304 can be used. Alternatively, a template matching method can be used. In the normalized image creation step S304, the eye position is detected with respect to the face in the input image. In step S307, a detection process is performed on the normalized image created in step S304. For this reason, in general, the magnitude of variation is smaller than the target face in step S304. Therefore, it is also possible to detect a facial feature part using a template generated for each facial feature part to be detected from an average image of a plurality of images. The template matching method is a known technique and is described in Non-Patent Document 7, for example.

  An example of the facial feature part detected in step S307 is shown in FIG. In FIG. 7, 701 indicates the right eye corner, 702 indicates the left eye corner, 703 indicates the right mouth end point, and 704 indicates the left mouth end point.

  In step S308, a feature amount is calculated from the position of the facial feature part detected in step S307. FIG. 9 is a diagram showing the feature amount calculated in step S308. Reference numeral 901 represents the distance between the right eye corner and the right mouth end point, 902 represents the distance between the left eye corner and the left mouth end point, and 903 represents the distance between the mouth end points. Each distance is calculated from the coordinate value on the normalized image of each facial feature part.

  In step S309, the feature amount evaluation unit 108 evaluates the feature amount obtained in step S308. This evaluation is performed to confirm whether or not the current input image is an expressionless expression or an expression with little change, and whether or not the facial feature portion detected in step S307 has been erroneously detected. The above confirmation can also be performed by directly using the position of the facial feature part detected in step S307. However, in the present embodiment, a feature amount that is the distance of the detected face feature part position is used.

  This is because the detection error of the position of the face center coordinate detected in step S302 can be canceled. In other words, when there is an error in the face detection position, the distribution of the facial feature part position when there is no expression or expression with little change is a wider distribution than the original distribution due to the error. Since the amount of error is not necessarily small compared to the amount of movement when the position of the facial feature part changes due to a facial expression such as a smile, in order to reduce the effect of the error, the above-mentioned feature amount is used. It is used for evaluation.

  The feature amount evaluation is performed, for example, by determining whether a face image in which all feature amount evaluations are included in a predetermined range is present in a predetermined ratio in the continuous image. The technique of this evaluation method is disclosed in Japanese Patent Application No. 2007-160680 filed earlier by the present applicant.

  If the reference data is inappropriate based on the evaluation result, the process returns to step S301 to perform the reference data search process for the new input image. If it is appropriate as reference data, the process proceeds to step S310.

  In step S310, the reference data setting unit 113 sets the feature amount calculated in step S308 as reference data in the reference data holding unit 114. For the calculation of the reference data, the feature amount calculated in step S308 may be set as it is. As another calculation method, for example, the feature amount evaluation in step S309 is performed by determining whether a face image in which all feature amount evaluations are included in a predetermined range is present in a predetermined ratio in the continuous image. Suppose it was done. In that case, you may use the average value of each feature-value within the predetermined range in a continuous image as reference data.

  In step S <b> 311, the operation mode setting unit 108 changes the operation mode from the reference data search mode to the facial expression determination mode, and holds the information in the operation mode holding unit 107.

  As described above, in the reference data search mode, the feature amount calculation unit 110, the facial expression degree calculation unit 111, the facial expression determination unit 112, the facial feature part position correction unit 115, and the correction result holding unit 117 in FIG. 1 do not operate.

  Next, the operation in the facial expression determination mode will be described. FIG. 4 is a flowchart showing a processing procedure in the facial expression determination mode.

  In step S401, the detection parameter setting unit 105 sets parameters used for detecting the position of the facial feature part in step S406 described later. Since the operation mode held in the operation mode holding unit 108 is the facial expression determination mode, the detection parameter setting unit 105 detects from the detection parameters held in the detection parameter holding unit 106 to be used in the facial expression determination mode. Select and set parameters.

  In step S402, an image to be processed is input, and in step S403, a face in the input image is detected.

  In step S404, a face to be subjected to creation of a normalized face image is selected from the faces detected in step S403. In step S405, a normalized face image is created from the face image selected in step S404.

  In step S406, the position of the facial feature part is detected. Note that the detection parameter used in this detection is a parameter for face feature part position detection processing in the facial expression determination mode set in step S401. That is, for example, the detection area mask for detecting the mouth end point is the detection area mask 602 in the facial expression determination mode shown in FIG. 6B, and the detection range is the detection area 604.

  In step S407, the facial feature part position detected in face feature part position detection step S406 is corrected. This correction processing is performed by the facial feature part position correcting unit 115, and the correction value is held in the correction result holding unit 117. Since the current operation mode is the facial expression determination mode, the switching unit 116 illustrated in FIG. 1 sends the facial feature part position information detected by the facial feature part position detection unit 104 to the facial feature part position correction unit 115. It has been switched to output.

  As shown in the background art, a certain amount of error occurs in the detection result position and the eye detection position of the face detection process. Then, when the normalized image is created, the estimated face size and the estimated rotation amount are different from the correct face size and the rotation amount, and the created normalized image is different between consecutive images. An example is shown in FIG. 10A shows an image when the eye detection result position 1001 is correct, and FIG. 10B shows a state where the eye detection result position 1002 is erroneously detected inside the correct answer. 10C shows a normalized face image created based on the eye detection result shown in FIG. 10A, and FIG. 10D shows a normalized face image created based on the eye detection result shown in FIG. 10B. A face image is shown.

  Comparing FIG. 10C and FIG. 10D, FIG. 10D shows that the face size in the normalized image is large, and the corners of the corners of the eyes and mouth are the positions of FIG. It turns out that it has moved outside. That is, for example, for a continuous image, a normalized image is created with the correct face detection / eye detection result as shown in FIG. 10A for an image at a certain time T, and the corners of the eyes and mouth are detected. Next, it is assumed that a normalized image is generated from the erroneously detected eye detection result as shown in FIG. 10B with respect to the image at time T + 1, and the corners of the corners of the eyes and the mouth are detected. Then, although there was no change in the facial expression in the input image, the mouth feature spread sideways and the distance between the corner of the eye and the mouth endpoint increased at the face feature position detected in step S406. It is evaluated as follows.

  The erroneous evaluation of changes accompanying such erroneous detection degrades the accuracy of facial expression determination. Therefore, the feature point detection position does not change suddenly, that is, even if such a change is detected, it is not easily affected, and the feature point changes gently on the normalized image. Then, processing for correcting the feature point position is performed.

For this correction, a process generally called robust estimation is used. In brief, this process is affected if it is in the vicinity of the previous value, but it is less likely to be affected if it exceeds a certain range. This robust estimation algorithm is expressed by the following equation. Assuming that the position of the facial feature part detected in the normalized image at time t is x _t , the correction value θ _{t at} that time is updated according to the following equations 1 and 2. At this time, the previous correction result θ _t-1 and the correction amount ∂E _t-1 / ∂θ are held in the correction result holding unit 117. Further, the newly obtained correction result θ _t and the correction amount ∂E _t / ∂θ are held in the correction result holding unit 117. However, in the initial value, Expression 3 is used instead of Expression 2.

Here, ρ (·) is a positive finite even function, α is a memory rate, and η is a constant representing an adaptation rate. ε is a function representing an error between an observed value and an estimated value, and is usually expressed as follows.
_{ε t = x t -θ t-} 1 ( Equation 4)
Here, the function of Expression 5 is used as a function used for ρ (·). This is because the detection result of the facial feature part position when the normalized image is incorrect deviates significantly compared to the result when the normalized image is correct. However, it quickly follows the close value. Is set in consideration of the purpose.

  This robust estimation largely depends on the initial value. In other words, if the position when the result of detecting the facial feature part position is erroneously detected is erroneously set as reference data, it is difficult to perform correction. Therefore, the reference data is determined as described in the parameter setting in step S306 in the reference data search mode. That is, the detection parameters obtained by using an image database in which faces that have no expression or little change in expression, such as a region detection mask with a limited range, are used. In step S309, the feature amount is also evaluated. In this way, facial expression recognition that is less susceptible to the error of the face detection result is performed by combining high accuracy of the initial value and robust estimation.

  In step S408, the feature amount shown in FIG. 9 is calculated. In step S409, the facial expression degree calculation unit 111 calculates the facial expression degree using the feature amount calculated in step S408 and the reference data held in the reference data holding unit 114 in the reference data search mode.

  In the facial expression degree calculation process, as shown in FIG. 11, first, a feature quantity ratio is obtained from the feature quantity and reference data in step S1101, and then in step S1102, the facial expression degree is computed from the feature quantity ratio.

In step S1101, the feature amount ratio is obtained based on the following Expression 6. In Expression 6, Cj represents a feature amount ratio, Fj represents a feature amount, and Rj represents reference data. The subscript j indicates the index of each feature shown in FIG.
Cj = Fj / Rj (Formula 6)
In step S1102, as shown in Equation 7 below, the facial expression degree S is obtained by the product-sum operation of each feature amount and the coefficient corresponding thereto. Wj indicates a coefficient for each feature quantity.
S = Σ (Wj · Cj) (Formula 7)
The method for obtaining the coefficient Wj is not particularly limited, and for example, a method called linear discriminant analysis (LDA) can be used. This will be briefly described below. Note that the linear discriminant analysis is described in detail in Non-Patent Document 8.

Characteristic ratio vector C = (C1, C2,..., Cn) ^t
And Note that t represents transposition. As shown in FIG. 9, the dimension of this vector is 3 in this embodiment. Linear discriminant analysis can be considered as a method of discriminating by projecting a feature vector in an n-dimensional space onto a one-dimensional space. The projection at this time is
y = W ^t C (Formula 8)
The coefficient W of the feature ratio vector C becomes a discrimination parameter and becomes a coefficient vector to be obtained this time. Further, y is an output value of the discriminator, and this value is used as the expression level S. This W is obtained by Equation 9.
W = Sw ⁻¹ (M1−M2) (Formula 9)
In Equation 9, Mi represents a sample average, and Sw represents a scattering matrix. The sample average Mi is obtained by Equation 10 and the scattering matrix Sw is obtained by Equation 11.

  If the class of facial expression to be determined (in this embodiment, a smile image) is represented by class 1, and the other facial expression class is represented by class 2, the sample average Mi of each class is represented by equation (10). Ni represents the number of samples in each class.

  Further, Sw is obtained from Equation 11 as a scattering matrix.

Specifically, in order to obtain W, an image of a facial expression to be determined in advance (a smile image in the present embodiment) and a number of other facial expressions are prepared, and each image is determined from the selected facial expression of the expressionless face On the other hand, a feature amount ratio Cj is obtained. Then, the sample mean M _i smile and other facial expressions from equation 10, and obtains a scatter matrix Sw from Equation 11. And the coefficient of Formula 7 can be calculated | required by calculating | requiring W using Formula 9.

  In step S410, the facial expression degree obtained in step S409 is compared with a threshold value, and the facial expression determination unit 112 determines whether the input image is a desired facial expression. If it is not a desired facial expression, the process returns to step S402 to process a new input image. This threshold value can be obtained when the coefficient Wj of Expression 7 used in the expression level calculation described above is obtained. That is, the smile degree S is obtained using the obtained coefficient Wj for all face images prepared in advance. Then, for each facial image belonging to each of the desired facial expression and the other facial expressions, a frequency distribution of the obtained smile degree is created, and threshold candidates are changed on the frequency distribution to achieve a desired detection accuracy. The threshold value at that time is set as the threshold value.

  FIG. 12 is a diagram illustrating a hardware configuration of a computer that implements the image processing apparatus according to the present embodiment. The CPU 121 executes various processes by executing a program and controls each part of the apparatus. The ROM 122 stores programs and various fixed data. The RAM 123 temporarily stores various data and programs and provides a work area to the CPU 121. The CPU implements the processing described above with reference to the flowchart by executing a program in the ROM 122 or a program loaded from the HDD 125 or the like into the RAM 123.

  The input unit 124 includes a keyboard, a mouse, and the like, and receives input of data and instructions from the user. The HDD 125 is a hard disk drive, and stores programs, dictionary data, image data, and the like in a nonvolatile manner. The display unit 126 displays input data and processing results. A CD ROM drive or the like may be provided instead of the HDD 125.

  The I / F 127 is connected to the imaging device, receives an image captured by the imaging device, and transmits an instruction to the imaging device. A bus 128 connects each part of the apparatus.

  As described above, according to this embodiment, in the determination of the set type of facial expression, the detection of the position of the facial feature part used in the calculation of the feature amount used for the determination of the facial expression is performed with high accuracy. For this purpose, parameters such as a detection area, which are used for detection of facial feature parts necessary for determination of the set type of facial expression, are set based on the operation mode. Further, the position of the detected facial feature part is corrected in order to make it less susceptible to errors during normalization image creation. As described above, highly accurate facial expression recognition is realized.

  Note that the object of the present invention can also be achieved by reading and executing software program codes for realizing the functions of the above-described embodiments from a computer-readable storage medium by a computer (or CPU) of the apparatus.

  In this case, the program code itself read from the computer-readable storage medium realizes the functions of the above-described embodiments, and the computer-readable storage medium storing the program code constitutes the present invention.

  Various media can be used as a computer-readable storage medium for supplying the program code. For example, a flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, DVD-ROM, DVD-R, magnetic tape, nonvolatile memory card, ROM, or the like can be used.

  Further, the functions of the above-described embodiments are not only realized by executing the program code read by the computer. This includes the case where the OS (operating system) running on the computer performs part or all of the actual processing based on the instruction of the program code, and the functions of the above-described embodiments are realized by the processing. It is.

It is a figure which shows the function structure of the image processing apparatus of embodiment. It is a flowchart which shows the setting procedure of an operation mode. It is a flowchart which shows operation | movement of reference data search mode. It is a flowchart which shows operation | movement of facial expression determination mode. It is a figure explaining the production | generation of a normalized face image. It is a figure explaining the example of a detection area mask. It is a figure explaining the example of a face feature part. It is a figure explaining the example of a face feature part detection area. It is a figure which shows the example of a feature-value. It is a figure which shows the relationship between the detection error of a face feature part, and a normalized face image. It is a flowchart which shows the operation | movement of a facial expression degree calculation process. It is a figure which shows the hardware constitutions of the computer which implement | achieves the image processing apparatus of this embodiment.

Claims (12)

Parameter setting means for setting parameters used for detection of facial feature parts in a setting mode for setting reference data relating to a face and a determination mode for determining a face using the reference data, based on each mode
In the setting mode and the determination mode, detection means for detecting the position of the facial feature portion present in the image input in each mode using the parameters set in each mode;
In the setting mode and the determination mode, a calculation unit that calculates a feature amount based on the position of the feature part detected in each mode;
In the setting mode, reference data setting means for setting reference data based on the feature amount calculated in the setting mode;
In the determination mode, using the feature amount calculated in the determination mode and the reference data set in the setting mode, a determination unit that determines a face existing in the image input in the determination mode; An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the parameter includes a parameter that specifies a region in which a facial feature part is detected.   The image processing apparatus according to claim 1, wherein the setting unit sets an area in which the feature portion of the face is detected in the determination mode to be wider than the setting mode.   The image processing apparatus according to claim 1, wherein the determination unit determines a facial expression.   The facial expression recognition apparatus according to claim 4, wherein the parameter is created based on facial data of a facial expression as a target in each of the modes.   The image processing according to claim 1, further comprising a correcting unit that corrects the position of the facial feature portion detected in the determination mode based on the position of the facial feature portion detected in the setting mode. apparatus.   The image processing apparatus according to claim 6, wherein the correction unit uses a position of a feature portion of the face detected in the setting mode as an initial value of a correction value.   The image processing apparatus according to claim 1, further comprising: a mode setting unit that initializes the setting mode and changes the setting to the determination mode based on an evaluation of a feature amount calculated in the setting mode.   The image according to claim 1, wherein the detection unit includes a unit that detects a face existing in the input image, and a unit that normalizes the size or orientation of the detected face. Processing equipment. In a setting mode for setting reference data relating to a face, a step of setting a first parameter used for detection of a facial feature part;
In the setting mode, detecting a position of a facial feature portion present in the image input in the setting mode using the first parameter;
Calculating the feature amount based on the position of the feature portion detected in the setting mode in the setting mode;
In the setting mode, setting reference data based on the feature amount calculated in the setting mode;
A step of setting a second parameter independent of the first parameter as a parameter to be used for detection of a facial feature part in a determination mode for determining a face using the reference data;
In the determination mode, detecting a position of a facial feature part present in the image input in the determination mode using the second parameter;
Calculating a feature amount based on the position of the feature portion detected in the determination mode in the determination mode;
In the determination mode, using the feature amount calculated in the determination mode and the reference data set in the setting mode, a step of determining a face existing in the image input in the determination mode An image processing method.   A program for causing a computer to execute the image processing method according to claim 10.   A computer-readable storage medium storing the program according to claim 11.

Images