JP2014119879A - Face expression evaluation result smoothing device and face expression evaluation result smoothing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To stably obtain a face expression evaluation result from video data including a series of face images.SOLUTION: A smoothing processing part 80 as a face expression evaluation result smoothing device includes: a face expression intensity value acquisition part 81 for fetching a plurality of face expression intensity values obtained for each face expression type on the basis of a face image for each frame; and a face expression intensity value smoothing processing part 82 for, by referring to the face expression intensity values for a plurality of frames fetched by the face expression intensity value acquisition part 81, calculating a representative face expression intensity value corresponding to the plurality of frames on the basis of a total value based on the face expression intensity values of the plurality of frames of each face expression type.

Description

  The present invention relates to a facial expression evaluation result smoothing apparatus and a facial expression evaluation result smoothing program.

  Analyzing image data including human face images, classify facial expressions in face images into 6 types (Anger; anger, Disgust; aversion, Fear; fear, Happiness; happiness, Sadness; sadness, Surprise). A technique for calculating the intensity of the facial expression is known (for example, see Non-Patent Document 1). The information processing apparatus to which the technology described in Non-Patent Document 1 is applied obtains the strength so that the order relation of the facial expression intensities matches for a plurality of facial images with different facial expressions, and the facial expression in each facial image is obtained. The six types of facial expressions can be classified.

Peng Yang, Qingshan Liu, Dimitris N. Metaxas, "RankBoost with l1 regularization for Facial Expression Recognition and Intensity Estimation", IEEE International Conference on Computer Vision (ICCV), pp. 1018-1025, 2009

  However, when video data is supplied to the above information processing apparatus and evaluation processing of each facial image in the video data is executed, the facial expressions in the series of facial images are different from the surrounding facial expressions. May appear suddenly. Changing facial expressions on a frame-by-frame basis is a phenomenon that is unlikely to occur normally, and this suddenly appearing facial expression is likely to be an error.

  The present invention has been made to solve the above-described problem. A facial expression evaluation result smoothing apparatus and a facial expression evaluation result smoothing apparatus that stably obtain facial expression evaluation results from video data including a series of facial images. The purpose is to provide a computerized program.

  [1] In order to solve the above-described problem, a facial expression evaluation result smoothing apparatus according to an aspect of the present invention provides, for each frame, a plurality of facial expression intensity values obtained for each facial expression type based on a facial image. The facial expression intensity value acquisition unit to be captured and the facial expression intensity values for a plurality of frames captured by the facial expression intensity value acquisition unit are referred to, and a total value based on the facial expression intensity values of the plurality of frames for each facial expression type is obtained. And a facial expression intensity value smoothing processing unit that calculates representative facial expression intensity values corresponding to the plurality of frames.

[2] The facial expression evaluation result smoothing apparatus according to [1], wherein the total value is a total value of facial expression intensity values for the plurality of frames for each facial expression type. .
[3] In the facial expression evaluation result smoothing apparatus according to [1], the total value is a total value obtained by counting the number of maximum facial expression intensity values in each of the plurality of frames for each facial expression type. It is characterized by being.
[4] In the facial expression evaluation result smoothing device according to any one of [2] or [3], the facial expression intensity value smoothing processing unit performs weighting according to the position of each frame in the plurality of frames. And calculating the total value.
[5] The facial expression evaluation result smoothing device according to any one of [1] to [4], wherein the facial expression intensity value smoothing processing unit obtains the maximum value among the total values for each facial expression type. A facial expression type smoothing processing unit that selects a facial expression type corresponding to the total value as a classification result of facial expressions corresponding to the plurality of frames.
[6] In the facial expression evaluation result smoothing device according to any one of [1] to [5], the facial expression intensity value smoothing processing unit uses a number of frames smaller than the plurality of frames as a shift amount, The plurality of frames are shifted in the time direction by the shift amount.

  [7] In order to solve the above problems, a facial expression evaluation result smoothing program according to an aspect of the present invention uses a computer to calculate a plurality of facial expression intensity values obtained for each facial expression type based on a facial image, Based on the facial expression intensity value of the plurality of frames for each facial expression type, referring to the facial expression intensity value acquisition unit captured for each frame and the facial expression intensity values for a plurality of frames captured by the facial expression intensity value acquisition unit The facial expression strength value smoothing processing unit calculates a representative facial expression strength value corresponding to the plurality of frames based on the total value.

  According to the present invention, a facial expression evaluation result can be stably obtained from video data including a series of facial images.

It is a block diagram which shows the function structure of the facial expression analysis apparatus to which the facial expression evaluation result smoothing apparatus which is 1st Embodiment of this invention is applied. FIG. 3 is a block diagram showing a functional configuration of a smoothing processing unit in the same embodiment. In the embodiment, it is a figure which shows notionally a part of data structure of the face image database used when a facial expression analyzer is set to learning mode and performs a learning process. In the same embodiment, it is a figure which shows the facial expression intensity | strength teacher value used when a facial expression analysis apparatus is set to learning mode and performs a learning process by matching with face image data. FIG. 4 is a diagram schematically showing image data, face area data extracted from the image data, and normalized face area data obtained by normalizing the face area data in the embodiment. In the embodiment, the analysis area determined by the analysis area determination unit from the normalized face area data is a diagram in which lines are drawn in an easy-to-understand manner. In the embodiment, the feature amount histogram generated by the image feature amount calculation unit in the upper analysis region, the feature amount histogram in the lower analysis region, and the feature amount in the entire analysis region obtained by connecting these two histograms. It is the figure which showed typically the histogram. In the embodiment, it is the figure which showed typically one regression model in the regression analysis process which a regression analysis part performs. 4 is a flowchart illustrating a procedure of learning processing executed by the facial expression analysis apparatus in the embodiment. 5 is a flowchart illustrating a facial expression evaluation process for one frame executed by the facial expression analysis apparatus in the embodiment. In the embodiment, a series of facial expression intensity value sets obtained by the facial expression evaluation unit by the facial expression analysis apparatus set in the facial expression analysis mode fetching the evaluation video data and repeatedly executing the facial expression evaluation process It is a figure which shows an example. In the same embodiment, it is a flowchart which shows the procedure of the facial expression evaluation result smoothing process for one area which a smoothing process part performs. In the same embodiment, it is a figure which shows typically each facial expression evaluation result before and after a smoothing process part performs a facial expression evaluation result smoothing process. In 4th Embodiment of this invention, it is a figure for demonstrating the movement of the area in a smoothing process part.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.
[First Embodiment]
The facial expression analysis apparatus to which the facial expression evaluation result smoothing apparatus according to the first embodiment of the present invention is applied switches between learning processing and facial expression analysis processing by switching control. The learning process includes a pre-process. The facial expression analysis process includes a facial expression evaluation process and a facial expression evaluation result smoothing process. In the learning process, the facial expression analysis device uses the image feature amount of the facial region of each facial image data in the facial image data sequence having different degrees (intensities) of facial expressions for each facial expression type, and subjective evaluation of the evaluator. By performing a regression analysis of the correspondence with the facial expression intensity teacher value indicating the degree of facial expression, the parameter value in the regression model for each facial expression type is obtained. In the facial expression evaluation process per frame, the facial expression analysis apparatus applies the image feature amount of the facial region in the evaluation facial image data (evaluation facial image data) to the regression model learned for each facial expression type. By doing this, the facial expression strength value is calculated for each facial expression type to generate a facial expression strength value set. Here, the facial expression strength value set is a set of facial expression strength values corresponding to each of a plurality of facial expression types for one frame. In addition, in the facial expression evaluation result smoothing process for each section including the facial expression intensity value sets for a plurality of frames, the facial expression analysis apparatus performs facial expression intensity values based on the facial expression intensity value sets for the plurality of frames. And smoothing facial expression types.

  FIG. 1 is a block diagram showing a functional configuration of a facial expression analysis apparatus to which the facial expression evaluation result smoothing apparatus according to the first embodiment of the present invention is applied. As shown in the figure, the facial expression analysis apparatus 1 includes an image data acquisition unit 10, a facial expression intensity teacher value acquisition unit 20, a face region extraction unit 30, an image feature quantity calculation unit 40, and a regression analysis unit 50. A regression model storage unit 60, a facial expression evaluation unit 70, a smoothing processing unit (facial expression evaluation result smoothing device) 80, and a mode switching unit 90.

  The mode switching unit 90 switches the facial expression analysis apparatus 1 between the learning mode and the facial expression analysis mode by switching control realized by the facial expression analysis apparatus 1 executing a control program, for example. The learning mode is an operation mode in which the facial expression analysis apparatus 1 executes pre-processing and learning processing. The facial expression analysis mode is an operation mode in which the facial expression analysis apparatus 1 executes facial expression analysis processing. Note that the mode switching unit 90 may switch between the learning mode and the facial expression analysis mode, for example, according to the switching operation of the facial expression analysis apparatus 1 by the operator.

  When the facial expression analysis device 1 is set to the learning mode by the mode switching unit 90, the facial expression analysis device 1 includes the image data acquisition unit 10, the facial expression strength teacher value acquisition unit 20, and the facial region extraction unit 30. The image feature amount calculation unit 40, the regression analysis unit 50, and the regression model storage unit 60 are caused to function. Further, when the facial expression analysis device 1 is set to the facial expression analysis mode by the mode switching unit 90, the facial expression analysis device 1 includes the image data acquisition unit 10, the face region extraction unit 30, and the image feature amount calculation unit. 40, the regression model storage unit 60, the facial expression evaluation unit 70, and the smoothing processing unit 80 are caused to function.

  The image data acquisition unit 10 takes in image data supplied from an external device (not shown). Specifically, when the facial expression analysis apparatus 1 is set to the learning mode, the image data acquisition unit 10 captures a plurality of face image data from a face image database, for example. The face image database is, for example, a database that stores a set of face image data sequences with different degrees of facial expressions of a plurality of people for each type of facial expression. When the facial expression analysis apparatus 1 is set to the facial expression analysis mode, the image data acquisition unit 10 captures evaluation video data for facial expression analysis supplied by, for example, a video photographing apparatus or a video recording apparatus. . The evaluation video data includes a plurality of evaluation face image data corresponding to time-series frames.

  When the facial expression analysis apparatus 1 is set to the learning mode, the image data acquisition unit 10 supplies the captured face image data to the face area extraction unit 30. Further, when the facial expression analysis apparatus 1 is set to the facial expression analysis mode, the image data acquisition unit 10 converts the facial image data from the captured evaluation video data sequentially or every predetermined number of frames. This is supplied to the region extraction unit 30.

  When the facial expression analysis device 1 is set to the learning mode, the facial expression strength teacher value acquisition unit 20 takes in a facial expression strength teacher value supplied from an external device (not shown), and uses the facial expression strength teacher value as a regression analysis unit. 50. The external device is, for example, an information processing device such as the face image database or computer device. The facial expression strength teacher value is a value representing the degree of facial expression in each facial image data according to the subjective evaluation of the evaluator for the facial image data sequence for each facial expression type stored in the facial image database. As an example, the facial expression intensity teacher value is represented by an integer from a lower limit value (for example, “0 (zero)”) to an upper limit value (for example, “100”). In this case, the smaller the facial expression intensity teacher value, the smaller the facial expression degree, and the larger the facial expression intensity teacher value, the larger the facial expression degree. Note that the number of evaluators who evaluate the degree of facial expression may be one or more. When there are a plurality of evaluators, the average value assigned by each evaluator may be used as the facial expression intensity teacher value.

  When the facial expression analysis apparatus 1 is set to the learning mode, the facial image data captured by the image data acquisition unit 10 and the facial expression strength teacher value captured by the facial expression strength teacher value acquisition unit 20 corresponding to the facial image data. Is the teacher data in the facial expression analysis apparatus 1.

  The face area extraction unit 30 takes in the image data (face image data or evaluation face image data) supplied by the image data acquisition unit 10 and extracts a face analysis area from the image data. Specifically, the face area extraction unit 30 includes a face area detection unit 31 and an analysis area determination unit 32.

  The face area detection unit 31 captures image data supplied from the image data acquisition unit 10 and detects a face area from the image data by executing face detection processing on the image data. The face area data (face area data) is, for example, rectangular image data.

  A known face detection algorithm (for example, AdaBoost) can be applied as the algorithm of the face detection process executed by the face area detection unit 31. Known face detection algorithms include, for example, PAUL VIOLA, MICHAEL J. JONES, “Robust Real-Time Face Detection”, International Journal of Computer Vision, 2004, Vol. 57, No. 2, pp. 137-154. Details are disclosed.

  The analysis area determination unit 32 normalizes the face area data detected by the face area detection unit 31 to a predetermined size. Then, the analysis area determination unit 32 extracts an analysis area from the normalized face area data (normalized face area data). Specifically, the analysis area determination unit 32 normalizes the face area data into normalized face area data of 128 pixels in the horizontal direction × 128 pixels in the vertical direction, for example. That is, the analysis area determination unit 32 generates normalized face area data by executing image processing for enlarging or reducing the face area data into a rectangular image of a predetermined size. That is, since the size of the face included in the image data varies depending on the image data, the analysis area determination unit 32 enlarges or reduces the face area so that the resolution of the face area in all the image data is approximately the same. . Thereby, the amount of information in the face area data with different resolutions can be made substantially equal.

  The analysis region determination unit 32 determines an analysis region that is a region for calculating the image feature amount from the normalized face region data, and extracts data of the analysis region (analysis region data). The analysis region is, for example, a central circle (ellipse or perfect circle) region in the normalized face region. Then, for example, the analysis region determination unit 32 bisects the analysis region by a horizontal straight line passing through the center of the analysis region, the upper region is an upper analysis region (first analysis partial region), and the lower region is The lower analysis area (second analysis partial area) is determined. In other words, the analysis area determination unit 32 divides a circular or elliptical analysis area smaller than a circle or an ellipse inscribed in the normalized face area in the vertical (vertical) direction to determine the upper analysis area and the lower analysis area. To do.

  The image feature amount calculation unit 40 calculates the image feature amount of the analysis region data extracted by the face region extraction unit 30.

  Specifically, in the pre-processing executed by the facial expression analysis apparatus 1 set to the learning mode, the image feature amount calculation unit 40 performs, for example, Scale Invariant on the upper analysis region in the analysis region determined by the analysis region determination unit 32. A local feature quantity such as a Feature Transformation (SIFT) feature quantity or a Speeded Up Robust Feature (SURF) feature quantity is calculated. The image feature quantity calculation unit 40 generates a cluster by performing clustering processing on local feature quantities for all face image data, and stores the cluster in a storage unit that incorporates the cluster. As the clustering process, for example, a K-average method is applied. The image feature amount calculation unit 40 also generates a cluster for the lower analysis region in the same manner as the upper analysis region, and stores the cluster in the storage unit. Note that the image feature amount calculation unit 40 may acquire clusters for each of the upper analysis region and the lower analysis region for all face image data used in the learning process from an external device and store them in the storage unit.

  In the learning process executed when the facial expression analysis apparatus 1 is set to the learning mode, or the facial expression evaluation process executed when the facial expression analysis mode is set to the facial expression analysis mode, the analysis region determination unit 32 determines the image feature amount calculation unit 40. A local feature amount such as a SIFT feature amount or a SURF feature amount is calculated from the upper analysis region in the analysis region. Then, the image feature quantity calculation unit 40 classifies these local feature quantities into clusters for the upper analysis region stored in the preprocessing, and Bag-of which is a histogram in which each cluster is a bin and the number of elements of each cluster is a frequency. -Generate Keypoints. The image feature quantity calculation unit 40 generates Bag-of-Keypoints for the lower analysis area as well as the upper analysis area.

  The image feature amount calculation unit 40 generates Bag-of-Keypoints for the entire analysis region by connecting Bag-of-Keypoints for each of the upper analysis region and the lower analysis region. Specifically, the image feature amount calculation unit 40, for example, connects the 175-dimensional Bag-of-Keypoints for the upper analysis area to the 125-dimensional Bag-of-Keypoints for the lower analysis area, and has 300 dimensions as the entire analysis area. Generate Bag-of-Keypoints.

  Regarding Bag-of-Keypoints, for example, Gabriella Csurka, Christopher R. Dance, Lixin Fan, Jutta Willamowski, Gedric Bray, "Visual Categorization with Bag of Keypoints", Proc. Of ECCV Workshop on Statistical Learning in Computer Vision, Details are disclosed in pp. 59-74, 2004.

  When the facial expression analysis apparatus 1 is set to the learning mode, the image feature amount calculation unit 40 supplies Bag-of-Keypoints of the entire analysis region to the regression analysis unit 50 as an image feature amount. When the facial expression analysis apparatus 1 is set to the facial expression analysis mode, the image feature amount calculation unit 40 supplies Bag-of-Keypoints of the entire analysis region to the facial expression evaluation unit 70 as an image feature amount. .

  When the facial expression analysis device 1 is set to the learning mode, the regression analysis unit 50 takes in the image feature amount for the facial image data supplied from the image feature amount calculation unit 40, and also acquires the facial expression intensity teacher value acquisition unit. The facial expression intensity teacher value corresponding to the face image data supplied by 20 is fetched.

  The regression analysis unit 50 has a regression model by executing a regression analysis process using an image feature amount for facial image data and a facial expression intensity teacher value for each facial expression type associated with the facial image data. The parameter value is updated for each facial expression type. The regression model is a calculation means for calculating a facial expression intensity value indicating the degree of facial expression from the image feature quantity of the facial region. This regression model is a mathematical model that has variable parameters and allows parameter values to be updated. For example, in the regression analysis when the facial expression type is “anger”, the regression analysis unit 50 uses the facial expression strength teacher value itself for the facial image data with the facial expression type “angry”, while the facial expression type For face image data other than “anger”, the facial expression strength teacher value is set to “0 (zero)” and the regression process is executed. Then, the regression analysis unit 50 stores the parameter value obtained by the regression process in the regression model storage unit 60. The regression analysis unit 50 does not execute the regression analysis process for each facial expression type, but the image feature amount for the facial image data and the facial expression intensity teacher value for all facial expressions associated with the facial image data. The regression analysis process may be executed using.

  The regression model storage unit 60 stores the parameter values supplied by the regression analysis unit 50 for each facial expression type. The regression model storage unit 60 is realized by, for example, a magnetic hard disk device or a semiconductor storage device.

  When the facial expression analysis apparatus 1 is set to the facial expression analysis mode, the facial expression evaluation unit 70 captures the image feature amount for the evaluation face image data supplied from the image feature amount calculation unit 40. Further, the facial expression evaluation unit 70 reads the parameter values of the regression model for each facial expression type from the regression model storage unit 60. Then, the facial expression evaluation unit 70 generates a facial expression strength value set by calculating the facial expression strength value for each facial expression type by applying the image feature amount to each regression model, and this facial expression strength value set is generated. This is supplied to the smoothing processing unit 80. This facial expression intensity value set is a data set for evaluation face image data for one frame. Specifically, the facial expression intensity value set associates each facial expression type with a facial expression intensity value.

  When the facial expression analysis apparatus 1 is set to the facial expression analysis mode, the smoothing processing unit 80 captures and stores the facial expression intensity value set supplied by the facial expression evaluation unit 70 in time series. Then, the smoothing processing unit 80 smoothes the facial expression intensity value for each section including the facial expression intensity value set for a plurality of frames, thereby smoothing the facial expression intensity value after smoothing for each frame in the section. And facial expression type information which is the classification result of facial expressions.

Next, details of the smoothing processing unit 80 will be described.
FIG. 2 is a block diagram illustrating a functional configuration of the smoothing processing unit 80. As shown in the figure, the smoothing processing unit 80 includes a facial expression strength value acquisition unit 81, a facial expression strength value smoothing processing unit 82, and a facial expression type smoothing processing unit 83.

  The facial expression intensity value acquisition unit 81 takes a facial expression intensity value set supplied by the facial expression evaluation unit 70 in time series and stores it in a built-in buffer. This buffer is a FIFO (First In / First Out) type storage unit having a capacity capable of storing a plurality of sets of facial expression intensity value sets in a section including a plurality of frames.

  The facial expression intensity value smoothing processing unit 82 includes a plurality of facial expression intensity value acquisition units that each facial expression intensity value acquisition unit 81 captures for each section including a plurality of frames of facial expression intensity value sets stored in the buffer. With reference to the facial expression intensity value set for each frame, the representative facial expression intensity value corresponding to the plurality of frames is calculated based on the total value based on the facial expression intensity values of the plurality of frames for each facial expression type. Specifically, the facial expression intensity value smoothing processing unit 82 performs processing for a plurality of frames in the section for each section including the facial expression intensity value set for a plurality of frames stored in the buffer by the facial expression intensity value acquisition unit 81. For the facial expression strength value set, the total facial expression strength value is calculated for each facial expression type. Then, the facial expression intensity value smoothing processing unit 82 calculates an average value for each facial expression type based on the total value of the facial expression intensity values for each facial expression type. The average value here is a simple average value. The facial expression strength value smoothing processing unit 82 then calculates the maximum average value of the facial expression intensity values for each facial expression type, in other words, the maximum of the total facial expression intensity values for each facial expression type. An average value obtained from the total value is output as a smoothed representative facial expression intensity value corresponding to the section.

  The facial expression type smoothing processing unit 83 obtains the facial expression intensity value smoothing processing unit 82 for each section including the facial expression intensity value sets for a plurality of frames stored in the buffer by the facial expression intensity value acquisition unit 81. The facial expression type corresponding to the maximum average value is selected as the classification result of the smoothed facial expression corresponding to the section. The facial expression type smoothing processing unit 83 generates facial expression type information indicating the classification result of the facial expression, and outputs the facial expression type information.

  FIG. 3 is a diagram conceptually showing a part of the data structure of the face image database used when the facial expression analysis apparatus 1 is set to the learning mode and executes the learning process. As shown in the figure, the facial image database stores the facial expression in a set of facial image data sequences having different degrees of facial expression from the neutral facial expression to the peak facial expression of each person (subject) for each facial expression type. A face image data group configured by associating labels indicating types is stored. There are six types of facial expression types, for example, “anger”, “disgust”, “fear”, “happiness”, “sadness”, and “surprise”. The neutral facial expression is a neutral facial expression of a person, for example, a facial expression ranging from a person's expressionless face to a face whose degree of expression is difficult to distinguish. In other words, the neutral facial expression has a range of facial expressions. The peak facial expression is a facial expression that expresses a person's emotions abundantly, and indicates a facial expression in which emotions such as anger, disgust, fear, happiness, sadness, and surprise are strongly expressed.

  For example, Patrick Lucey, Jeffrey F. Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, "The Extended Cohn-Kanade Dataset (CK +): A complete dataset for action unit and emotion-specified expression", the Third The Cohn-Kanade Facial Expression Database described in IEEE Workshop on CVPR for Human Communicative Behavior Analysis, pp. 94-101, 2010 can be applied.

  FIG. 4 is a diagram showing the facial expression intensity teacher value used when the facial expression analysis apparatus 1 is set to the learning mode and executes the learning process, in association with the facial image data. As shown in the figure, the facial expression intensity teacher value is a subjective evaluation by the evaluator for the degree of facial expression of each facial image data for each facial image data string of each subject for each facial expression type in the facial image data group. Is represented by an integer from the lower limit “0 (zero)” to the upper limit “100”.

  In FIG. 4, for the face image data string of the first subject whose facial expression type is “happy”, the facial expression intensity teacher value corresponding to the neutral facial expression is “0 (zero)”, and the degree of facial expression increases. Accordingly, the facial expression intensity teacher value increases to, for example, “8”, “46”, “83”, and the facial expression intensity teacher value corresponding to the peak facial expression is “100”. For the face image data string of the second subject whose facial expression type is “happy”, the facial expression intensity teacher value corresponding to the neutral facial expression is “0 (zero)”, and the degree of facial expression increases. For example, the facial expression intensity teacher value increases to, for example, “6”, “52”, “79”, and the facial expression intensity teacher value corresponding to the peak facial expression is “100”. In addition, for a facial image data string whose facial expression type is “surprise”, the facial expression intensity teacher value corresponding to the neutral facial expression is “0 (zero)”, and the facial expression intensity teacher increases as the degree of facial expression increases. The value is increased to, for example, “7”, “43”, “88”, and the facial expression intensity teacher value corresponding to the peak facial expression is “100”. In addition, as in this example, it may be an indispensable condition to provide a lower limit value and an upper limit value of the facial expression intensity teacher value for the facial image sequence in which the facial expression changes from the neutral facial expression to the peak facial expression, It does not have to be an indispensable condition.

FIG. 5 is a diagram schematically showing image data, face area data extracted from the image data, and normalized face area data obtained by normalizing the face area data. That is, the figure shows the image data 2 acquired by the image data acquisition unit 10, the face region data 2a detected by the face region detection unit 31, and the normalization that the analysis region determination unit 32 normalizes (here, reduces). The face area data 2b is shown in time series. As shown in the figure, the image data 2 is an image including the upper side of the person's neck. The face area data 2a is an image including a face extracted from the image data 2. An image including a face is, for example, an image including main parts (both eyebrows, both eyes, nose, mouth) of a face that determine the facial expression of a person. The normalized face area data 2b is an image obtained by normalizing the face area data 2a to a horizontal pixel number L _X × vertical pixel number _LY size. The relationship between the horizontal pixel number L _X and the vertical pixel number L _Y is, for example, a relationship in which the normalized face area is a square.

FIG. 6 is a diagram in which the analysis region determined by the analysis region determination unit 32 from the normalized face region data 2b is drawn in a line that is visually easy to understand. As shown in the figure, the analysis region determination unit 32, about the central position of the horizontal pixel number L _X × vertical pixel number L _Y of the normalized face region data 2b, circular included in the normalized face region data 2b The analysis area 3 is determined. Horizontal diameter of the analysis region 3, for example, a 0.8 times larger than the horizontal pixel number L _X, the diameter of the vertical direction, for example, has a 0.8 times the size of the vertical pixel number L _Y . In this way, by making the diameter of the analysis region 3 smaller than the diameter of the inscribed circle of the normalized face region data 2b, information on hair, ears, earrings, etc. that are less important for face recognition and facial expression recognition are obtained. Can be excluded. The analysis region determination unit 32 divides the analysis region 3 into an upper analysis region 3U and a lower analysis region 3D by a horizontal line passing through the center of the analysis region 3. By dividing in this way, the upper analysis region 3U includes both eyebrows and both eyes, and the lower analysis region 3D includes the nasal head and mouth.

  FIG. 7 shows a feature amount histogram generated by the image feature amount calculation unit 40 in the upper analysis region, a feature amount histogram in the lower analysis region, and a feature amount in the entire analysis region in which these two histograms are connected. It is the figure which showed typically the histogram. This figure is an example in which a histogram of feature amounts in the lower analysis region is connected to a histogram of feature amounts in the upper analysis region. As described above, the image feature quantity calculation unit 40 can add position information to Bag-of-Keypoints by generating and connecting a histogram in each divided area. Note that the image feature amount calculation unit 40 may generate a feature amount histogram in the entire analysis region by concatenating the feature amount histogram in the upper analysis region after the feature amount histogram in the lower analysis region.

  FIG. 8 is a diagram schematically showing one regression model in the regression analysis process executed by the regression analysis unit 50. In the figure, the horizontal axis represents an independent variable in the regression equation, and in this embodiment, represents the image feature amount of the face area of the face image data. The vertical axis represents the dependent variable in the regression equation, and in this embodiment represents the facial expression intensity teacher value. The distribution of a plurality of square marks in the figure shows the correspondence between the image feature quantity and the facial expression intensity teacher value for this image feature quantity. Moreover, the solid line represented by the curve in the same figure is a graph which shows the regression type obtained by the regression analysis process which the regression analysis part 50 performs.

  The regression analysis unit 50 executes a regression analysis process by applying, for example, a linear regression model, a logistic regression model, or a support vector regression model as a regression model. Next, a regression analysis process using each regression model will be described.

[1] Linear Regression Model When a linear regression model is applied as the regression model, the regression analysis unit 50 represents the relationship between the image feature amount and the facial expression intensity teacher value as the linear regression analysis process as shown in the following equation (1). Model as a product-sum function. Here, Y is a facial expression intensity teacher value, and X _i is an image feature amount (i = 1,..., I). Α and β _i are parameters.

The regression analysis unit 50 estimates the sum-of-products function shown in Expression (1) by regressing the correspondence between the image feature quantity and the facial expression intensity teacher value for the image feature quantity by, for example, the least square method. Specifically, the regression analysis unit 50 calculates the sum of squares of the approximation errors so that the formula (1) is an optimal approximation formula for the paired data of the image feature quantity and the facial expression intensity teacher value for the image feature quantity. The parameters α and β _i that minimize the value are obtained by the steepest descent method, for example. The regression analysis unit 50 may eliminate or suppress multiple collinearity by deleting one of the independent variables having strong correlation (for example, the correlation coefficient is “0.5” or more) in the regression analysis processing. . For the independent variable (image feature amount) that is “0 (zero)” in common for all facial expression types, the regression analysis unit 50 may perform processing for deleting the independent variable.

[2] Logistic Regression Model When a logistic regression model is applied as the regression model, the regression analysis unit 50 represents the relationship between the image feature quantity and the facial expression intensity teacher value as the logistic regression analysis process as shown in the following equation (2). Model into a function. Here, Y is a facial expression intensity teacher value, and X _i is an image feature amount (i = 1,..., I). Α and β _i are parameters.

The regression analysis unit 50 obtains the parameters α and β _i by regressing the correspondence between the image feature quantity and the facial expression intensity teacher value for the image feature quantity. By applying this logistic regression model, the regression analysis unit 50 can obtain a regression equation in which the facial expression intensity teacher value Y with respect to the image feature amount X _i falls within a range from 0 to 100 (0 ≦ Y ≦ 100). it can.

[3] Support Vector Regression Model When the support vector regression model is applied as the regression model, the regression analysis unit 50 performs the image feature quantity X _i (i = _i ) in the form of the following formula (3) as the support vector regression analysis process. 1,..., I) and the facial expression intensity teacher value Y are related.

In equation (3), the function φ is a mapping function that maps an I-dimensional feature vector into a J-dimensional vector (row vector). This support vector regression model uses a kernel trick with a function φ. β _j (j = 1,..., J) is a weighting coefficient corresponding to each element of the vector after mapping by the function φ. Α is a bias term. Regression analysis unit 50 performs a regression in the form of equation (3) using a number of facial expression intensity teaching value inputted, the parameter alpha, beta _1, · · ·, seek beta _J. For parameter calculation itself, for example, an existing support vector regression learning method can be applied based on the Newton method.

Next, the operation of the facial expression analysis apparatus 1 will be described.
First, the facial expression analysis apparatus 1 set to the learning mode fetches all face image data used in the learning process from the face image database, and executes the following pre-processing. That is, the image data acquisition unit 10 captures face image data from the face image database. Next, the face area extraction unit 30 normalizes the size of the captured face image data and extracts analysis areas (upper analysis area and lower analysis area). Next, the image feature amount calculation unit 40 calculates a local feature amount such as a SIFT feature amount or a SURF feature amount for the upper analysis region. Next, the image feature amount calculation unit 40 generates a cluster by executing clustering processing on the local feature amount for the upper analysis region of all the face image data, and stores the cluster in the storage unit. Further, the image feature quantity calculation unit 40 also generates a cluster for the lower analysis region as in the upper analysis region, and stores the cluster in the storage unit.

Next, the learning process of the facial expression analysis apparatus 1 will be described.
FIG. 9 is a flowchart showing the procedure of the learning process executed by the facial expression analysis apparatus 1.
In step S <b> 1, for example, the image data acquisition unit 10 takes in one face image data from a plurality of face image data stored in the face image database, and supplies the face image data to the face region extraction unit 30.
Next, in step S2, the facial expression intensity teacher value acquisition unit 20 sets the facial expression intensity teacher value corresponding to the facial image data captured by the image data acquisition unit 10 in the process of step S1 to an external device (for example, face The facial expression intensity teacher value is fetched from the image database) and supplied to the regression analysis unit 50.

  Next, in step S <b> 3, the face area extraction unit 30 takes in the face image data supplied from the image data acquisition unit 10 and executes face detection processing on the face image data to thereby determine the human face from the face image data. Detect face area. Next, the analysis area determination unit 32 normalizes the face area data detected by the face area detection unit 31 to a predetermined size (for example, 128 pixels in the horizontal direction × 128 pixels in the vertical direction). Next, the analysis region determination unit 32 extracts an analysis region from the normalized face region data, and determines two analysis partial regions (an upper analysis region and a lower analysis region) from this analysis region.

  Next, in step S <b> 4, the image feature amount calculation unit 40 calculates the image feature amount of the analysis region data extracted by the face region extraction unit 30. Specifically, the image feature amount calculation unit 40 calculates a local feature amount such as a SIFT feature amount or a SURF feature amount from the upper analysis region. Next, the image feature quantity calculation unit 40 classifies these local feature quantities into clusters for the upper analysis region stored in the preprocessing, and generates a histogram with each cluster as a bin and the number of elements in each cluster as a frequency. Further, the image feature quantity calculation unit 40 calculates a local feature quantity such as SIFT feature quantity or SURF from the lower analysis region. Next, the image feature quantity calculation unit 40 classifies these local feature quantities into clusters for the lower analysis region stored in the preprocessing, and generates a histogram with each cluster as a bin and the number of elements of each cluster as a frequency. Next, the image feature amount calculation unit 40 generates a histogram of the entire analysis region, in other words, Bag-of-Keypoints of the entire analysis region by connecting the histograms for the upper analysis region and the lower analysis region. Next, the image feature amount calculation unit 40 supplies Bag-of-Keypoints of the entire analysis region to the regression analysis unit 50 as an image feature amount.

  Next, in step S5, when the acquisition of all the face image data to be acquired from the face image database is completed (step S5: YES), the facial expression analysis apparatus 1 proceeds to the process of step S6. On the other hand, when all the facial image data to be captured from the facial image database has not been captured (step S5: NO), the facial expression analysis apparatus 1 returns to the process of step S1.

In step S6, the regression analysis unit 50 executes the regression analysis process by using the image feature amount for the facial image data and the facial expression intensity teacher value for each facial expression type associated with the facial image data. The parameter value of the regression model is updated for each facial expression type. Next, the regression analysis unit 50 supplies parameter values obtained by performing the regression process to the regression model storage unit 60.
Next, in step S7, the regression model storage unit 60 stores the parameter values supplied by the regression analysis unit 50 for each facial expression type.

FIG. 10 is a flowchart illustrating a procedure of facial expression evaluation processing for one frame executed by the facial expression analysis apparatus 1.
In step S <b> 21, the image data acquisition unit 10 takes in evaluation face image data for facial expression analysis supplied by, for example, a video shooting device or video recording device, and supplies this evaluation face image data to the face region extraction unit 30. To do.

  Next, in step S22, the face area extraction unit 30 takes in the evaluation face image data supplied by the image data acquisition unit 10 and executes face detection processing on the evaluation face image data to thereby evaluate the face image data. The face area of a person is detected from the above. Next, the analysis area determination unit 32 normalizes the evaluation face area data detected by the face area detection unit 31 to a predetermined size (for example, 128 pixels in the horizontal direction × 128 pixels in the vertical direction). Next, the analysis region determination unit 32 extracts an analysis region from the normalized face region data, and determines two analysis partial regions (an upper analysis region and a lower analysis region) from this analysis region.

  Next, in step S23, the image feature amount calculation unit 40 calculates the image feature amount of the analysis region data extracted by the face region extraction unit 30. For example, the image feature amount calculation unit 40 uses the histogram for the data of the upper analysis region and the lower analysis region in the analysis region determined by the analysis region determination unit 32 as in the process of step S4 (see FIG. 9) in the learning process. Calculate Next, the image feature quantity calculation unit 40 generates a histogram of the entire analysis region, that is, Bag-of-Keypoints of the entire analysis region by connecting the histograms for the upper analysis region and the lower analysis region. Next, the image feature amount calculation unit 40 supplies Bag-of-Keypoints of the entire analysis region to the facial expression evaluation unit 70 as an image feature amount.

  Next, in step S <b> 24, the facial expression evaluation unit 70 takes in the image feature amount for the evaluation face image data supplied by the image feature amount calculation unit 40. Next, the facial expression evaluation unit 70 reads the regression model parameter values from the regression model storage unit 60 for each facial expression type. Next, the facial expression evaluation unit 70 generates a facial expression strength value set by calculating the facial expression strength value for each facial expression type by applying the image feature amount to each regression model.

  FIG. 11 shows a series of facial expression intensity values obtained by the facial expression evaluation unit 70 when the facial expression analysis apparatus 1 set in the facial expression analysis mode takes in the evaluation video data and repeatedly executes the facial expression evaluation process. It is a figure which shows an example of a set. In the figure, the shaded facial expression intensity value is a facial expression intensity value set, that is, the maximum facial expression intensity value (maximum facial expression intensity value) in each frame. According to the figure, in 8 frames from time (t-3) to time (t + 4), time (t-3), (t-2), (t-1), (t + 1), (t + 2), The facial expression type (representative facial expression type) corresponding to the maximum facial expression intensity value in each of the (t + 3) and (t + 4) facial expression intensity value sets is “sadness”. On the other hand, the representative facial expression type corresponding to the maximum facial expression intensity value in the facial expression intensity value set at time t is “happy”. The facial expression evaluation result smoothing process executed by the smoothing processing unit 80 when six frames from time (t−2) to time (t + 3) are defined as one section will be described with reference to FIG.

  In the smoothing processing unit 80, the facial expression intensity value acquisition unit 81 takes the facial expression intensity value set supplied by the facial expression evaluation unit 70 in time series and stores it in a built-in buffer. That is, the facial expression intensity value acquisition unit 81 stores the facial expression intensity value set for six frames from time (t−2) to time (t + 3) in the buffer.

  When the facial expression intensity value set for six frames is stored in the buffer of the facial expression intensity value acquisition unit 81, the facial expression intensity value smoothing processing unit 82 stores the facial expression intensity value for six frames stored in the buffer. For the set, the average value of the facial expression intensity values in the section is calculated for each facial expression type.

  Specifically, the facial expression intensity value smoothing processing unit 82 calculates the average value of six facial expression intensity values whose facial expression type is “anger” {(6.1 + 2.4 + 3.2 + 3.5 + 4) in the section. .1 + 5.2) / 6} to obtain the average value “4.1” of the facial expression intensity for the facial expression type “anger”. In addition, the facial expression intensity value smoothing processing unit 82 averages {(2.7 + 3.3 + 2.4 + 2.1 + 0.8 + 0) of six facial expression intensity values whose facial expression type is “disgust” in the section. .1) / 6} to obtain an average value “1.9” of the facial expression intensity for the facial expression type “disgust”. In addition, the facial expression intensity value smoothing processing unit 82 calculates an average value of six facial expression intensity values whose facial expression type is “fear” {(8.9 + 11.1 + 5.2 + 7.8 + 2.3 + 1) in the section. .7) / 6} to obtain the average value “6.2” of the facial expression intensity for the facial expression type “fear”. Also, the facial expression intensity value smoothing processing unit 82 averages {(18.8 + 48.3 + 78.2 + 25.5 + 60.2 + 40) of the six facial expression intensity values whose facial expression type is “happy” in the section. .1) / 6} to obtain an average value “45.2” of the facial expression intensity for the facial expression type “happy”. In addition, the facial expression intensity value smoothing processing unit 82 averages six facial expression intensity values {(68.3 + 70.1 + 72.3 + 74.5 + 72.2 + 74) whose facial expression type is “sadness” in the section. .5) / 6} to obtain an average value “72.0” of the facial expression intensity for the facial expression type “sadness”. Further, the facial expression intensity value smoothing processing unit 82 averages {(1.8 + 9.2 + 12.8 + 6.5 + 2.1 + 4) of the six facial expression intensity values whose facial expression type is “surprise” in the section. .4) / 6} to obtain an average value “6.1” of the facial expression intensity for the facial expression type “surprise”.

  Then, the facial expression intensity value smoothing processing unit 82 sets “72.0”, which is the maximum average value among the average values of the facial expression intensity values for each of the six types of facial expression types, at time t in the section. The corresponding representative facial expression intensity value after smoothing is output.

In other words, the smoothing processing unit 80, by the following equation (4), to calculate a representative facial expression intensity values I _t at time t in the section. Where e is the facial expression type (eg, “Anger”, “Disgust”, “Fear”, “Happiness”, “Sadness”, and “Surprise ( "Surprise)"). I _eT is a facial expression intensity value corresponding to the facial expression type e at time T ((t−m) ≦ T ≦ (t + n)). Therefore, the sum of the facial expression intensity values _IeT from time (t−m) to time (t + n) is the total value. When FIG. 11 is applied, m is “2” and n is “3”.

  The facial expression type smoothing processing unit 83 corresponds to the maximum average value “72.0” of the average facial expression intensity values for each facial expression type obtained by the facial expression intensity value smoothing processing unit 82. The facial expression type “sadness” to be selected is selected as the classification result of the smoothed facial expression corresponding to time t in the section. Then, the facial expression type smoothing processing unit 83 generates facial expression type information indicating “sadness”, which is the classification result of the facial expression, and outputs the facial expression type information.

That is, the smoothing processing unit 80 calculates the facial expression type information E _t at the time t in the section by the following equation (5).

  In this way, the smoothing processing unit 80 uses the facial expression intensity value set for each facial expression type in this section as a reliability by using a plurality of facial expression intensity value sets in one section, that is, the facial expression in that section. Smooth the evaluation results. Therefore, the smoothing processing unit 80 can obtain highly reliable facial expression intensity values and facial expression type information within the section.

FIG. 12 is a flowchart showing the procedure of the facial expression evaluation result smoothing process for one section executed by the smoothing processing unit 80.
In step S41, the facial expression intensity value acquisition unit 81 takes in the facial expression intensity value set for one frame supplied by the facial expression evaluation unit 70, and stores the facial expression intensity value set in a buffer that incorporates the facial expression intensity value set.

  Next, in step S42, when the facial expression intensity value acquisition unit 81 stores the facial expression intensity value set for one section in the buffer (step S42: YES), the process moves to the process of step S43, and one section is stored in the buffer. If the facial expression intensity value set for minutes is not stored (step S42: NO), the process returns to step S41.

  In step S43, the facial expression intensity value smoothing processing unit 82 sets the facial expression intensity value set for six frames stored in the buffer of the facial expression intensity value acquisition unit 81 for each facial expression type for each facial expression type. Calculate the average value of intensity values. Next, the facial expression intensity value smoothing processing unit 82 outputs the maximum average value of the facial expression intensity values for each facial expression type as the representative facial expression intensity value after smoothing corresponding to the section. To do.

  Next, in step S44, the facial expression type smoothing processing unit 83 determines the facial expression corresponding to the maximum value of the average facial expression intensity value for each facial expression type obtained by the facial expression intensity value smoothing processing unit 82. The type is selected as the classification result of the smoothed facial expression corresponding to each frame in the section. Next, the facial expression type smoothing processing unit 83 generates facial expression type information indicating the classification result of the facial expression, and outputs the facial expression type information.

  FIG. 13 is a diagram schematically illustrating the facial expression evaluation results before and after the smoothing processing unit 80 performs the facial expression evaluation result smoothing process. The upper graph in the figure is a graph showing the facial expression evaluation results in time series before the smoothing processing unit 80 executes the facial expression evaluation result smoothing process. In the upper graph, the horizontal axis is the time axis, and the vertical axis is the facial expression intensity maximum value in the facial expression intensity value set. As shown in the upper graph, the facial expression intensity maximum value before the smoothing processing unit 80 executes the facial expression evaluation result smoothing process varies with time.

  In addition, Δ, ○, and □ symbols (referred to as facial expression symbols) shown immediately below the upper graph are symbols representing facial expressions represented by the representative facial expression types corresponding to the maximum facial expression intensity values, and the time axis of the graph Are shown in association with each other. Here, Δ is a symbol indicating “happy”, ○ is “surprise”, and □ is “angry”. According to a series of facial expression symbols immediately below the upper graph, suddenly “surprise” and “angry” facial expressions appear in the facial expression indicating “happiness”.

On the other hand, the lower graph in FIG. 13 is a graph showing the facial expression evaluation results in time series after the smoothing processing unit 80 executes the facial expression evaluation result smoothing process. In the lower graph, the horizontal axis is the time axis, and the vertical axis is the smoothed facial expression intensity value. As shown in the lower graph, the facial expression strength value after the smoothing processing unit 80 executes the facial expression evaluation result smoothing process is every plural frames (10 frames in the figure), that is, time T ₁ , T ₂ , T ₃ , T ₄ ,..., The facial expression intensity values have no variation. In addition, according to the series of facial expression symbols immediately below the lower graph, the results of stable facial expression classification are shown without sudden facial expressions appearing.

  As described above, the smoothing processing unit 80 smoothes the facial expression evaluation result for each section including a plurality of frames by using the facial expression intensity value set for the plurality of frames, thereby generating an error in the facial expression evaluation result. Can be obtained, and a stable facial expression strength value and a facial expression classification result can be obtained.

[Second Embodiment]
In the first embodiment, the facial expression strength value smoothing processing unit 82 in the smoothing processing unit 80 sets the average value of facial expression strength values for each facial expression type for a set of facial expression strength values for a plurality of frames in one section. (Simple average value) was calculated. On the other hand, in the second embodiment of the present invention, the facial expression intensity value smoothing processing unit performs a weighted average of facial expression intensity values for each facial expression type for a set of facial expression intensity values for a plurality of frames in one section. Calculate the value. Since the configuration of the facial expression analysis apparatus in the second embodiment is the same as the configuration of the facial expression analysis apparatus 1 in the first embodiment, the functions different from those in the first embodiment will be described with reference to FIGS. Only explained.

  In the second embodiment, the facial expression intensity value smoothing processing unit 82 performs a plurality of sections in the section including the facial expression intensity value sets for a plurality of frames stored in the buffer by the facial expression intensity value acquisition unit 81. For the facial expression intensity value set for the frame, the total value of the facial expression intensity values weighted according to the frame position is calculated for each facial expression type. Then, the facial expression intensity value smoothing processing unit 82 calculates an average value for each facial expression type based on the total weighted facial expression intensity value for each facial expression type. That is, this average value is a weighted average value. The weighting coefficient (weighting coefficient) for each frame is a value that monotonously increases or decreases monotonically in a series of frames included in one section, or monotonously decreases after increasing monotonously. For example, taking one section shown in FIG. 11 as an example, a weighting factor that increases as time t (from time -2) approaches time t, and a weighting factor that decreases as time t (from time t + 3) approaches time t. , The weighting coefficient for each of the six frames. Specifically, for example, “0.05” for time (t−2), “0.2” for time (t−1), “0.4” for time t, time (t + 1) ) For the time (t + 2), and “0.05” for the time (t + 3).

  Then, the facial expression intensity value smoothing processing unit 82 weights the maximum weighted average value among the weighted average values of the facial expression intensity values for each facial expression type, in other words, the facial expression intensity values for each facial expression type. An average value obtained from the maximum total value among the total values is output as a smoothed representative facial expression intensity value corresponding to time t in the section.

  By configuring in this way, the facial expression intensity value smoothing processing unit 82 can smooth the facial expression intensity value by giving a high reliability to a predetermined frame in one section and a frame close to this frame.

  Further, in the second embodiment, the facial expression type smoothing processing unit 83 performs the facial expression intensity value for each section including the facial expression intensity value set for a plurality of frames stored in the buffer by the facial expression intensity value acquisition unit 81. The facial expression type corresponding to the maximum value in the weighted average value obtained by the smoothing processing unit 82 is selected as a classification result of the smoothed facial expression corresponding to each frame in the section. The facial expression type smoothing processing unit 83 generates facial expression type information indicating the classification result of the facial expression, and outputs the facial expression type information.

[Third Embodiment]
In the first embodiment and the second embodiment, the facial expression strength value smoothing processing unit 82 in the smoothing processing unit 80 sets the facial expression for each facial expression type for the facial expression strength value set for a plurality of frames in one section. The average value of intensity values (simple average value, weighted average value) was calculated, and the maximum average value was used as the representative facial expression intensity value. On the other hand, in the third embodiment of the present invention, the facial expression intensity value smoothing processing unit represents the number of facial expression intensity maximum values in the facial expression intensity value set for a plurality of frames in the section for each section. Based on counting for each facial expression type, a representative facial expression intensity value is obtained. Since the configuration of the facial expression analysis apparatus in the third embodiment is the same as the configuration of the facial expression analysis apparatus 1 in the first embodiment, the functions different from those in the first embodiment will be described with reference to FIGS. Only explained.

  In the third embodiment, the facial expression intensity value smoothing processing unit 82 refers to the facial expression intensity value for one section stored in the buffer, and for each representative facial expression type, the facial expression intensity of the representative facial expression type. The number of frames when the value becomes the maximum value (maximum facial expression intensity value) in the frame is counted over the relevant section. Then, the facial expression intensity value smoothing processing unit 82 sets the count result as the total value for the representative facial expression type. Then, the facial expression intensity value smoothing processing unit 82 calculates a simple average value of the facial expression intensity maximum values in the section for the representative facial expression type corresponding to the maximum total value among the total values for each representative facial expression type. This simple average value is used as the representative facial expression intensity value in the section.

  In addition, the facial expression type smoothing processing unit 83 selects the facial expression type corresponding to the maximum total value obtained by the facial expression intensity value smoothing processing unit 82 as the classification result of the smoothed facial expression corresponding to the section. Elected as.

  Specifically, taking the facial expression intensity value for one section shown in FIG. 11 as an example, the facial expression intensity value smoothing processing unit 82 has the maximum facial expression intensity in the facial expression intensity values for six frames in the section. “Happy” and “sadness” are extracted as the facial expression types (representative facial expression types) corresponding to the values (shaded numerical values). Then, the facial expression intensity value smoothing processing unit 82 counts the number of maximum facial expression intensity values for each of the extracted representative facial expression types, and adds the total value “1” to the representative facial expression type “happy”. A total value “5” is obtained for the expression type “sadness”. Then, the facial expression intensity value smoothing processing unit 82 calculates the average value of facial expression intensity maximum values in the section for the representative facial expression type “sadness” corresponding to the maximum total value “5” of the total values “( 68.3 + 70.1 + 72.3 + 74.5 + 72.2 + 74.5) /6=72.0 ”, and this average value“ 72.0 ”is set as the representative facial expression intensity value in the section. Further, the facial expression type smoothing processing unit 83 sets the facial expression type “sadness” corresponding to the maximum total value “5” obtained by the facial expression intensity value smoothing processing unit 82 to the post-smoothing corresponding to the section. Selected as a result of classification of facial expressions.

[Fourth Embodiment]
In the third embodiment, the facial expression intensity value smoothing processing unit 82 in the smoothing processing unit 80 counts the number of facial expression intensity maximum values in the facial expression intensity value set in one section for each representative facial expression type. The representative facial expression intensity value was obtained. In contrast, in the fourth embodiment of the present invention, the facial expression intensity value smoothing processing unit weights the number of facial expression intensity maximum values in the facial expression intensity value set in one section according to the position of the frame. The representative facial expression intensity value is obtained by counting for each representative facial expression type. Since the configuration of the facial expression analysis apparatus in the fourth embodiment is the same as the configuration of the facial expression analysis apparatus 1 in the first embodiment, the functions different from those in the first embodiment will be described with reference to FIGS. Only explained.

  In the fourth embodiment, the facial expression intensity value smoothing processing unit 82 performs a plurality of sections in the section including the facial expression intensity value sets for a plurality of frames stored in the buffer by the facial expression intensity value acquisition unit 81. The number of facial expression intensity maximum values in the frame facial expression intensity value set is weighted according to the position of the frame, and the weighted number is counted for each facial expression type. The weighting value (weight) according to the position of each frame is a value that monotonously increases or monotonically decreases in a series of frames included in one section, or monotonously decreases after increasing monotonously. For example, taking one section shown in FIG. 11 as an example, a weight that increases from time (t−2) to time t and a weight that decreases from time t to time (t + 3) The weight for each frame. Specifically, for example, “1” for time (t−2), “2” for time (t−1), “4” for time t, and “3” for time (t + 1). ”,“ 2 ”for time (t + 2), and“ 1 ”for time (t + 3). Then, the facial expression intensity value smoothing processing unit 82 sets the maximum total value among the total number of the numbers as the representative facial expression intensity value.

  In addition, the facial expression type smoothing processing unit 83 selects the facial expression type corresponding to the maximum total value obtained by the facial expression intensity value smoothing processing unit 82 as the classification result of the smoothed facial expression corresponding to the section. Elected as.

  Specifically, the above weights are taken as an example, and the facial expression intensity values for one section shown in FIG. 11 are taken as an example. The facial expression intensity value smoothing processing unit 82 extracts “happy” and “sadness” as the representative facial expression types corresponding to the maximum facial expression intensity values in the facial expression intensity values for six frames in the section. Then, the facial expression intensity value smoothing processing unit 82 weights each extracted representative facial expression type according to the position of the frame, and counts the number of maximum facial expression intensity values. That is, the facial expression intensity value smoothing processing unit 82 obtains a total value “1 × 4 = 4” for the representative facial expression type “happy”. Further, the facial expression intensity value smoothing processing unit 82 obtains a total value “1 × 1 + 1 × 2 + 1 × 3 + 1 × 2 + 1 × 1 = 9” for the representative facial expression type “sadness”. The facial expression intensity value smoothing processing unit 82 then calculates a weighted average value of facial expression intensity values in the section for the representative facial expression type “sadness” corresponding to the maximum total value “9” among the total values “( 1 × 68.3 + 2 × 70.1 + 4 × 72.3 + 3 × 74.5 + 2 × 72.2 + 1 × 74.5) /13=72.3 ”, and this weighted average value“ 72.3 ”is represented in the section. The facial expression intensity value is used. Further, the facial expression type smoothing processing unit 83 converts the facial expression type “sadness” corresponding to the maximum total value “9” obtained by the facial expression intensity value smoothing processing unit 82 into the post-smoothing corresponding to the section. Selected as a result of classification of facial expressions.

[Fifth Embodiment]
In the first embodiment, the smoothing processing unit 80 acquires the facial expression intensity value and facial expression type information smoothed for each section. On the other hand, in the fifth embodiment of the present invention, the facial expression intensity value and the facial expression type information are obtained while shifting the above-described section in the time direction. Since the configuration of the facial expression analysis apparatus in the fifth embodiment is the same as the configuration of the facial expression analysis apparatus 1 in the first embodiment, refer to FIG. 1 and FIG. Only explained.

  In the fifth embodiment, the facial expression intensity value smoothing processing unit 82 sets the number of frames smaller than a plurality of frames included in one section as a shift amount (shift amount), and shifts the section in the time direction by the shift amount. For example, the facial expression intensity value smoothing processing unit 82 sets the shift amount to one frame, and shifts the section in the time direction by one frame. The facial expression intensity value smoothing processing unit 82 calculates an average value of facial expression intensity values for each facial expression type with respect to the facial expression intensity value set in the section for each shifted section. Then, the facial expression intensity value smoothing processing unit 82 outputs the maximum average value of the facial expression intensity values for each facial expression type as the smoothed representative facial expression intensity value corresponding to the section. .

  In the fifth embodiment, the facial expression type smoothing processing unit 83 corresponds to the smoothed facial expression intensity value obtained by the facial expression intensity value smoothing processing unit 82 in the section shifted by the shift amount. The facial expression type is selected as the classification result of the smoothed facial expression corresponding to each frame in the section. The facial expression type smoothing processing unit 83 generates facial expression type information indicating the classification result of the facial expression, and outputs the facial expression type information.

  Note that the processing of shifting the section in the time direction by the shift amount by the facial expression intensity value smoothing processing unit 82 can be applied to any of the second to fourth embodiments.

FIG. 14 is a diagram for explaining the movement of the section in the smoothing processing unit 80. In each graph in the figure, the horizontal axis is the time axis, and the vertical axis is the facial expression intensity value. Time t ₁ , time t ₂ , and time t ₃ are times for successive frames. That is, time t ₂ is a time corresponding to the next frame of the frame at time t ₁ , and time t ₃ is a time corresponding to the next frame of the frame at time t ₂ . The time (t _p + t _f ) is one section. Therefore, the upper in the figure, middle, and lower stage of the graph, one frame as a shift amount, time, (t 1 _{-t p)} from the time _(t 1 ₊ t _f) how shifted sequentially time direction section up Show.

  According to the fifth embodiment, the smoothing processing unit 80 can output a facial expression intensity value with improved reliability by suppressing variation in one section for each shift amount in the time direction.

[Other embodiments]
In addition to obtaining Bag-of-Keypoints as the image feature amount, the image feature amount calculation unit 40 in the first to fourth embodiments of the present invention described above includes, for example, a local binary pattern (Local Binary Patterns; LBP), Alternatively, an extended local binary pattern (extended LBP) may be obtained.

  The local binary pattern is binarized by comparing the target pixel scanned and selected in the analysis region by the image feature amount calculation unit 40 and peripheral pixels (for example, eight adjacent pixels) of the target pixel. The binary pattern obtained by doing this is used as the image feature amount. The image feature amount calculation unit 40 may sequentially scan each pixel included in the analysis region as a target pixel, or may scan discretely at a predetermined number of pixel intervals.

  For local binary patterns, see, for example, Timo Ojala, Matti Pietikainen, Senior Member, IEEE and Topi Maenpaa, "Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24 , no. 7, July 2002, details are disclosed.

  Specifically, the analysis region determination unit 32 of the face region extraction unit 30 divides the normalized face region data into a grid (for example, eight divisions in the horizontal direction and the vertical direction). That is, the analysis area determination unit 32 sets each divided block data of the normalized face area data as analysis area data. The image feature quantity calculation unit 40 calculates LBP for each divided block data, for example, for each pixel. Then, the image feature amount calculation unit 40 generates a histogram in which all LBP patterns are bins and the frequency of output of each pattern is a frequency. Then, the image feature amount calculation unit 40 uses a connected histogram obtained by connecting the histograms of the respective divided blocks as the face image feature amount.

  Further, the extended local binary pattern is a binary pattern obtained by extending the local binary pattern in the time series direction as a feature amount. That is, the extended local binary pattern is a feature quantity useful when the facial expression analysis apparatus 1 captures evaluation video data. The image feature amount calculation unit 40, when obtaining the feature amount of the current evaluation face image data included in the evaluation video data as a local binary pattern, evaluates past and future evaluations from the current evaluation face image data and the current evaluation face image data. The result of pixel comparison with face image data is also included in the binary pattern.

  Regarding extended LBP, for example, Guoying Zhao, Matti Pietikainen, "Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions", IEEE Transactions on Patterns Analysis and Machine Intelligence, vol. 29, no. 6, June 2007, Details are disclosed.

  When the parameter value is obtained by applying either the linear regression model or the support vector regression model to the regression analysis unit 50, the facial expression intensity value output from the facial expression evaluation unit 70 is a lower limit value (for example, “0”). (Zero) ") to the upper limit (for example," 100 ") may not fit. Therefore, when applying either the linear regression model or the support vector regression model to the regression analysis unit 50, the facial expression evaluation unit 70 determines that the calculated facial expression intensity value is less than “0 (zero)”. When the value exceeds (zero) ”or“ 100 ”, the facial expression strength value may be output as“ 100 ”.

Alternatively, the facial expression analysis apparatus 1 in which either the linear regression model or the support vector regression model is applied to the regression analysis unit 50 includes a face recognition processing unit that performs face recognition processing on each evaluation face image data and identifies a person. Further, it may be provided. In this case, when the face display analysis device 1 is set to the facial expression analysis mode, the facial expression evaluation unit 70 determines the maximum facial expression intensity value int _max for each person recognized by the facial recognition processing unit in a predetermined period. Using the minimum value int _min , the facial expression intensity value int may be normalized to a value int ′ within a range from 0 to 100 by the following equation (6).

  In the first to fourth embodiments, linear regression analysis processing, logistic regression analysis processing, and support vector regression analysis processing are shown as the regression analysis processing executed by the regression analysis unit 50. The regression analysis process executed by the regression analysis unit 50 is not limited to these examples, and other regression analysis processes can also be applied. For example, the regression analysis unit 50 may apply a learning process using a neural network to the regression analysis process.

  Further, in the first to fifth embodiments, the facial expression analysis apparatus 1 may be an apparatus that operates only in the facial expression analysis mode. Specifically, the facial expression intensity teacher value acquisition unit 20, the regression analysis unit 50, and the mode switching unit 90 may be deleted from the facial expression analysis apparatus 1, and only the facial expression analysis processing may be performed. In this case, the regression model storage unit 60 stores parameter values optimized for each facial expression type in advance.

  Moreover, you may make it implement | achieve the one part function of the facial expression analysis apparatus 1 in each embodiment mentioned above, for example, the function of the smoothing process part 80, with a computer. In this case, a facial expression evaluation result smoothing program for realizing the function is recorded on a computer-readable recording medium, and the facial expression evaluation result smoothing program recorded on the recording medium is read into a computer system, You may implement | achieve by performing this computer system. This computer system includes an operating system (OS) and hardware of peripheral devices. The computer-readable recording medium is a portable recording medium such as a flexible disk, a magneto-optical disk, an optical disk, or a memory card, and a storage device such as a magnetic hard disk or a solid state drive provided in the computer system. Furthermore, a computer-readable recording medium dynamically holds a program for a short time, such as a computer network such as the Internet, and a communication line when transmitting a program via a telephone line or a cellular phone network. In addition, a server that holds a program for a certain period of time, such as a volatile memory inside a computer system serving as a server device or a client in that case, may be included. Further, the facial expression evaluation result smoothing program may be for realizing a part of the above-described functions, and further, the above-described functions are realized by a combination with a program already recorded in a computer system. You may do.

  As mentioned above, although embodiment of this invention was explained in full detail with reference to drawings, the specific structure is not restricted to that embodiment, The design of the range which does not deviate from the summary of this invention, etc. are included.

DESCRIPTION OF SYMBOLS 1 Face expression analyzer 10 Image data acquisition part 20 Face expression intensity teacher value acquisition part 30 Face area extraction part 31 Face area detection part 32 Analysis area determination part 40 Image feature-value calculation part 50 Regression analysis part 60 Regression model memory | storage part 70 Face Facial expression evaluation unit 80 Smoothing processing unit (Facial expression evaluation result smoothing device)
81 Facial expression intensity value acquisition unit 82 Facial expression intensity value smoothing processing unit 83 Facial expression type smoothing processing unit 90 Mode switching unit

Claims (7)

A facial expression intensity value acquisition unit that captures a plurality of facial expression intensity values obtained for each facial expression type based on a facial image for each frame;
Refer to the facial expression intensity values for a plurality of frames captured by the facial expression intensity value acquisition unit, and correspond to the plurality of frames based on a total value based on the facial expression intensity values of the plurality of frames for each facial expression type. A facial expression intensity value smoothing processing unit for calculating a representative facial expression intensity value;
A facial expression evaluation result smoothing apparatus comprising: The total value is the total value of the facial expression intensity values for the plurality of frames for each facial expression type.
The facial expression evaluation result smoothing apparatus according to claim 1. The total value is a total value obtained by counting the number of maximum facial expression intensity values in each of the plurality of frames for each facial expression type,
The facial expression evaluation result smoothing apparatus according to claim 1. The facial expression intensity value smoothing processing unit calculates the total value by weighting according to the position of each frame in the plurality of frames.
4. The facial expression evaluation result smoothing apparatus according to claim 2, wherein the facial expression evaluation result is smooth. A face selected from the facial expression types corresponding to the maximum total value among the total values for each facial expression type obtained by the facial expression intensity value smoothing processing unit, as a classification result of facial expressions corresponding to the plurality of frames. Facial expression type smoothing processing unit,
The facial expression evaluation result smoothing device according to claim 1, further comprising: The facial expression intensity value smoothing processing unit sets the number of frames smaller than the plurality of frames as a shift amount, and shifts the plurality of frames in the time direction by the shift amount.
The facial expression evaluation result smoothing apparatus according to claim 1, wherein the facial expression evaluation result is smooth. Computer
A facial expression intensity value acquisition unit that captures a plurality of facial expression intensity values obtained for each facial expression type based on a facial image for each frame;
Refer to the facial expression intensity values for a plurality of frames captured by the facial expression intensity value acquisition unit, and correspond to the plurality of frames based on a total value based on the facial expression intensity values of the plurality of frames for each facial expression type. A facial expression intensity value smoothing processing unit for calculating a representative facial expression intensity value;
Program for smoothing facial expression evaluation results.

Images