JPH04342078A - Device for recognizing facial expression - Google Patents

Abstract

PURPOSE:To offer an expression recognizing device with accuracy by measuring the slight movement of muscle all over a face, making the time change into a pattern and recognizing expression expressing feeling based on the pattern. CONSTITUTION:The expression recognizing device is constituted in such a way that an optical flow computer is used as a module 1 measuring the movement of skin, the time change pattern of skin movement is converted (2) into a first and second moments and a feature vector in a standard expression picture is set and used in a recognition mode.

Description

[Detailed description of the invention]

0001

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a facial expression recognition device that measures facial expressions using a computer based on time-series images of facial expressions and performs machine recognition of facial expressions.

0002

[Prior Art] In the field of psychology, several methods have been proposed for analyzing human facial expressions through human observation, as described in, for example, "Introduction to Facial Expression Analysis," edited by Riki Kudo, Seishin Shobo. Recently, when transmitting human facial expressions and movements in systems such as videophones and videoconferencing, the trend is to communicate not in the form of image signals but as symbolic data such as "smile". has been done. In Proceedings of the 1988 Institute of Electronics, Information and Communication Engineers Spring National Conference D-97, Uchiyama et al.'s "Facial Expression Analysis of Facial Images in Analysis and Synthetic Image Coding," a method for facial expression analysis is proposed.
This is an attempt to specify facial feature points in advance and recognize facial expressions from their changes, but it is difficult to automatically detect these feature points with high precision, and the locations that can be used as feature points are limited, so it is not always possible to recognize facial expressions by specifying feature points on the face. The drawback was that it was unable to capture even the subtlest movements. Another problem was that these data had little correlation with the movements of the muscles that control facial expressions, and were not analyzed physiologically.

[0003] Also, the inventor filed a patent application: Japanese Patent Application No. 2003-
As seen in No. 50246 "Facial Expression Recognition Device" (filed on March 1, 1990), they also announced a device that measures minute muscle movements and recognizes facial expressions based on patterns of temporal changes. has been done. The invention utilized the output of a "facial expression measurement device" (Japanese Patent Application No. 50247/1999) that detects movements of facial muscles assuming locations corresponding to the positions of the muscles. According to this, when the location of the measured muscle is required and sufficient information is not necessarily provided, there is a problem that facial expressions are recognized using useless information, and the recognition rate may not be sufficient. .

0004

[Problem to be Solved by the Invention] Furthermore, the goal of these facial expression recognition methods and devices is to describe facial expressions using a combination of symbols, such as "laughter", "anger", "
They had not yet been able to express and recognize their feelings in words such as "surprise."

[0005] On the other hand, many inventions have been reported regarding methods for recognizing images and audio patterns, but it is important to know how well to input feature values that depend on the object into the recognition method.
This greatly affects the recognition rate. However, there have been only a few reports on devices that recognize emotional facial expressions, and it was unclear what specific features should be used to apply previously proposed pattern recognition methods.

[0006] The present invention provides an accurate facial expression recognition device that measures minute movements of muscles over the entire face, patterns their temporal changes, and recognizes facial expressions expressing emotions based on the patterns. The purpose is to

[0007]

[Means for Solving the Problems] The present invention provides a system that performs efficient recognition by learning from data on the deformation of the epidermis of various facial expressions and performing recognition using the location with the greatest amount of information. FIG. 1 shows a basic configuration diagram of the present invention.

In the present invention, (1) an optical flow calculator is used as the module 1 for measuring the movement of the epidermis, and (2) a converter is used to convert the temporal change pattern of the movement of the epidermis into first and second moments. 2 is used to make this a recognition pattern, and (3) further, it is determined based on statistical analysis in the identification effective reference amount processing unit 3 for feature amounts in standard facial expression images, thereby determining a low-dimensional feature vector, and ( 4) It is used in recognition mode.

[0009]

[Effect] This makes it possible to efficiently select feature vectors from limited standard facial expression data, and by increasing the number of standard facial expression images, it is possible to increase the learning effect of facial expressions and achieve a high recognition rate. A facial expression recognition device can be configured. In addition, it is possible to provide a means for verbally transmitting facial expressions in an image communication system, and also to provide a means for transmitting human intentions to a machine in an interface with a computer.

0010

DESCRIPTION OF THE PREFERRED EMBODIMENTS The operation of the apparatus of the present invention will be explained below with reference to the drawings. FIG. 2 is a diagram illustrating the overall configuration of the facial expression recognition device of the present invention, and 101 is a video input device consisting of an image pickup device such as a television camera, an A/D conversion circuit, and the like. The facial expression recognition device 102 of the present invention includes an epidermal deformation measurement unit 103, a facial expression feature extraction unit 105, a standard feature calculation unit 106,
It is composed of an expression feature comparison section 108, a standard expression feature accumulation section 109, and recognition learning switching interlock switches A and B 104 and 107.

The operation is first in the standard expression learning mode, and learning is performed by flipping the interlocking switch A104 and switch B107 in the figure to the lower terminal side. In learning, first, standard facial expression motion images for learning are captured, and the time-series images are input to the epidermal deformation measuring unit 103 to measure epidermal movements related to facial expression generation of the entire face, and standard facial expression features are measured. The extraction unit 105 performs pattern learning and determines the space of facial expression feature vectors. Furthermore, the value of the facial expression feature vector is calculated for each standard facial expression motion image, and the standard facial expression feature storage unit 1
Store it in 09.

Next, the operation of the facial expression recognition mode will be explained. For the facial expression image to be recognized, the interlocking switch 104
, 107 to the upper terminal side for recognition. In the recognition mode, first, facial expression movement images are photographed, and the time-series images are input to the epidermal deformation measurement unit 103 to measure epidermal movement related to facial expression generation. Since the position of the epidermis of the face for creating a feature vector is known at the time of learning as described later, the movement of the entire face may be measured and sent to the facial expression feature calculation unit 106, or the Position information may be obtained from the facial expression feature calculation unit 106 and epidermal deformation may be measured only at that location.

[0013] A case will be described below in which the movement of the entire face is measured and data is transferred. Next, the facial expression feature calculation unit 106 that has received the epidermal deformation data calculates the value of the facial expression feature vector for the input image based on the facial expression feature vector space previously determined in the learning mode, and sends it to the facial expression feature comparison unit 108. send. The facial expression feature comparison unit 108 compares the facial expression feature vector of the recognition target image with the standard facial expression feature vector stored in the standard facial expression feature storage unit 109 in the learning mode, searches for a similar standard facial expression feature vector, and performs recognition. Outputs the recognition result of what expression the target image has.

The operation of each part will be explained below. The epidermal deformation measurement unit 103 is a circuit that measures the movement of the epidermis related to facial expressions, and is a circuit that realizes a function of calculating a motion vector called optical flow for each point in an image. Various methods have already been proposed for calculating optical flow. For example, Horn's “Robot
Vision” (MIT Press, 198
It is explained in detail in Chapter 12 of 5). or,
It is also possible to implement and utilize a motion vector detection mechanism such as that used in image encoding devices.

Here, for the sake of explanation, from time t to time t+
Each position (x, y
) at (ut (x, y), vt
A set of vertical and horizontal components such as (x, y)) is output as the amount of skin deformation. In the learning mode, the facial expression feature extracting unit 105 receives the above-mentioned data in which the amount of epidermal deformation changes over time for each standard facial expression image, and operates as follows to determine the space of feature vectors and the standard facial expression feature vector. .

First, when the size of an image is N×M pixels, it is divided into square regions of n×m r×r pixels. Then, for each i (0<i≦n) row, j (0<j≦m) column region R(i,j), the next first order that most accurately and simply expresses the characteristics of the amount of epidermal deformation. Statistics due to moment and second moment, average μu,i,j,μv,
i,j and the variances σuu,i,j, σuv,i,j and σvv,i,j are calculated.

[0017]

[Math 1]

[0018]

[Math 2]

[0019]

[Math 3]

[0020]

[Math 4]

[0021]

[Math 5]

When these calculation results are arranged, the following K=5
mn-dimensional vector, F={μu,1,1,μv,1,1,σuu,
1,1,σuv,1,1,σvv,1,1,...μu,i
, j , μv, i, j , σuu, i
,j,σuv,i,j,σvv,i,j,...μu,n,
m , μv, n, m , σuu, n,
m, σuv, n, m, σvv, n, m}
(6) can be done, but for recognition it is necessary to reduce the number of dimensions. Therefore, using the following discrimination effective reference quantity (evaluation function), a vector element with a large evaluation quantity is selected as a recognition feature parameter, and a target feature vector is created.

[0023]

[Math 6]

[0024] However, varW (k) varB (k
) are the inter-class variance and intra-class variance of the k-th vector element fk, respectively, and are calculated using the following equations.

[0025]

[Math 7]

[0026]

[Math. 8]

Note that c is the number of classes to be recognized; for example, when recognizing "laughter, sadness, anger, surprise", c = 4, and θi is the vector calculated from the training sample of the i-th class. Indicates a collection of elements. Also,

[0028]

[Math. 9]

is the average value of the vector element fk for the training samples in class i,

[0030]

[Math. 10]

is the average value of all the learning samples. Traditionally, this formula was used to derive pattern discrimination functions, but
Here, it is used to reduce the dimension of the feature vector using fractional training samples. For this purpose, arrange the J(k) calculated above in descending order, and

[0032]

[Math. 11]

[0033] Select the feature vector

[0034]

[Math. 12]

Determine . In other words, a list of which areas and what feature quantities the selected elements are is sent to the facial expression feature calculation unit 106 . Furthermore, for each learning sample image, the selected feature parameters are arranged in order to create a feature vector, which is transferred to the standard facial expression feature storage unit 109. FIG. 3 is a configuration example of the feature parameter position type list 201 sent to the facial expression feature calculation unit 106. The type (one of the equations (1) to (5)) is determined by which of the equations (1) to (5) above each selected feature vector element corresponds to. ) and the position (i and j in equations (1) to (5)).

For example, the feature vector element at the top of the diagram is given when i=4 and j=5 in equation (1). The contents of the litos are just an example, and may actually change depending on the learning data. FIG. 4 is an example of feature vector data 301 sent to and stored in the standard facial expression feature storage unit 109. For each learning sample image, the type of facial expression (instructed by the person) is associated with the respective feature vector.

Next, in the target facial expression image recognition mode stage, the operation is as follows. As in the learning mode, the muscle movements calculated by the epidermal deformation measurement unit 103 are input to the facial expression feature calculation unit 106. The calculation unit calculates statistics necessary for creating a feature vector from the input data based on the feature parameter position type list 201. That is, according to list 201, for example

[0038]

[Math. 13]

Calculate μu,5,6 as follows. The facial expression feature calculation unit 106 calculates these elements.

[0040]

[Math. 14]

After the calculation, a feature vector is constructed and output. The output of the facial expression feature calculation section 106 is input to the facial expression feature comparison section 108. When the facial expression feature comparison unit 108 receives the feature vectors of the recognition data, it sequentially reads the feature vectors of the learning data one by one from the standard facial expression feature storage unit 109 and calculates the distance between the two feature vectors. After calculating the distances to all the learning data, the corresponding facial expressions are checked from the table of feature vector data 301 of the k pieces of learning data with the minimum distance, and the majority vote is taken to output the most facial expression labels as results. The table of feature vector data 301 is an example when there are L pieces of learning data. for example,
As a result of calculating the distance between the feature vector of the unknown recognition data and the feature vector data 301 of the learning data, the three with the shortest distance were data number 1 (laughter), number 3 (anger), and number 4 (laughter). In this case, ``laughter'' is the recognition result by majority vote. This facial feature comparison unit 108 is k
-The value of k is “1” to “5” using a method called the nearest neighbor method.
Usually used.

[0042]

[Effects of the Invention] As explained above, according to the present invention,
Since facial expressions can be recognized without contact based on the deformation of the facial skin, it can be used as a facial expression analysis module in image communication, and has the advantage of being able to automatically measure subtle muscle movements without contact in the fields of psychology and medicine. There is. Additionally, it can provide a function to convey human emotions to machines as a computer interface.

[Brief explanation of drawings]

FIG. 1 is a diagram showing the principle configuration of the present invention.

FIG. 2 is a diagram illustrating the overall configuration of the facial expression recognition device of the present invention.

FIG. 3 is an example of a feature parameter position type list.

FIG. 4 is an example of feature vector data accumulated in a standard facial expression feature accumulation section.

[Explanation of symbols]

1 Module that measures epidermal movement 2 Transducer 3 Processing section 101 Video input device 102 Facial expression recognition device 103 Epidermal deformation measurement section 104 Switch A 105 Facial expression feature extraction part 106 Facial expression feature calculation section 107 Switch B 108 Facial expression feature comparison section 109 Standard facial features storage unit

Claims (1)

[Claims] Claim 1: In a facial expression recognition device that recognizes facial expressions based on time-series images obtained by photographing a face exhibiting an expression using a video input device, an epidermal deformation measurement unit detects an expression on the time-series images. , is equipped with an expression feature extraction unit, an expression feature calculation unit, an expression feature comparison unit, and a standard expression feature accumulation unit, and inputs the time-series images to an epidermal deformation measurement unit to measure epidermal movements related to expression generation as optical flow data. The effective reference amount for discrimination is calculated from the first- and second-order statistical patterns of the optical flow data of the standard facial expression time-series images prepared at the time of learning, and the feature vectors for identifying facial expressions are determined. During recognition, equivalent feature vectors are calculated from the given facial expression time-series images of the recognition target, and the distance between the accumulated standard facial expression feature vectors and the recognition target feature vector is calculated. A facial expression recognition device that selects a standard facial expression that is close to the recognition target based on the above, and outputs a recognition result of what kind of facial expression the facial expression time-series image of the recognition target is based on the result.