KR20090050199A - Real-time facial expression recogintion based on optical flow and hidden markov models - Google Patents

Abstract

The present invention tracks motion information of feature points such as eyebrows, eyes, and mouth of a human face using an optical flow algorithm, and recognizes facial expressions using a hidden Markov model. The facial expression recognition system is composed of preprocessing, face detection, facial feature point tracking, and facial expression recognition. In the preprocessing process, the image is improved by using the maximum-minimum normalization method and the reference white as a step to minimize the influence of illumination on the image input in real time. The face region should be detected for the improved image. In this system, the candidate region of the face is detected using the H-T color model, and the exact face region is detected using the principal component analysis and the template matching. In the facial feature point tracking step, the positions of eyebrows, eyes, and mouth feature points are acquired using geometric information of the face, and the motion of each feature point is tracked using an optical flow algorithm. In the facial expression recognition step, facial expression recognition is performed by applying the facial expression model proposed in this system to the hidden Markov model.

Face, facial expression, recognition, optical flow, hidden markov model

Description

Real-Time Facial Expression Recogintion based on Optical Flow and Hidden Markov Models}

Facial expression recognition is being actively researched in various fields such as psychology research, facial animation synthesis, robotics, and HCI. Facial expressions contain not only human emotions but also a variety of information, including state of mind, social interactions, and physiological signals. The importance of research on facial expressions is increasing because it is closely related to research in various areas such as face detection, face tracking, and face recognition due to the reduction of processing cost due to the improvement of computer performance. Because it can develop into. Facial expression recognition can be developed in connection with research in various areas such as virtual reality, image comprehension, psychology research, medical field, facial image compression, and facial animation synthesis.

Facial expression recognition must take into account a variety of factors, including a person's age, ethnicity, gender, and culture. Facial expression recognition consists of three stages: face detection, facial feature extraction, and facial expression classification.

Facial feature extraction can be classified into deformation-based extraction and motion-based extraction. Deformation-based extraction method is a method of extracting the analysis of the global or local region for the face image, and requires a preprocessing to separate the face and the background in order to accurately extract. The method of extracting the model-based facial feature deformation is a method of extracting the deformation by using face shape and texture information. ASM is a method of statistically analyzing the changing shape using a shape model composed of vertices, and AAM is a method of analyzing a face image using the shape and texture information of the face. There is a Labeled Graph method that uses a graph consisting of Gabor Jet extracted by applying Gabor Wavelet to the facial feature points.

The method of extracting the motion of facial features is as follows. The Dense Optical Flow method analyzes the movement of the face by extracting the flow of change in pixel values for the global and local facial features. Feature Point Tracking is a method of extracting the movements of the main features of the face. Marker Tracking is a method of acquiring a motion by tracking a position change of a marker caused by facial expression by attaching a marker directly on the face.

The facial expression classification process can be largely divided into a spatial approach and a space-time approach. Spatial approach is to classify facial expressions through analysis of images and classify facial expressions using methods such as PCA, ICA, and Gabor Wavelet. The spatiotemporal approach classifies facial expressions by analyzing patterns of facial feature changes over time, and classifies facial expressions using methods such as HMM and Spatio-Temporal Motion-Energy Template.

Facial expression recognition consists of face region detection, feature point tracking, and facial expression recognition. To solve the problem of detecting an accurate face region from an input image, accurately acquiring facial feature points for facial expression recognition, and recognizing various facial expression changes.

It is an object of the present invention to provide a method for recognizing facial expressions by tracking changes in facial feature points based on real time. To achieve the objective, facial expression recognition is performed using optical flow algorithm and hidden Markov model. After applying the maximum-minimum normalization method and the reference white to the input image in real time, the pre-processing process is minimized to minimize the influence of illumination, and then the face candidate region is detected using the H-T color model. An accurate face region is obtained by using principal component analysis and template matching on the detected candidate region. Features of eyebrows, eyes, and mouths are extracted using the geometric information of the face with respect to the detected face region, and the movement of each feature point is tracked using an optical flow algorithm. The facial information is recognized by applying the motion information of the feature points extracted every frame to the hidden Markov model.

The existing facial expression recognition method recognizes only the change between the basic facial expression and each final facial expression. In the present invention, by designing a facial expression pattern model that can recognize the pattern of change between the final facial expression and the basic facial expression and each final facial expression, it is possible to recognize more various expressions compared to the conventional facial recognition method It works.

Hereinafter, the configuration of the present invention with reference to the accompanying drawings.

1 is a block diagram of a facial expression recognition system, and includes a face detection step, a facial feature point tracking step, and a facial expression recognition step.

The first step is to improve the image through a preprocessing process for the image input in real time. The maximum-minimum normalization method of equation (1) and the reference white of equation (2) are applied.

(1)

(One)

(2)

(2)

The face candidate region is detected using the H-T color model for the improved image. 2 is a diagram illustrating a process of detecting a face candidate area. The input image of the RGB color model is obtained by converting the input image of the RGB color model into the H value of the HSI color model and the T value of the TSL color model.

As shown in FIG. 3, the face candidate region is used to obtain an accurate face region using principal component analysis. Principal component analysis is a method of reducing dimensional data into low dimensional data by removing dominant components by rearranging eigenvalues in descending order. The image obtained by obtaining the average face image from the training data of the face image and projecting the eigenvector of the covariance matrix composed of the difference vectors between the training data and the average image in the eigenspace is the eigenface. High similarity with After applying principal component analysis to the face template image obtained from the input face candidate region and the face training data, the Euclidean distance of Equation (3) between two feature vectors is calculated to obtain an accurate face region.

(3)

(3)

Using the geometric information of the face, we extract the features of the eyebrows, eyes and mouth of the face and trace the movement of each feature point. The face is divided into three parts, with the eyebrows and eyes in the upper two thirds and the mouth in the lower thirds. After identifying the areas where the feature points exist, the template matching is used to obtain the feature positions of the eyebrows, eyes and mouth of the face. In order to track the movement of the feature points of the face, the present invention uses an optical flow algorithm (FIG. 4). The optical flow algorithm calculates and displays the displacement of pixels in two consecutive images. In the present invention, the LK algorithm proposed by Lucas-Kanade is used. The LK algorithm measures a motion field as motion information by using a difference in brightness values of an image under the assumption that the motion speeds in the feature region of the image are the same. It has the advantage of being accurate and robust against other optical flow algorithms.

Facial expression recognition is performed by applying motion information of facial feature points to a hidden Markov model. The facial expression recognition environment in the present invention is non-discrete and takes into account the continuity of time, and has a feature that it is impossible to predict the next state. The hidden Markov model is a finite state machine that has a probability of transition between states. It is a method of estimating states through observation symbols that cannot be directly observed and observed from each state with a certain probability. The hidden Markov model consists of the state transition probability matrix A, the observed symbol probability matrix B, and the initial state distribution π (Equation 4).

(4)

(4)

To apply the hidden Markov model, we find a probability estimation problem that evaluates the probability that an observation string will be observed in a given model, given an observation string and a model, and finds an optimal state string that best represents the observation column and a given observation model. We need to solve the optimal state sequence search problem and the parameter estimation problem of estimating the parameters of the model using the given training data. The probabilistic evaluation problem can be solved using the forward-backward algorithm combining the forward and backward algorithms, and the optimal state sequence search problem can be solved by using the Viterbi algorithm. And the parameter estimation problem can be solved using the Baum-Welch re-estimation algorithm (Fig. 5).

Although facial expression recognition using the conventional hidden Markov model recognizes only a change between the basic facial expression and each final facial expression, the present invention proposes a model capable of recognizing a change pattern between each final facial expression as shown in FIG. 6. There is an effect that can be recognized for a variety of facial expression changes.

1 is a block diagram of a facial expression recognition system according to the present invention

2 is a face candidate region detection process applied to the present invention

3 is a facial region detection process using the principal component analysis applied to the present invention

4 is a feature tracking process using the optical flow algorithm applied to the present invention

5 is a method for solving three problems of the hidden Markov model applied to the present invention.

6 is a facial expression change pattern model applied to the present invention

Claims (3)

A preprocessing step of improving an image with respect to an input image, a face detection step of detecting a face region, a feature tracking of eyebrows, eyes, and mouth by using geometrical information of the face to track the movement of each feature point, and facial expression recognition In the step, the existing facial expression pattern was only possible to recognize between the basic facial expression and each maximum facial expression, but in the present invention, a model is designed to recognize the change pattern between each maximum facial expression and facial expression recognition using the same. The method of claim 1, A method of extracting features of eyebrows, eyes, and mouth of a face using geometric information of a face and tracking the movement of each feature point. The method of claim 1, A method of recognizing facial expressions by designing a model to recognize a change pattern between each maximum facial expression in addition to the change pattern between the basic facial expression and each maximum facial expression.

Images