KR101198322B1 - Method and system for recognizing facial expressions - Google Patents

Abstract

PURPOSE: A facial expression recognizing method and a system thereof are provided to enhance regional spatial characteristics and weaken information of shapes which include unnecessary information by applying a DoG(Difference of Gaussian) kernel to an image applied with an AAM(active appearance model). CONSTITUTION: A DoG kernel(514) convolutes an input image. An AAM unit(516) extracts a facial area from the convoluted image, extracts an appearance and shape parameter from the facial area, and converts an AAM based on face feature elements in order to combine the AAM with the facial area. The AAM unit recognizes a facial expression by updating the appearance and shape parameter and the image forming the facial area to which a synthetic face image is inputted. An EFM(Enhanced Fisher Model) classifier(518) processes the appearance and shape parameter through an EFM classifying method. [Reference numerals] (510) Camera; (512) Image pre-processing unit; (514) DOG kernel; (516) AAM modeling unit; (518) EFM classifier; (520) Display unit; (522) Database

Description

Facial expression recognition method and system {METHOD AND SYSTEM FOR RECOGNIZING FACIAL EXPRESSIONS}

The present invention relates to a method and a system for recognizing facial expressions, and more particularly, a DoG (Difference of Gaussian) kernel is applied to an image applied to an AAM (Active Appearance Model) to enhance the visibility of a detailed area and reduce noise. By reinforcing the features of the local area such as the nose, mouth, etc. and weakening the information of the appearance that contains unnecessary information such as the ball, not only the unnecessary information can be removed from the image, but also the object feature extraction is removed by lighting. The present invention relates to a facial expression recognition method and system capable of maintaining important information.

Various techniques for extracting a facial expression of a user from the acquired image have been introduced. Techniques related to the detection of facial features that are currently introduced include methods using edge information, luminance, chrominance, geometric appearance and symmetry of faces, principal component analysis (PCA), There is a method using template matching, an approach using a curvature of a face, a method using a neural network, and the like.

Since edge information is mostly binarized, it is difficult to produce stable results due to loss of edge information when lighting is uneven. In addition, Principal Component Analysis has the advantage of easy real-time implementation, but it is sensitive to area separation and background image, and the neural network approach can operate stably under limited conditions, but it is difficult to adjust network variables when learning new inputs. The disadvantage is that it takes a lot of time.

After further expanding the technology of facial feature detection to recognize a face region included in a planar 2D image, studies are being actively conducted to realize a 3D image. Most of these studies produce a three-dimensional face model of a specific person by synthesizing two-dimensional images in various directions in four directions, eight directions, or sixteen directions, and using a panoramic face image as a texture image.

A representative model of the continuous facial expression recognition method is AAM (Active Appearance Model). AAM applies principal component analysis (PCA) to face model vectors and face surface texture vectors to warn sample face models created using a variety of human face statistics to normalize the sample face data. The error square of the face data of the captured image 2D is minimized. Use this data to find facial feature points. AAM has the advantage of speed calculation and tracking.

However, AAM has a disadvantage in that a lot of performance is degraded in a mobile environment in which lighting changes are severe. The reason is that the error between the model and the input image is used to calculate the optimized parameters used to update the AAM. This is because it is difficult to calculate the parameter.

Korean Laid-Open Patent No. 2010-0081874 (July 15, 2010): User-tailored facial expression recognition method and apparatus

The present invention has been made to solve the above-described problems, by applying a DoG (Difference of Gaussian) kernel to the image applied to the AAM (Active Appearance Model), thereby improving the visibility of the detailed area and reducing the noise due to lighting There is a technical problem to provide a facial expression recognition method and system capable of maintaining important information to be removed.

According to another aspect of the present invention, there is provided a facial expression recognition method comprising: convolving an input image using a DoG kernel; Extracting a face region from the convolved image; Extracting an appearance parameter and a shape parameter from the face region; Converting a statistical face model (AAM; Active Appearance Model) previously stored based on the extracted facial feature elements and synthesizing the facial region; And recognizing a facial expression by updating the appearance and shape parameters until the synthesized facial image converges within the preset mapping value with the image forming the input face region.

Here, in the step of convolving the input image using a DoG kernel, two Gaussian kernels having different standard deviations are convolved to make a blood image and then create a difference image between the two images. Calculating may be included.

The method may further include classifying facial expressions by processing the appearance parameters and the shape parameters with an Enhanced Fisher Model (EMF) classification method.

The present invention for achieving the above object is a facial expression recognition system, the DoG kernel to convolution the input image using the DoG kernel; And extracting a face region from the convolved image, extracting an appearance parameter and a shape parameter from the face region, and storing a statistical face model (AAM; Active) based on the extracted facial feature elements. After converting an Appearance Model and synthesizing the face region, the parameters of appearance and shape are updated until the synthesized face image converges within the preset mapping value with the image forming the input face region. AAM modeling unit for recognizing facial expressions.

Here, the DoG kernel generates a blood image by convolving images with two Gaussian kernels having different standard deviations, and then calculates a difference image between the two images to provide the AAM modeling unit. Can be.

The apparatus may further include an EFM classifier for classifying facial expressions by processing the appearance parameters and shape parameters with an Enhanced Fisher Model (EMF) classification method.

As described above, the method and system for recognizing facial expressions of the present invention improves visibility of detailed regions and reduces noise by applying a DoG (Difference of Gaussian) kernel to an image applied to an AAM (Active Appearance Model). Local area features, such as squeezing, squeezing, and so forth, can enhance and weaken information in shapes that contain unnecessary information, such as balls.

In addition, the facial expression recognition method and system of the present invention, by applying the DoG (Difference of Gaussian) kernel to the image applied to the AAM (Active Appearance Model), after removing unnecessary information from the image to extract the object feature to remove due to lighting Maintain important information.

1 is a conceptual diagram of a facial expression recognition method according to an embodiment of the present invention,
2 is a control flowchart of a facial expression recognition system according to an embodiment of the present invention;
3 and 4 are comparative experiment results of facial expression recognition results of the present invention and the prior art under the condition that the illumination changes,
5 and 6 are comparative experiment results of facial expression recognition results of the present invention and the prior art under the condition that the expression changes,
7 is a control block diagram of a facial expression recognition system according to an embodiment of the present invention;
8 is a control flowchart of a facial expression recognition system according to another embodiment of the present invention;
9 is a state diagram used in the facial expression recognition system according to an embodiment of the present invention.

Hereinafter, a facial expression recognition method and system according to an exemplary embodiment of the present invention will be described in detail with reference to the accompanying drawings. However, in describing the present invention, when it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted.

1 is a conceptual diagram of a facial expression recognition method according to an embodiment of the present invention.

As shown in FIG. 1, the present invention convolves an input image A with a Difference of Gaussian (DoG) kernel to generate a convolved image B. FIG. Subsequently, an AAM model is generated by fitting an AAM (Active Appearance Model) image to the convoluted image (B), and then a training set is applied to the output image (D) where a facial expression is recognized. Output

DoG Kernel is an image processing algorithm that removes noise of gray images and detects features. The DOG kernel is composed of two Gaussian kernels with different standard deviations, each of which convolves an image to produce a blurred image, and then calculates a difference image between the two images. Such a DoG kernel can be defined as in Equation 1 below.

[Equation 1]

In equation (5.16), L (x, y, kσ) and L (x, y, σ) are Gaussian kernels with different standard deviations kσ and σ.

The DOG kernel is an algorithm aimed at detecting image features, which is useful for enhancing the visibility of edges and other details in digital images. Since the DoG kernel reduces noise through Gaussian filtering, it can not only remove unnecessary information from the image, but also maintain important information provided by lighting through object feature extraction. In particular, when DoG kernels are applied to face images, local features such as eyes, noses, mouths, etc. may be strengthened, and information of shapes containing unnecessary information, such as balls, may be weakened.

The image B convolved with the DoG kernel has a local shape containing a lot of information in the face image, for example, a shape of a feature part such as an eye, a nose, a mouth, and the like, thereby recognizing a face shape.

The facial expression may be recognized by performing AAM fitting on the convolved image B. FIG.

When the DoG kernel is applied to AAM, the expression of AAM can be defined as [Equation 2] as follows.

&Quot; (2) "

In the above formula, * means the image to which the DoG kernel is applied, that is, the convolved image (B).

The fitting algorithm used in AAM extracts facial feature elements, transforms the statistical face model based on the extracted facial feature elements, and models a synthetic face image matching the face region. After that, the parameters of the appearance and shape are repeatedly updated until the synthesized face image converges within the input face region within a preset mapping value, thereby reducing the error between the model and the image.

When the external parameters and the shape parameters of the input image are measured, the input image is aligned on the coordinate frame, and the current model instant (C) and the training set are convolved to obtain an error image between the AAM-fitted image, thereby reducing and optimizing the error. The fitting algorithm continues to iterate until the error satisfies the aforementioned threshold or the specified number of times is repeated, thereby recognizing the facial expression with the optimized error.

As described above, according to the present invention, by applying the DoG kernel to the AAM, the feature of the local shape containing a lot of information in the object of the face image, for example, the shape of the eye, nose, mouth, etc. After attenuating the shape information, the performance of the AAM fitting algorithm can be improved by performing the AAM fitting algorithm.

2 is a control flowchart of a facial expression recognition system according to an embodiment of the present invention.

The facial expression recognition system according to an embodiment of the present invention trained an Enhanced Fisher Model (EMF) model using an AAM model for facial expression recognition and an image used to generate the AAM model (S110). In the AAM model where facial expression data is stored, facial expression image sequences of males and females of various ethnic backgrounds are stored. For example, facial expression images such as joy, surprise, anger, disgust, fear, and sadness may be stored. The EFM model training can be applied to the images used to generate the AAM model. EFM was introduced to improve the performance of the standard Fisher linear discriminant (FLD) based approach, and the first EFM was applied to Principle Component Analysis (PCA) for dimension reduction and used to identify the reduced PCA subspace. In facial expression recognition systems, EFM classifies facial expressions by determining features between facial expressions.

The image including the face to be recognized is input to the facial expression recognition system (S112).

The face is recognized from the input image (S114). Here, DoG kernel is applied to face recognition. By convolving the input image using the DoG kernel, the visibility of edges and other details included in the image is increased to accurately recognize a face included in the image.

Thereafter, AAM image fitting is started on the recognized face image (S116).

Through the AAM image fitting process, parameters of appearance and shape of the face are extracted from the image (S118 and S120).

The parameters of the appearance and shape of the extracted face are processed using an EFM classifier (S122). EFM classifies facial expressions by determining features between facial expressions.

When the expression is classified by the EFM classifier (S124), the training set reflecting the expression and the shape is convolved with the input image (S126).

Thereafter, the parameters of appearance and shape are updated repeatedly until a certain threshold is met, thereby reducing the error between the model and the image. For example, if the parameter of the current shape is measured, it fits the input image on the model coordinate frame and obtains an error image between the current model instant and the image fitted by AAM to reduce and optimize the error. The fitting algorithm continues to iterate until the error meets the aforementioned threshold or iterations for a specified number of times.

3 and 4 are comparison graphs of facial expression recognition results of the present invention and the prior art under the condition that the illumination changes.

Figure 3 shows the AAM fitting result when the lighting changes, when only AAM is applied (1-1), when AAM is applied to the image convolved with the DoG kernel (1-2), Canny Edge Detector is When the AAM is applied to the applied image (1-3) shows the facial expression recognition results.

As shown in FIG. 3, when AAM is applied to the image convolved with the DoG kernel (1-2) and AAM is applied to the image to which the Canny Edge Detector is applied (1-3), AAM fitting is more effectively performed. You can check it.

4 is a graph showing the accuracy of facial expression recognition results according to the prior art and facial expression recognition results according to the present invention when illumination changes.

The graph shows the facial expression recognition result when the image captured in the environment of changing lighting is applied only with AAM, when AAM is applied to the image convolved with the DoG kernel, and when AAM is applied to the image with Canny Edge Detector. The accuracy of the calculation is shown in the graph.

In order to evaluate the accuracy of each recognition result, the standard deviation and mean error were calculated between shape and ground truth using RMS error. In addition, in order to measure the performance of each AAM more accurately, the number of images corresponding to the mean error was cumulatively calculated.

As shown in the graph of FIG. 4, it can be seen that the AAM to which the DoG kernel and the Canny Edge Detector are applied has accumulated more images at a smaller error rate. In particular, AAM with DoG kernel accumulates the most images in the section where the average error is small.

5 and 6 are comparison graphs of the facial expression recognition results of the present invention and the prior art under the condition that the expression changes.

5 shows the AAM fitting result when the illumination changes, when only AAM is applied (2-1), when AAM is applied to the image convolved with the DoG kernel (2-2), and the Canny Edge Detector is shown. When the AAM is applied to the applied image (2-3) shows the facial expression recognition results.

When AAM is applied to the image convolved with the DoG kernel as shown in FIG. 5 (2-2), when AAM is applied to the image to which the Canny Edge Detector is applied (2-3), the AAM fitting is more effectively confirmed. Can be. This is because the features of the model with Canny Edge Detector do not fit the features of the face well because the features of the image are weaker than those with the DoG kernel.

6 is a graph showing the accuracy of the facial expression recognition result according to the prior art and the facial expression recognition result according to the present invention under a facial expression change.

In the graph shown, when AAM is applied to an image taken in an environment where facial expressions are changed, when AAM is applied to an image convolved with a DoG kernel, AAM is applied to an image to which Canny Edge Detector is applied. The accuracy of the facial expression recognition result is calculated and shown in the graph.

In order to evaluate the accuracy of AAM, we calculate the standard deviation and mean error between shape and ground truth by using root mean square (RMS) errorr, and measure the number of images corresponding to mean error to measure AAM performance more accurately. Cumulative calculations.

As shown in the graph of FIG. 6, when AAM is applied to an image convolved with a DoG kernel, it can be seen that more images are accumulated at a smaller error rate.

On the other hand, in the case of AAM with Canny Edge Detector, the performance is not good. This is because the features of the model with Canny Edge Detector are weaker than those with the DoG kernel, and do not fit the features of the face well.

As can be seen from the above experimental results, when the DoG kernel is applied to the image used for AAM, the expression recognition performance can be guaranteed even when the lighting changes or the expression changes.

7 is a control block diagram of a facial expression recognition system according to an embodiment of the present invention, illustrating a case where the facial expression recognition system of the present invention is applied to a portable terminal such as a smartphone.

As shown in FIG. 7, the facial expression recognition system includes a camera 510, an image preprocessor 512, a DOG kernel 514, an AAM modeling unit 516, an EFM classifier 518, and a display unit 520. Database 522.

The camera 510 generates an 2D image by capturing an image of an object, for example, a face of a specific person.

The image preprocessor 512 converts the 2D image provided from the camera 510 into an image for face recognition.

The DOG kernel 514 reduces the noise of the input image through Gaussian filtering to enhance the visibility of edges and other details and to attenuate information in the shape that contains redundant information that is repeated. The image convolved with the DOG kernel 514 is a local shape containing a lot of information of the face image, for example, the shape of the feature portion, such as eyes, nose, mouth, etc. is enhanced to recognize the face shape.

The AAM modeling unit 516 extracts parameters of appearance and shape from the image convolved with the DOG kernel 514 and converts a statistical face model based on the extracted facial feature elements to match the face region. A synthetic face image is modeled. The AAM modeling unit 516 repeatedly updates the parameters of appearance and shape until the synthetic face image converges within the preset mapping value with the face image to reduce the error between the model and the image.

The EFM classifier 518 recognizes an expression of the face image based on parameters of appearance and shape. The facial expression recognition result of the EFM classifier 518 is provided to the AAM modeling unit 516 and used to determine the appearance and shape parameters for modeling the composite face image.

The database 522 stores a facial expression image sequence for performing AAM. In addition, the EFM model generated by training the facial expression image sequence is stored.

The display unit 520 displays the face recognition result determined by the AAM modeling unit 516.

8 is a control flowchart of a facial expression recognition system according to another embodiment of the present invention.

The user may photograph a face with the camera 510 and input a face image (S612).

The input face image is converted using the DoG kernel 514 (S614). Accordingly, the facial features are recognized by enhancing local features included in the facial image (S616).

The AAM modeling unit 516 performs an AAM fitting on the image converted into the DoG kernel 514 to detect parameters of appearance and shape for modeling a face image (S618).

The EFM classifier 518 classifies facial expressions based on parameters of appearance and shape in the AAM modeling unit 516 (S620).

The AAM modeling unit 516 repeats the updating of appearance and shape parameters by referring to the classification result of the EFM classifier 518 to recognize the expression of the input face image and displays the recognition result (S622). ).

9 is a diagram illustrating a usage state of the facial expression recognition system according to an exemplary embodiment of the present invention, and illustrates a usage state when a service is provided by applying the facial expression recognition system according to an exemplary embodiment of the present invention to a portable terminal.

The face image of the user photographed by the camera 510 may be input to a facial expression recognition system mounted in the portable terminal so that the expression of the current user may be recognized in real time.

The recognized user's expression is reflected in real time on the character displayed on the portable terminal.

Accordingly, the user's current facial expression and the facial expression of the character displayed on the portable terminal are linked to each other to display emotions such as (a) no expression, (b) joy, (c) surprise, and (d) sadness through the character in real time. Can be.

On the other hand, in the above-described embodiment has been described a case where the user's facial expression is reflected on the emoticon character, the facial expression recognition result of the facial expression recognition system of the present invention is the display of emotions such as the user's avatar, image, animal or animation character All possible graphic images are applicable. In addition, various applications are possible, such as outputting a facial expression recognition result as text, changing the graphic color of a display screen, or a theme.

As described above, according to the present invention, the DOG kernel 514 is used to remove the noise of the image and maintain only the features, thereby removing the information not necessary for the AAM and maintaining the information necessary for the fitting algorithm. Accordingly, excellent facial recognition performance can be obtained even in an image having a change in illumination and a facial expression, and excellent performance can be guaranteed even in a low quality image obtained from a portable terminal such as a smart phone.

Thus, those skilled in the art will appreciate that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the embodiments described above are to be considered in all respects only as illustrative and not restrictive. The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

According to the present invention, by applying a DoG (Difference of Gaussian) kernel to an image applied to an AAM (Active Appearance Model), the local area, such as the eyes, nose, mouth, etc. is enhanced by enhancing the visibility of the detailed area and reducing the noise. Facial expression recognition method that not only removes unnecessary information from the image but also maintains important information removed by lighting through object feature extraction by weakening appearance information containing unnecessary information repeated as seen. And systems.

510: camera 512: image preprocessor
514: DOG kernel 516: AAM modeling unit
518: EFM classifier 520: display unit
522: database

Claims (6)

Convolving the input image using the DoG kernel;
Extracting a face region from the convolved image;
Extracting an appearance parameter and a shape parameter from the face region;
Classifying a facial expression by processing the appearance parameter and the shape parameter with an Enhanced Fisher Model (EMF) classification method;
Reflecting the classification result, converting a previously stored statistical facial model (AAM; Active Appearance Model) based on the appearance parameter and the shape parameter to synthesize the face area; And
Recognizing a facial expression by updating the parameters of the appearance and shape until the composite face image converges within the preset mapping value with the image forming the input face region .
The method of claim 1,
Convolving the input image using a DoG kernel,
A facial expression recognition method comprising calculating a difference image between two images after creating a blood image by convolving images with two Gaussian kernels having different standard deviations, respectively.
delete A DoG kernel which convolves the input image using the DoG kernel;
Extract a facial region from the convolved image, extract an appearance parameter and a shape parameter from the facial region, and store a statistical face model (AAM; Active Appearance) based on the extracted facial feature elements. Model, and synthesizes the face region, and then updates the appearance and shape parameters until the synthesized face image converges within the preset mapping value with the image forming the input face region. An AAM modeling unit for recognizing facial expressions; And
An EFM classifier for classifying facial expressions by processing the extracted appearance parameters and shape parameters with an Enhanced Fisher Model (EMF) classification method, and providing a classification result to the AAM modeling unit;
Facial expression recognition system comprising a.
5. The method of claim 4,
The DoG kernel is
Facial expression recognition, characterized by convoluting images with two Gaussian kernels having different standard deviations to create a blooded image, and calculating the difference image of the two images and providing them to the AAM modeling unit. system.
delete

Images