KR20240077290A - Apparatus for recognizing facial expression intensity - Google Patents

Abstract

The present invention relates to an facial expression intensity recognition device, comprising: an image pre-processing module that performs image pre-processing of an input image; a processor that processes the image on which the image preprocessing has been performed according to a designated algorithm to classify facial expressions and facial expression intensity; and a fine expression recognition module that recognizes and outputs facial expressions of the input image in granular stages based on the results of the expression classification and expression intensity classification.

Description

Facial expression intensity recognition device {APPARATUS FOR RECOGNIZING FACIAL EXPRESSION INTENSITY}

The present invention relates to an facial expression intensity recognition device, and more specifically, to an facial expression intensity recognition device that enables recognition of segmented micro-expressions by recognizing the intensity of the facial expression when recognizing it. .

Recently, various algorithms have been proposed to classify and process human facial expressions through computers.

The most popular algorithm for learning facial expressions is CNN (Convolutional Neural Networks), which extracts facial features from images showing changes in the target's expression, combines them with facial expression datasets, and learns them through a deep neural network to classify and process them. Examples of learning methods, etc. can be provided.

However, facial expression recognition using the general CNN learning method has limitations in recognizing fine-grained micro-expressions that reflect human emotions.

The background technology of the present invention is disclosed in Republic of Korea Patent Publication No. 10-2020-0047311 (published on April 20, 2020, face recognition device using deep learning).

According to one aspect of the present invention, the present invention was created to solve the above problems, and enables recognition of fine-grained fine expressions by recognizing the intensity of facial expressions when recognizing facial expressions. The purpose is to provide a facial expression intensity recognition device.

An facial expression intensity recognition device according to an aspect of the present invention includes an image pre-processing module that performs image pre-processing of an input image; a processor that processes the image on which the image preprocessing has been performed according to a designated algorithm to classify facial expressions and facial expression intensity; and a fine expression recognition module that recognizes and outputs facial expressions of the input image in granular stages based on the results of the expression classification and expression intensity classification.

According to one aspect of the present invention, when recognizing facial expressions, the present invention enables recognition of the intensity of various facial expressions, thereby enabling recognition of fine-grained fine facial expressions.

Additionally, the present invention enables more accurate recognition of low-intensity facial expressions that previously could not be accurately recognized.

Figure 1 is an exemplary diagram showing the schematic configuration of an facial expression intensity recognition device according to an embodiment of the present invention.
Figure 2 is a flowchart shown to explain a method for recognizing facial expression intensity according to an embodiment of the present invention.
FIG. 3 is an example diagram schematically shown to explain a method of configuring a learning dataset for fine facial expression recognition in which the intensity of facial expressions is tagged in FIG. 1.
FIG. 4 is an exemplary diagram showing the results of an expression intensity recognition test using an expression intensity recognition device in FIG. 1.

Hereinafter, an embodiment of the facial expression intensity recognition device according to the present invention will be described with reference to the attached drawings. In this process, the thickness of lines or sizes of components shown in the drawings may be exaggerated for clarity and convenience of explanation. In addition, the terms described below are terms defined in consideration of functions in the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, definitions of these terms should be made based on the content throughout this specification.

In general, facial expression recognition technology is a technology that recognizes what kind of expression a face detected from an input image has. Various methods have previously been proposed for facial expression recognition, and facial expressions are inferred by analyzing each facial action unit. Alternatively, landmarks of key parts of the face were extracted and used for facial expression recognition.

Meanwhile, with the recent development of deep learning technology, methods of applying image features extracted through the Convolutional Neural Network (CNN) technique to facial expression recognition have been proposed, and the accuracy of facial expression recognition has been greatly improved.

For reference, the number of facial expressions that can be recognized in existing facial expression recognition technology varies depending on the composition of the training data, but commonly used facial expression recognition models or public datasets usually have about 6 to 8 representative facial expressions (e.g., angry). , joy, sadness, surprise, disgust, dislike, etc.) are used.

Therefore, previously, it was impossible to recognize intensity changes (e.g., very happy -> slightly happy) or subtle facial expressions (i.e., low-intensity facial expressions) in the same facial expression, and as a result, learners, patients, customers, interviewers, and drivers It had limited utility in commercialization areas such as monitoring systems, depression diagnosis, and interaction with artificial intelligence.

Meanwhile, conventional methods for fine facial expression recognition generally consist of a CNN-based main network that uses images as input, and a DNN (or RNN)-based auxiliary network that uses feature point information as input.

However, the feature point-based deep learning model is very simple compared to the main network, so it is difficult to expect good performance, and in particular, expressions with weak (or low) intensity (e.g., a little surprise, a little dislike, etc.) are compared to expressions with high intensity. Compared to this, because the movement of the main parts of the face is not large, the distribution of feature points is similar for each expression, which causes a problem in that recognition accuracy is significantly reduced. In addition, there is a problem in that errors are propagated in the process of extracting feature points themselves, reducing recognition accuracy.

Therefore, this embodiment describes a technology that allows recognition of fine-grained, fine-grained facial expressions by recognizing the intensity of various facial expressions when recognizing facial expressions.

Figure 1 is an exemplary diagram showing the schematic configuration of an facial expression intensity recognition device according to an embodiment of the present invention.

As shown in FIG. 1, the facial expression intensity recognition device according to this embodiment includes an image pre-processing module 110, a processor 120, and a fine facial expression recognition module 130.

The image pre-processing module 110 detects the face from the input image, cuts out only the face portion, and converts it to the input standard of the processor for fine facial expression recognition.

At this time, the preprocessing process may include changing the resolution of the input image, and image augmentation or data augmentation processes such as rotation and inversion.

The processor 120 extracts features of the face image based on CNN.

For example, the processor 120 may extract features of a face image using a designated network (e.g., VGGNet, Resnet, etc.).

Additionally, the processor 120 shares a CNN backbone network and simultaneously performs facial expression classification and facial expression intensity classification.

For example, the processor 120 can perform multi-task learning and can be implemented with a designated network (eg, fully-connected neural network, dense neural network, etc.).

In addition, the processor 120 performs learning in advance using a CNN network and a dataset containing tagging information for facial expression and facial expression intensity recognition (i.e., a learning dataset for fine facial expression recognition).

In this embodiment, the “learning dataset for fine facial expression recognition” consists of a dataset that can perform both facial expression recognition and facial expression intensity recognition.

Accordingly, the existing "learning dataset for facial expression recognition" simply tags the expression information for each face image, but the "learning dataset for fine expression recognition" in this embodiment is In order to more accurately recognize expression intensity, the expression intensity information of each face image is configured to be additionally tagged (see Figure 3).

For example, in this embodiment, the processor 120 selects 10 types of facial expressions (e.g., neutral, angry, surprise, fear, disgust, sadness, happy, dislike, sleepy, calm) and 4 levels of intensity (e.g., level 1, level 2). , Step 3, Step 4), assuming that the "learning dataset for micro-expression recognition" has been formed and learned, it is possible to recognize 40 types of micro-expressions through 10 types of facial expression classification and 4-step intensity classification.

The micro-expression recognition module 130 maps (recognizes) the facial expressions of the input image to various micro-expressions based on the results of facial expression classification and intensity classification in the processor 120 and outputs them.

Figure 2 is a flowchart shown to explain a method for recognizing facial expression intensity according to an embodiment of the present invention.

Referring to FIG. 2, the image pre-processing module 110 detects a face from an input image (S101), cuts out only the face part detected from the input image, and the processor 120 performs fine facial expression recognition according to a designated algorithm. Image preprocessing is performed according to the input standard (S102).

For example, in order to improve the facial expression recognition performance of the processor 120, the image pre-processing may include image pre-processing processes such as changing the resolution of the input image, image augmentation such as rotation and inversion, or data augmentation. .

When image preprocessing is performed as described above, the processor 120 extracts features of the face image based on CNN (S103).

For example, the processor 120 may extract features of a face image using a designated network (e.g., VGGNet, Resnet, etc.).

When the features of the face image are extracted as described above, the processor 120 shares the CNN backbone network and performs facial expression classification and facial expression intensity classification simultaneously (S104, S105), thereby performing the facial expression classification and intensity classification. Based on the results, fine facial expressions are recognized and output (S106).

At this time, the facial expression classification and intensity classification algorithms performed by the processor 120 may be implemented based on a designated network (e.g., fully-connected neural network, dense neural network, etc.).

FIG. 3 is an example diagram schematically shown to explain a method of configuring a learning dataset for fine facial expression recognition in which the intensity of facial expressions is tagged in FIG. 1.

For example, in this embodiment, the learning dataset for fine expression recognition shows face images to different people (e.g., id1, id2, id3), tags the expression felt by each person in "recommend_exp", and additionally each person It is constructed by tagging the intensity of each thinking facial expression with “intensity.”

Accordingly, referring to FIG. 3, with respect to the face images shown to people (e.g., id1, id2, and id3), the expression felt by each person (e.g., id1, id2, and id3) is the same as “angry,” but each person (e.g., id1, id2, and id3) feels the same as “angry.” It can be seen that the intensity of the facial expressions (e.g. id1, id2, id3) feels slightly different (e.g. 3, 3, 4).

Therefore, the processor 120 learns a “training dataset for fine expression recognition” that can perform both facial expression recognition and facial expression intensity recognition, thereby recognizing facial expressions and facial expression intensity at the same time to recognize even finely segmented facial expressions. It becomes possible.

FIG. 4 is an exemplary diagram showing the results of an expression intensity recognition test using an expression intensity recognition device in FIG. 1.

The test using this example applies dataset learning of 10 types of facial expressions and 4 levels of intensity. As shown in FIG. 4, the face part of the input image is detected and displayed as a bounding box, and 10 types of facial expressions are selected from the smiling expressions. Happiness was recognized among the facial expressions, and the intensity was recognized as level 3, which is higher than the middle among levels 1, 2, 3, and 4, and the results are shown.

That is, in this embodiment, as a result of learning using a learning dataset for fine facial expression recognition, even fine facial expressions can be recognized by classifying facial expressions and their intensity from a single input image.

As described above, this embodiment, unlike the conventional method of recognizing expression intensity using landmarks from face images, additionally collects information on the intensity of expression felt directly by a person when looking at the face, and sets this as a dataset. By configuring and learning it and then using it to recognize facial expressions, there is an effect of enabling more accurate recognition of low-intensity expressions (i.e., subtle expressions) that previously could not be accurately recognized.

In addition, this embodiment shares the CNN backbone network and learns various expression intensities, including fine expressions, through multi-task learning that simultaneously performs facial expression classification and facial expression intensity classification. It has the effect of allowing the intensity to be recognized.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely exemplary, and those skilled in the art will recognize that various modifications and other equivalent embodiments are possible therefrom. You will understand. Therefore, the scope of technical protection of the present invention should be determined by the scope of the patent claims below.

110: Image pre-processing module
120: processor
130: Fine facial expression recognition module

Claims (1)

An image pre-processing module that performs image pre-processing of the input image;
a processor that processes the image on which the image preprocessing has been performed according to a designated algorithm to classify facial expressions and facial expression intensity; and
An facial expression intensity recognition device comprising a fine facial expression recognition module that recognizes and outputs facial expressions of the input image in granular stages based on the results of the facial expression classification and facial expression intensity classification.

Images