KR20210018600A - System for recognizing facial expression - Google Patents

Abstract

According to one embodiment of the present invention, a facial expression recognition system simultaneously extracts a local appearance feature and a scene feature having space-time information by using a whole sequence and individual frames from an input video. The scene feature has a change of a general facial expression in an image, and the appearance feature connotes detailed information indicating each frame. According to one embodiment of the present invention, the facial expression recognition system is capable of extracting a facial expression by early fusing a scene feature and an appearance feature, thereby being operated very strongly in a video of an actual environment including a pose change and low illuminance as well as an image of an artificial environment.

Description

Facial expression recognition system {SYSTEM FOR RECOGNIZING FACIAL EXPRESSION}

The present invention relates to a facial expression recognition system, and more particularly, to a facial expression recognition system that enables robust facial expression recognition from data acquired in an actual environment.

Facial expression recognition is a technology field that continues to receive attention in the fields of computer vision and human-computer interaction. Facial expression recognition technology can be used in various fields such as driver monitoring in a vehicle, human-robot interaction, psychology, and digital entertainment.

Although considerable progress has been made in the level of facial expression recognition technology with the recent development of machine learning such as deep learning, there are still difficulties in operating robustly. This is because the actual environment includes various performance degradation factors such as occlusion, complex motion, and low light.

In the early days, facial expression recognition technologies were mainly used to classify the emotions of a person in a single image, but recently, research on analyzing the emotions of a person in a video is also active. Meanwhile, there are various data sets for learning or verifying facial expression recognition based on video.

1 is a diagram showing an example of images of a data set of CK+ (The Extended Cohn-Kanade) used in a conventional facial expression recognition technology, and FIG. 2 is a diagram showing facial expressions appearing in a video captured in an actual environment to be.

For example, as shown in FIG. 1, most of the images in the CK+ (The Extended Cohn-Kanade) data set are images in which subjects look at the front and express unnaturally artificial emotions. Also, since face images of the front face were acquired in a constant illumination environment, poses and illumination levels are not considered in the algorithm development. However, since videos captured in an actual environment as shown in FIG. 2 have various poses and illuminances, a robust facial expression recognition technique is required.

The matters described as the background art are only for enhancing an understanding of the background of the present invention, and should not be taken as acknowledging that they correspond to the prior art already known to those of ordinary skill in the art.

KR 10-2007-0012395 A

Accordingly, an object of the present invention is to provide a facial expression recognition system capable of strongly recognizing facial expressions from data acquired in an actual environment such as various poses and low-light environments.

The present invention as a means for solving the above technical problem,

A first neural network for extracting external features including region information from each frame included in the input video;

A second neural network for extracting scene features including spatiotemporal information from the sequence of the video; And

It provides a facial expression recognition system including a third neural network that performs emotion classification by fusing the external features and the scene features.

An embodiment of the present invention further includes a preprocessor for recognizing a face in the video, cropping and aligning the recognized area, and increasing the length of the video by applying a frame interpolation technique when the video is shorter than a preset reference can do.

In one embodiment of the present invention, the first neural network may be a finely tuned 2D convolutional neural network based on DenseNet.

In one embodiment of the present invention, the first neural network may be a DenseNet having a fully-connected layer.

In one embodiment of the present invention, the DenseNet may be pretrained using the ImageNet data set and then retrained to the FER2013 data set composed of emotion data to be fine tuned.

In one embodiment of the present invention, the second neural network may be a 3D convolutional neural network.

In one embodiment of the present invention, the 3D convolutional neural network may be pretrained with a Sports-1M data set or a kinetic data set.

In one embodiment of the present invention, the 3D convolutional neural network may include an auxiliary classifier including a fully-connected layer, batch normalization, dropout, and ReLU.

In an embodiment of the present invention, in the 3D convolutional neural network, parameters are fixed so that a convolutional block included therein is not learned, and only all connection layers included therein and the auxiliary classifier may be learned.

In one embodiment of the present invention, the first neural network may be pre-trained using emotion-related data, and the second neural network may be pre-trained using behavior recognition-related data.

In an embodiment of the present invention, the third neural network may be a scene-aware recurrent neural network, which is a temporal network in which the scene feature and the external feature are fused at a feature level.

In an embodiment of the present invention, a long short-term memory (LSTM) model may be applied to the scene recognition recurrent neural network.

In an embodiment of the present invention, the Long Short-Term Memory (LSTM) model may use the scene feature extracted from the second neural network as a hidden state.

According to the facial expression recognition system, by effectively extracting temporal change information for an overall scene of a video, it is possible to recognize a facial expression robustly even with low illumination and pose change. As a result, strong facial expression recognition can be achieved not only for specific images acquired in an artificial environment, but also for images acquired in an actual environment.

In addition, according to the facial expression recognition system, it is possible to achieve synergy while preserving unique characteristics of each signal through early fusion.

The effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned can be clearly understood by those of ordinary skill in the art from the following description. will be.

FIG. 1 is a diagram illustrating an example of images of a data set of CK+ (The Extended Cohn-Kanade) used in a conventional facial expression recognition technology.
2 is a diagram illustrating facial expressions appearing in a video captured in an actual environment.
3 is a block diagram of a facial expression recognition system according to an embodiment of the present invention.
4 is a block diagram showing the structure of a DenseNet having two full connection layers employed as a first neural network according to an embodiment of the present invention.
5 is a block diagram showing the structure of a 3D convolutional neural network applied to a second neural network of a facial expression recognition system according to an embodiment of the present invention.
6 is a block diagram showing the structure of an LSTM, which is a model of a scene recognition recurrent neural network applied to a third neural network of a facial expression recognition system according to an embodiment of the present invention.

Hereinafter, a facial expression recognition system according to various embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The facial expression recognition technique applied to the facial expression recognition system according to various embodiments of the present invention may utilize a video signal as two aspects.

One aspect is spatial information in units of frames included in a video signal. Since each frame only has a spatial signal, appearance and geometry information such as a person's expression at a specific point in time can be extracted from the frame. These pieces of information correspond to the details you want to grasp.

Another aspect is a video sequence itself having spatiotemporal information. Since the sequence basically includes temporal information, it provides information on changes in facial expressions and atmosphere. Therefore, the entire sequence corresponds to information on the overall context to be grasped, that is, the overall scene. Various embodiments of the present invention provide a network capable of accurately extracting the above two pieces of information from a moving picture signal and effectively fusing them.

3 is a block diagram of a facial expression recognition system according to an embodiment of the present invention.

Referring to FIG. 3, a facial expression recognition system according to an embodiment of the present invention includes a first neural network 20 for extracting external features including detailed region information from each frame included in a video signal, and a video signal. The second neural network 30 extracting scene features including spatiotemporal information from the video sequence included in the second neural network 30 and the appearance features extracted from the first neural network 20 and the scene features extracted from the second neural network 30 It may be configured to include a third neural network 40 that performs emotion classification.

In addition, the facial expression recognition system according to an embodiment of the present invention may further include a preprocessor 10 that recognizes a face from an input video signal, crops and aligns frames, or interpolates frames.

Hereinafter, components of the facial expression recognition system according to an embodiment of the present invention will be described in more detail. Through the following description, the features of each component and the effect of the facial expression recognition system expressed through organic coupling between components may be more clearly understood.

Pre-treatment unit (10)

The preprocessor 10 is an element for processing the moving picture signal so that it can be easily used by the first and second neural networks 20 and 30 at the rear end thereof.

The preprocessor 10 may perform simple face recognition, alignment, and/or frame interpolation in the video signal.

For example, a wild data set, such as an Acted Facial Expressions in the Wild (AFEW) data set that can be used for training and/or testing of the first and second neural networks 20, 30, exhibits low illumination and occlusion. Since a large number of videos are included, the preprocessor 10 may apply a known means for performing face recognition through convolution, such as multitask cascaded convolutional networks, for robust face detection. . The preprocessor 10 may crop a region recognized after face recognition using a multi-task cascade convolutional network and set it as inputs of the first and second neural networks 20 and 30 at the rear end.

In addition, the preprocessor 10 applies a known frame interpolation technique, such as a separable convolution based frame interpolation technique, when the input video is shorter than a preset criterion such that the temporal feature cannot be derived. You can also increase the length of the input video.

The first neural network (20)

Recently, a transfer learning technique has been proposed in the field of deep learning as a solution to insufficient training data. When the transfer learning technique is applied, it is known that the model trained in the source domain can increase the performance in the target domain. In one embodiment of the present invention, the first neural network 20 is a fine tuned 2D convolutional neural network (CNN) as a transfer learning tool for utilizing learned knowledge of facial expressions. It can be implemented as In one embodiment of the present invention, a feature extracted from the first neural network 20, which is a fine-tuned two-dimensional CNN, will be referred to as an appearance feature.

In particular, an embodiment of the present invention may employ DenseNet as a finely tuned two-dimensional convolutional neural network to effectively extract facial features. DenseNet is published in "G. Huang, Z. Liu, KQ Weinberger, and L. van der Maaten, "Densely connected convolutional networks," in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, no. 2, pp. As a neural network introduced in "3, 2017", according to the above paper, DenseNet is known to average feature maps through a global average pooling (GAP) layer and immediately perform classification. In an embodiment of the present invention, since a feature vector having a lot of information needs to be input to the network, the GAP layer in DenseNet can be replaced by two fully-connected layers (FC layers).

A structure of a DenseNet having two fully-connected layers (FC layers) employed as the first neural network 20 is shown in FIG. 4.

In one embodiment of the present invention, fine tuning of DenseNet can be done in two steps. First, the authentic DenseNet is ImageNet (O. Russakovsky, J. Deng, H. Su, et. al., "ImageNet Large Scale Visual Recognition Challenge," International Journal of Computer Vision, vol. 114, no. 3, pp. 211 -252, 2015.) Pre-trained using the dataset. The pretrained DenseNet is publicly available at the code level. Next, DenseNet is FER2013 (IJ Goodfellow, et. al., "Challenges in representation learning: A report on three machine learning contests," in International Conference on Neural Information Processing, pp. 117-124, 2013.) Fine tuning can be achieved by re-learning. Through these two stages of fine tuning, the knowledge of the first stage can be transferred to the emotion of the second stage. As a result, the fine-tuned DenseNet will be able to work on all test videos in the AFEW data set at the inference stage and extract 4096 dimensions of cosmetic features.

The second neural network (30)

5 is a block diagram showing the structure of a second neural network of a facial expression recognition system according to an embodiment of the present invention.

In an embodiment of the present invention, a 3D convolutional neural network as shown in FIG. 5 may be used as the second neural network 30. Since the 3D convolutional neural network takes the entire sequence of the input video signal as an input, it can play a role of capturing the entire visual scene of the input video sequence. In one embodiment of the present invention, a feature acquired from a 3D convolutional neural network may be used as a kind of scene, and such a feature may be referred to as a scene feature.

In one embodiment of the present invention, C3D (D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, "Learning spatiotemporal features with 3D convolutional networks," which is a neural network known as a 3D convolutional neural network. "in Proceedings of IEEE International Conference on Computer Vision, pp. 4489-4497, 2015.), ResNet3D and ResNeXt3D (K. Hara, H. Kataoka, and Y. Satoh, "Can spatiotemporal 3D CNNs retrace the history of 2D CNNs and ImageNet," in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 18-22, 2018.), etc. may be employed. Like the first neural network 20, transfer learning may be applied to a 3D convolutional neural network selected as the second neural network 30. In general, C3D is a Sports-1M dataset (A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei- Fei, "Large-scale video classification with convolutional neural networks," in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1725-1732, 2014.), and ResNet3D and ResNeXt3D are Kinetic datasets (W. Kay, J. Carreira, K. Simonyan, et al.) al., "The kinetics human action video dataset," arXiv preprint arXiv:1705.06950, 2017.). Like the first neural network 20, the second neural network 30 also uses a pre-trained model, but uses behavioral recognition-related data rather than emotion-related data during pre-training. This is because more dynamic videos can provide richer scene features, and to capture motion information such as changes in facial expressions.

In one embodiment of the present invention, assuming that a convolution block of a three-dimensional convolutional neural network already sufficiently extracts motion information, in the fine tuning process, parameters are fixed so that not all of the convolutional blocks are learned, Only FC layers and auxiliary classifiers can be learned.

Meanwhile, according to an embodiment of the present invention, the auxiliary classifier 31 may be added to the final layer of the second neural network 30, that is, a 3D convolutional neural network. The auxiliary classifier 31 not only mitigates the vanishing gradient phenomenon, but also provides a normalization effect in the learning process, thereby providing stable learning and performance improvement effects. As shown in FIG. 5, the auxiliary classifier 31 may have a structure including an FC layer, batch normalization (BN), dropout, and ReLU, and finally a softmax function may be located.

Third Neural Network (40)

The facial expression recognition system according to an embodiment of the present invention is capable of fusing an entire sequence-based scene feature and an appearance feature that is a detailed regional feature of each video frame at a feature level. A scene-aware recurrent neural network (RNN), which is a temporal network, is applied as the third neural network 40. The scene feature of the 3D convolutional neural network can be used as a context because there is time information that does not exist in individual frames. The method of fusing two features by applying such a scene recognition recurrent neural network corresponds to early fusion, and can be fused while preserving unique information of each feature.

In one embodiment of the present invention, as a model of a scene recognition recurrent neural network, LSTM (Long Short-Term Memory) (S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural computation, vol. 9, no. 8, pp. 1735-1780, 1997.) can be used.

6 is a block diagram showing the structure of an LSTM, which is a model of a scene recognition recurrent neural network applied to a third neural network of a facial expression recognition system according to an embodiment of the present invention.

LSTM, which is easy to process sequence data, can be learned while appropriately adjusting the relationship between current and previous information. When the current input is defined as x _t and the hidden state is defined as h _t , in a typical LSTM, the hidden state is initialized to 0 at the beginning of learning (ie, h ₀ = 0). However, in the LSTM model applied to the scene recognition recurrent neural network according to an embodiment of the present invention, the scene feature extracted from the 3D convolutional neural network, which is the second neural network 30, is simply used as a hidden state. The advantage of this connection is that temporal information that cannot be grasped by individual frames of a video can be grasped by using previous information temporally as an overall visual scene contained in a scene feature. In addition, it can be simply implemented without changing the LSTM formula and structure as shown in FIG. 6. Scene feature v can be summarized as follows

[Equation 1]

[Equation 2]

Here, V represents the video sequence input, F _θ (·) is a function representing the second neural network, that is, a 3D convolutional neural network, and θ represents all learning parameters of the 3D convolutional neural network.

The facial expression recognition system according to an embodiment of the present invention configured and operated as described above may have the following entire learning process.

Learning performed in the facial expression recognition system according to an embodiment of the present invention may be performed in two steps.

In the first step, learning about the DenseNet having the FC layer, which is the first neural network 20, is performed, and as a result of the learning, an appearance feature for a frame of an input video may be extracted.

In the second step, the 3D convolutional neural network having the auxiliary classifier 31 as the second neural network 30 and the scene recognition recurrent neural network SA-RNN as the third neural network 40 may be trained. As described above, the parameters of all layers except the FC layer and the auxiliary classifier 31 of the pre-trained 3D convolutional neural network are fixed to maintain the function of the convolutional layer, so the parameters actually learned are 3D These are the parameters of the FC layer of the convolutional neural network and the auxiliary classifier 31 and the scene recognition recurrent neural network (SA-RNN). In an embodiment of the present invention, a cross-entropy function may be used as a loss function of the entire structure including an auxiliary classifier, and when the cross-entropy function is used, the final loss function L _TOTAL is as follows.

[Equation 3]

Here, L is the loss function of the entire network, L _aux is the loss function of the auxiliary classifier, and λ is a hyper-parameter that determines the reflection ratio of L _aux . Learning can be performed in a manner that minimizes Equation 3, which is the final loss function (objective function).

The facial expression recognition system according to various embodiments of the present invention may effectively extract temporal change information for an overall scene of a moving picture, and thus strongly recognize facial expressions even with low illuminance and pose changes. As a result, strong facial expression recognition can be achieved not only for specific images acquired in an artificial environment, but also for images acquired in an actual environment.

In addition, the facial expression recognition system according to various embodiments of the present invention can be fused so as to exhibit synergy while preserving the unique characteristics of each signal through early fusion.

Although shown and described in connection with specific embodiments of the present invention above, it will be apparent to those of ordinary skill in the art that the present invention can be variously improved and changed within the scope of the claims. .

10: pretreatment unit
20: first neural network (DenseNet with full connection layer)
30: second neural network (three-dimensional convolutional neural network with auxiliary classifier)\
31: secondary classifier
40: third neural network (scene recognition recurrent neural network)

Claims (13)

A first neural network for extracting external features including region information from each frame included in the input video;
A second neural network for extracting scene features including spatiotemporal information from the sequence of the video; And
A facial expression recognition system including a third neural network that performs emotion classification by fusing the external features and the scene features. The method according to claim 1,
Recognizing a face in the video, cropping and aligning the recognized area, and applying a frame interpolation technique when the video is shorter than a preset criterion to increase the length of the video facial expression recognition, characterized in that system. The method according to claim 1,
The first neural network is a fine-tuned two-dimensional convolutional neural network based on DenseNet. The method of claim 3,
The facial expression recognition system, wherein the first neural network is a DenseNet having a fully-connected layer. The method according to claim 3 or 4,
The DenseNet is pre-trained using the ImageNet data set, and then retrained with the FER2013 data set composed of emotion data to fine tune. The method according to claim 1,
The second neural network is a facial expression recognition system, characterized in that the 3D convolutional neural network. The method of claim 6,
The 3D convolutional neural network is a facial expression recognition system, characterized in that pre-trained with a Sports-1M data set or a kinetic data set. The method of claim 6,
The 3D convolutional neural network, a facial expression recognition system comprising an auxiliary classifier including a fully-connected layer, a batch normalization, a dropout, and a ReLU. The method of claim 8,
In the 3D convolutional neural network, parameters are fixed so that the convolutional block included therein is not learned, and only all connection layers included therein and the auxiliary classifier are learned. The method according to claim 1,
The first neural network is pre-trained using emotion-related data, and the second neural network is pre-trained using behavior-recognition-related data. The method according to claim 1,
And the third neural network is a scene-aware recurrent neural network, which is a temporal network in which the scene feature and the external feature are fused at a feature level. The method of claim 11,
The scene recognition recurrent neural network is a facial expression recognition system, characterized in that the LSTM (Long Short-Term Memory) model is applied. The method of claim 12,
The LSTM (Long Short-Term Memory) model uses the scene feature extracted from the second neural network as a hidden state.

Images