KR20200000824A - Method for recognizing facial expression based on deep-learning model using center-dispersion loss function - Google Patents

Abstract

The present invention relates to a technique regarding video processing using deep learning, which is a method for recognizing a facial expression. A plurality of training images regarding a facial expression is received to extract a training feature so as to learn a classification model of an artificial neural network including a loss function, and a target image including a face is received to extract a target feature to use the learned classification model of the artificial neural network to classify the target feature, so that the loss function is configured to combine a first loss function increasing intra-class variation and a second loss function increasing an inter-class difference while the facial expression of the target image is identified.

Description

Method for recognizing facial expression based on deep-learning model using center-dispersion loss function}

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to image processing using deep learning, and more particularly, to a method and apparatus for classifying facial expressions using classification loss functions and identifying corresponding emotions.

In everyday interactions, people express their emotions primarily through facial expressions, along with voices, gestures and gestures. Understanding emotions and knowing how to respond to people's expressions facilitates interaction. Accordingly, facial expression recognition has become important in areas of computer science, philosophy, neuroscience and other related fields.

Facial expressions have been studied for many years in the field of research, and they are attracting attention because of their importance and the application of Human-Computer Interaction (HCI). The goal of facial expression research is regarded as a problem of classifying facial expressions into seven basic human emotions: anger, disgust, fear, happiness, neutrality, sadness and surprise. In processing these facial expressions through a computer, the lack of facial expression data that varies by facial appearance or head posture, lighting, and obstruction, large dissimilarities in the same classification, and large similarities between different classifications face the problem of facial expression recognition. Is one of the problems.

Many manual algorithms have been proposed to classify facial expressions over the years. However, applying deep learning frameworks has proven to be more efficient than manual methods. To this end, facial features are extracted from static images or video frames representing changes in facial expressions.

Because of the lack of facial expression data, four types of facial expression datasets can be collected and combined to learn deep neural networks. You can also learn from custom combination datasets and then test neural networks in the FER2013 test dataset. To learn facial expressions, traditional Convolutional Neural Networks (CNN) used softmax loss for classification problems. However, it was pointed out that Softmax loss is not easy to identify by classification and can not reduce the similarity problem between classification and classification in classification.

Shan, C., Gong, S., & McOwan, P. W. (2009). Facial expression recognition based on local binary patterns: A comprehensive study. Image and Vision Computing, 27 (6), 803-816.

The technical problem to be solved by the present invention is to solve the problem that the similarity between the classification and the similarity in the classification due to the characteristics of the facial expression dataset in performing the face recognition in the conventional computer vision technology, due to the characteristics of the conventional classification loss function It is intended to overcome the weakness that it is difficult to reduce the similarity between classifications in situations where classification is difficult to distinguish or when different clusters are closely overlapped.

In order to solve the above technical problem, the facial expression recognition method according to an embodiment of the present invention, the facial expression recognition apparatus receives a plurality of training images related to the facial expression to extract a training feature (feature) includes a loss function Training a classification model of an artificial neural network; And receiving, by the facial expression recognition apparatus, a target image including a face, extracting a target feature, and classifying the target feature using a trained classification model of the artificial neural network to identify the facial expression of the target image. And the loss function is configured by combining a first loss function that increases intra-class variation and a second loss function that increases inter-class difference.

In the facial expression recognition method according to an embodiment, the artificial neural network is a convolutional neural network (CNN), and the second loss function calculates a different classification by calculating a mean square difference between the centers of the respective classifications. Center Dispserion Loss function that maximizes the distance between centers. In addition, the first loss function may be configured as a softmax loss function.

In the facial expression recognition method according to an embodiment, the loss function is configured to perform the first loss function and the second loss function using a hyperparameter set in consideration of a balance between the first loss function and the second loss function. It can be calculated by linear combination.

In the facial expression recognition method according to an embodiment of the present disclosure, the training of the classification model of the artificial neural network may include receiving a plurality of training images collected from a lab environment and a wild environment with respect to the facial expression. Extracting: generating a detection result through a landmark detector on the extracted training feature; Training a deep network structure composed of a plurality of convolution layers using the generated detection result; And fine-tuning the classification model of the artificial neural network by re-learning a pre-training model through transfer learning.

In order to solve the above technical problem, the facial expression recognition method according to another embodiment of the present invention, the facial expression recognition apparatus receives a plurality of training images related to the facial expression to extract a training feature (feature) includes a loss function Training a classification model of an artificial neural network; And receiving, by the facial expression recognition apparatus, a target image including a face, extracting a target feature, and classifying the target feature using a trained classification model of the artificial neural network to identify the facial expression of the target image. Wherein the loss function comprises: a first loss function that increases intra-class variation, a second loss function that increases inter-class difference, and an agent that reduces change in classification. Composed of three loss functions.

In a facial expression recognition method according to another embodiment, the artificial neural network is a convolutional neural network (CNN), and the second loss function calculates a different mean by calculating a mean square difference between the centers of the respective classifications. Center Dispserion Loss function that maximizes the distance between centers, wherein the third loss function calculates the sum of the squared distances between the corresponding centers in the sample and the feature space, thereby calculating the distance between the deeper features of the respective classifications and the corresponding classification centers. It can be configured as a center loss function that minimizes In addition, the first loss function may be configured as a softmax loss function.

In the facial expression recognition method according to another embodiment, the loss function may include the second loss function and the third loss using a first hyperparameter set in consideration of a balance between the second loss function and the third loss function. A linear loss of the combined loss function, and a linear loss of the first loss function and the combined loss function using a second hyperparameter set in consideration of the balance of the first loss function and the combined loss function. Can be.

In the facial expression recognition method according to another embodiment, the training of the classification model of the artificial neural network may include receiving a plurality of training images collected from a lab environment and a wild environment with respect to the facial expression. Extracting: generating a detection result through a landmark detector on the extracted training feature; Training a deep network structure composed of a plurality of convolution layers using the generated detection result; And fine-tuning the classification model of the artificial neural network by re-learning a pre-training model through transfer learning.

Meanwhile, the following provides a computer-readable recording medium having recorded thereon a program for executing the above-described facial expression recognition methods on a computer.

In order to solve the above technical problem, the facial expression recognition apparatus according to another embodiment of the present invention, the input unit for receiving a plurality of training images and a target image including a face; A memory for learning a facial expression and storing a program for recognizing the facial expression of the target image using the learned model; And a processor for executing a program stored in the memory, wherein the program stored in the memory extracts a training feature from an input training image to train a classification model of an artificial neural network including a loss function, and And receiving a target image including a target image, extracting a target feature, and classifying the target feature using a trained classification model of the artificial neural network, thereby identifying a facial expression of the target image. It consists of combining a first loss function that increases intra-class variation and a second loss function that increases inter-class difference.

In a facial expression recognition apparatus according to another embodiment, the artificial neural network is a convolutional neural network (CNN), the first loss function is a softmax loss function, and the second loss. The function may consist of a Center Dispserion Loss function that maximizes the distance between different classification centers by calculating the mean square difference between the centers of the respective classifications.

In the facial expression recognition apparatus according to another embodiment, the loss function is the first loss function and the second loss function using a hyperparameter set in consideration of a balance between the first loss function and the second loss function. Can be calculated by linearly combining

In the facial expression recognition apparatus according to another embodiment, the loss function may further include a third loss function for reducing a change in classification. In addition, the third loss function may be configured as a center loss function that minimizes the distance between the depth feature of each classification and the corresponding classification center by calculating the sum of the squared distances between the sample and the corresponding center in the feature space. . Furthermore, the loss function may linearly combine the second loss function and the third loss function using a first hyperparameter set in consideration of the balance of the second loss function and the third loss function to form a combined loss function. The first loss function and the combined loss function may be calculated by linearly combining the first loss function and the combined loss function using a second hyperparameter set in consideration of the balance between the first loss function and the combined loss function.

In the facial expression recognition apparatus according to another embodiment, the program stored in the memory receives a plurality of training images collected from a lab environment and a wild environment with respect to the facial expression, and extracts a training feature, Generating a detection result through a landmark detector on the extracted training feature, and using the generated detection result, learning a deep network structure composed of a plurality of convolution layers; By fine-tuning the classification model of the artificial neural network by re-learning a pre-training model through transfer learning, a command for training the classification model of the artificial neural network may be performed. Can be.

Embodiments of the present invention provide a more balanced and higher classification accuracy by introducing a central variance loss function that can reduce both similarity between classifications and similarities between classifications, resulting in better discrimination of deep facial features. As a result, facial expression recognition performance improved.

1 is a diagram illustrating a deep neural network structure for processing facial expression recognition.
FIG. 2 is a diagram for explaining the comparison between softmax loss and center loss.
3 is a flowchart illustrating a method of recognizing a facial expression based on a deep learning model using a central variance loss function according to an embodiment of the present invention.
4A is a diagram illustrating two types of datasets: an experimental environment (performed in a laboratory) and a real environment (collected in a real environment).
4B-4E illustrate datasets for training.
5 is a flowchart illustrating a process of learning a classification model of an artificial neural network in the facial expression recognition method of FIG. 3 according to an embodiment of the present invention in more detail.
6 is a diagram illustrating a VGGFace model as an example of a classification model.
7 is a block diagram illustrating an apparatus for recognizing a facial expression based on a deep learning model using a central variance loss function according to an embodiment of the present invention.

Prior to describing embodiments of the present invention, various element technologies that can be utilized in recognizing facial expressions in a deep learning model are introduced, and embodiments of the present invention to solve the weaknesses of the prior art discussed above. The technical measures suggested by them are introduced sequentially.

Facial expressions can be empirically evaluated based on statistical local features, Local Binary Patterns (LBP) for individual-independent facial expression recognition. Boosted-LBP can be formulated to extract the most discreet LBP features. Recognition performance can be achieved using a Support Vector Machine (SVM) classifier with Boosted-LBP features.

In addition, Gabor wavelets, Scale Invariant Feature Transform (SIFT) features, histograms of Oriented Gradients (HOG), histograms of Local Binary Patterns (LBP), and local phase quantization The histogram of Quantization (LPQ) and the Local Gabor Binary Patterns (LGBP) are some of the most successful human design features for facial recognition.

Meanwhile, a deep neural network structure may be utilized to process facial expression recognition for a number of well-known standard facial datasets. 1 is a diagram illustrating a deep neural network structure for processing facial expression recognition. Each deep network of the deep neural network includes two convolutional layers followed by max pooling, and a deep network. It can consist of four Inception layers. Such a network can classify images into seven basic expressions.

Previously, when using a conventional classification loss function, in particular Softmax Loss, it has been pointed out that the classification is not easy to distinguish by classification or the intra-class variation is increased. To overcome this, we can consider the center loss.

FIG. 2 is a diagram for explaining a comparison of softmax loss and center loss, in which (a) shows softmax loss and (b) shows center loss.

The center loss function can learn the center of the deep features of each class at the same time and minimize the distance between the deep features and the center of the class. More effective than loss. In other words, minimizing the center loss tends to reduce intra-class variation of deep features. The function for such a central loss is expressed by Equation 1 below.

In Equation 1, the center loss can be calculated from the sum of the squared distances between the sample and the corresponding center in the feature space.

In training the classification model, using only softmax loss results in high intra-class variation, whereas using only center loss results in deeply learned features and centers of zero. Thus, one might consider a strategy that combines softmax and center losses to fine-tune the classification model VGGface to clarify the in-depth features for similarity between classifications and dissimilarities.

The total loss function combined for neural network learning can be presented as Equation 2 below, and the deep network can be trained to obtain deep features with reduced variation in classification and reduced similarity between classifications.

In Equation 2, L _s is a softmax loss function, L _c is a central loss function, and hyper parameter λ is used to balance two terms (loss functions).

In the embodiments of the present invention, an in-depth learning framework was applied to classify seven basic facial expressions such as anger, disgust, fear, happiness, neutrality, sadness, and surprise. However, in order to reduce the similarity between the change in the classification and the classification of the facial expression classification, it was found that the above equations 1 and 2 alone are not sufficient in the classification performance. For example, clusters of different classifications can be formed in some overlapping region, where the central loss function can reduce changes in the classifications but does not perform well in reducing similarity between classes. Found. Accordingly, embodiments of the present invention described below propose a center dispersion loss function as a new loss function capable of simultaneously reducing both the change in classification and the similarity between classifications of the facial expression classification.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, in the following description and the accompanying drawings, detailed descriptions of well-known functions or configurations that may obscure the subject matter of the present invention will be omitted. In addition, the term 'comprising' a certain component throughout the specification means that it may further include other components, without excluding other components unless specifically stated otherwise.

In addition, terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms may be used for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "include" are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof that is described, and that one or more other features It should be understood that it does not exclude in advance the possibility of the presence or addition of numbers, steps, operations, components, parts or combinations thereof.

Unless specifically defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as meanings consistent with the meanings in the context of the related art, and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. .

3 is a flowchart illustrating a deep learning model-based facial expression recognition method using a central variance loss function according to an embodiment of the present invention, and learning based on a convolutional neural network (CNN). To perform.

In operation S310, the facial expression recognition apparatus receives a plurality of training images regarding a facial expression, extracts a training feature, and trains a classification model of an artificial neural network including a loss function. In this case, the loss function includes a center dispersion loss function that increases an inter-class difference. Such a loss function may further comprise at least one or more other loss functions in addition to the central variance loss function to form a total loss function by combining them, and more specific embodiments of the total loss function will be described later.

A loss function is a function that calculates a loss value by defining a difference between a desired result and an operation result as a loss value, and the feature or vector passed to the current loss function is for labeled training. It's from the image. Therefore, since both the correct result and the actual calculation result are known, an error (or loss value), which is a difference between the correct result and the calculation result, can be obtained.

Now, the classification loss function classifies facial expressions using the extracted training features, and has the difference between the facial expressions derived from the training features extracted through the training features and the actual facial expressions. As a classification loss function, a Softmax function or a sigmoid function has been conventionally used. However, as described above, embodiments of the present invention derive a center dispersion loss function from a classification. Implemented so that the difference (inter-class difference) can be judged more clearly.

On the other hand, in the learning process of step S310, because the lack of facial expression data can combine a plurality of facial expression dataset for higher learning accuracy. In-depth face verification models such as VGGFace or Facenet trained with large datasets can take advantage of transfer learning and derive differential neural network losses specialized for facial expression recognition.

In operation S320, the apparatus for recognizing a facial expression receives a target image including a face, extracts a target feature, and classifies the target feature using a trained classification model of the artificial neural network to identify the facial expression of the target image. .

(1) data collection

4A is a diagram illustrating two types of datasets: an experimental environment (performed in a laboratory) and a real environment (collected in a real environment).

By combining existing datasets, we collected larger training data and collected key facial features. Then, three kinds of landmark detectors were applied: dlib face detector (Multi-view HOG), HOG + Adaboost and Haar + Adaboost. From this, only the best detection result was selected, and the number of images was increased by horizontal inversion and finally resized to 224 × 224 pixels.

Hereinafter, an example of a data set for training will be introduced with reference to FIGS. 4B to 4E. The facial expression learning dataset may be configured by combining the SFEW, FER2013, CK + and KDEF datasets.

Fer2013 (Wild)

Referring to FIG. 4B, the FER2013 dataset includes 28,709 training images, 3,589 validation (public dataset) images and 3,589 trial (personal dataset) images. The size of the dataset is 48x48 and represented in grayscale.

SFEW  (Wild)

Referring to FIG. 4C, the static facial expressions in the wild (SFEW) dataset is composed of frames selected from the Acted Facial Expressions in the Wild (AFEW) and includes 700 images taken from a movie.

KDEF  (Lab)

Referring to FIG. 4D, the Karolinska Directed Emotional Faces (KDEF) dataset is a set of 4,900 images of human facial expressions on emotions. The dataset contains 70 subjects representing seven emotional expressions, each of which was shot twice from five angles.

CK + (Lab)

Referring to FIG. 4E, the Extended Cohn-Kanade (CK +) dataset consists of a total of 593 sequences across 123 subjects, with a total of eight classifications (ie neutral, angry, disdain, disgusted, frightened, happy, sad, Surprise).

(2) Transfer Learning

5 is a flowchart illustrating in more detail a process of learning a classification model of an artificial neural network (S310) in the facial expression recognition method of FIG. 3 according to an exemplary embodiment of the present invention.

In this process of learning a classification model of an artificial neural network (S310), first, a plurality of training images collected from a lab environment and a wild environment regarding facial expressions are input, and a training feature is extracted. In operation S312, a detection result is generated through a landmark detector for the training feature. Then, the deep network structure composed of a plurality of convolution layers is trained using the detection result generated in step S313, and the learning is completed through transfer learning in step S314. By re-learning the model (Pre-Training Model), it is possible to fine-tune the classification model of the artificial neural network. The landmark detector may include at least one of a dlib face detector (Multi-view HOG), HOG + Adaboost, and Haar + Adaboost, and the training image may include at least two or more data sets of SFEW, FER2013, CK +, and KDEF. The combination model may be configured as a VGGFace model.

Many models of machine learning are efficient assuming that the data you are applying has the same distribution as the data you are learning. When solving a new problem, if the distribution of data changes, the existing statistical model must be rebuilt with new data, which is expensive. Therefore, a well-trained model is already available, and it is desirable to use Transfer Learning, especially when trying to solve a problem similar to that model.

Transfer learning can use deep learning only as a feature extractor, and can use the extracted features to learn other models. When you create a new model using an existing model, you can speed up learning and make better predictions. Pre-training models such as VGG, ResNet, and gooGleNet have already been pre-trained to perform fine-tuning on the user's desired learning through transition learning.

FIG. 6 illustrates a VGGFace model as an example of a classification model, wherein the user-defined training dataset fine tunes the VGGFace model using Keras on Python. As a result of evaluation in the FER experimental set, the learning accuracy was 93.5% and the test accuracy was 63.2%.

(3) Loss Functions

Hereinafter, two loss functions proposed by the embodiments of the present invention will be introduced. Both are designed to take into account the Center Dispersion Loss, which increases the inter-class difference, but there are some differences in the details of the total loss function.

3-1) First Example : Center Dispersion Loss

Center loss learns the centers of the deeper features of each class simultaneously and minimizes the distance between the deeper features and the centers of the class. Minimizing the center loss tends to reduce changes in the classification of deep features, making it easier to identify by category than Softmax loss. As introduced through Equation 1 above, the center loss may be calculated from the sum of the squared distances between the corresponding centers in the sample space and the feature space.

However, in some cases clusters of different classifications may overlap each other. Problems have been found that the central loss reduces changes in the classifications but does not have great success in reducing the similarities between the classifications. Accordingly, embodiments of the present invention proposed a Center Dispersion Loss function that maximizes the distance between classification centers to increase the difference between classifications.

Equation 3 below can calculate the center variance loss by calculating the mean square difference between the centers.

Referring to Equation 3, c _j and c _k each represent a center belonging to a different classification, and the difference between the classifications may be reflected in the loss function through the mean square difference between them.

On the other hand, using only Softmax Loss may increase intra-class variation, and using only Center Dispersion Loss may reduce the center to '0'. Thus, we can combine the softmax loss with the central dispersion loss proposed above to fine-tune VGGFace to clarify non-similarity in classification.

Note that minimizing the central variance loss tends to reduce the similarity between the classifications of the deep features, but is not successful in reducing the changes in the classifications, so that the new total loss function for neural network learning, Was derived.

Where L _s is the softmax loss, L _cD is the central variance loss, and the hyper parameter λ is used to equalize the two terms (loss functions).

For example, the VGGFace learning accuracy fine-tuned with λ = 0.1 was 97.29% and the FER2013 test accuracy was 70.16%.

In summary, the total loss function in the first embodiment is constructed by combining a first loss function that increases intra-class variation and a second loss function that increases inter-class difference. It became. In this case, the first loss function is a Softmax Loss function, and the second loss function calculates a mean square difference between the centers of the respective classifications to maximize the distance between different classification centers. Dispserion Loss) function is preferable. In addition, the total loss function is preferably calculated by linearly combining the first loss function and the second loss function using a hyperparameter set in consideration of the balance between the first loss function and the second loss function. .

3-2) 2nd Example : Combined  Combined Center Dispersion Loss

Minimizing the central loss tends to reduce changes in the classification of deep features, but does not reduce the similarity between classifications. In addition, minimizing the central dispersion loss tends to reduce the similarity between classifications of deep features, but is not effective in reducing changes in classification. Therefore, the present embodiment proposes a technique of combining both the center loss and the center variance loss together with the softmax loss in order to reduce the change in the classification and the similarity between the classifications.

First, the combined center-dispersion loss (L _cc ) may be derived as shown in Equation 5 below.

Where L _c is the central loss, L _cD is the central variance loss, and the hyperparameter λ ₁ (eg, can be set to λ ₁ = 0.1) is used to balance the two terms (loss functions) It became.

Now the total loss function for training the neural network can be derived as in Equation 6 below.

Where L _s is the softmax loss, L _cc is the combined central variance loss, and the hyperparameter λ (e.g. can be set to λ = 0.1) is used to balance the two terms (loss functions). Was used.

As a result of experiment, VGGFace was finely tuned to achieve 97.9% learning accuracy and 70.89% accuracy of FER2013 test.

In summary, the total loss function in the second embodiment is a first loss function that increases intra-class variation, a second loss function that increases inter-class difference, and a classification within the classification. It was constructed by combining a third loss function that reduces the change. In this case, the first loss function is a Softmax Loss function, and the second loss function calculates a mean square difference between the centers of the respective classifications to maximize the distance between different classification centers. Dispserion Loss) function, wherein the third loss function is a center loss function that minimizes the distance between the deep feature of each classification and the corresponding classification center by calculating the sum of the squared distances between the sample and the corresponding center in the feature space. It is preferable. In addition, the total loss function is a combined loss function by linearly combining the second loss function and the third loss function using a first hyperparameter set in consideration of the balance between the second loss function and the third loss function. It is preferable to calculate and calculate by linearly combining the first loss function and the combined loss function using a second hyperparameter set in consideration of the balance of the first loss function and the combined loss function.

FIG. 7 is a block diagram illustrating a deep learning model based facial expression recognizing apparatus 700 using a central variance loss function according to an embodiment of the present invention. Each step of the facial expression recognition method of FIG. 3 is configured in hardware. Reconstructed in terms of Therefore, in order to avoid duplication of description here, only the functions and operation operations of each configuration will be outlined.

The input unit 10 is configured to receive a plurality of training images related to facial expressions and a target image including a face.

The memory 30 is a component for learning a facial expression and storing a program for recognizing the facial expression of the target image using the learned model, and the processor 20 is a component for executing the program stored in the memory 30. .

More specifically, the program stored in the memory 30 extracts a training feature from the input training image, trains a classification model of an artificial neural network including a loss function, and receives a target image including a face. And extracting a feature and classifying the target feature using the trained classification model of the artificial neural network to identify a facial expression of the target image. In this case, the loss function is configured by combining a first loss function that increases intra-class variation and a second loss function that increases inter-class difference.

This artificial neural network is a convolutional neural network (CNN), the first loss function is a Softmax Loss function, and the second loss function is the mean square difference between the centers of the respective classifications. By calculating, it can be configured as a function of Center Dispserion Loss that maximizes the distance between different classification centers. The loss function may be calculated by linearly combining the first loss function and the second loss function using a hyperparameter set in consideration of a balance between the first loss function and the second loss function.

Meanwhile, the loss function may further include a third loss function for reducing the change in the classification. In this case, the third loss function may be configured as a center loss function that minimizes the distance between the depth feature of each classification and the corresponding classification center by calculating the sum of the squared distances between the sample and the corresponding center in the feature space. .

The loss function may be configured to linearly combine the second loss function and the third loss function using a first hyperparameter set in consideration of the balance between the second loss function and the third loss function to form a combined loss function. The first loss function and the combined loss function may be calculated by linearly combining the first loss function and the combined loss function using a second hyperparameter set in consideration of the balance between the first loss function and the combined loss function.

Furthermore, the program stored in the memory 30 receives a plurality of training images collected from a lab environment and a wild environment regarding facial expressions, extracts training features, and lands the extracted training features. A detection result is generated through a mark detector, a deep network structure composed of a plurality of convolution layers is trained using the generated detection result, and transfer learning is performed through transfer learning. By fine-tuning the classification model of the artificial neural network by re-learning a pre-training model, the instruction for training the classification model of the artificial neural network may be performed.

(4) experimental results

Experimental results for the combined data set yielded the results shown in Table 1 in the FER2013 test set.

Next, Table 2 compares the method proposed in the embodiments of the present invention with other deep learning learning based methods of the FER2013 test results.

Table 3 below shows the best results for the FER2013 test set without deep learning learning.

According to embodiments of the present invention, a new central variance loss function for CNN is proposed to improve discrimination of learned deep facial features. Experimental results for the FER2013 dataset show that the proposed central variance loss and the combined central variance loss both show improved performance for facial recognition. The proposed loss functions have the effect of reducing the change in classification by the similarity between classifications, compared with the case of using either softmax loss or central loss.

On the other hand, embodiments of the present invention can be implemented in a computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored.

Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion. And functional programs, codes and code segments for implementing the present invention can be easily inferred by programmers in the art to which the present invention belongs.

The present invention has been described above with reference to various embodiments thereof. Those skilled in the art will understand that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

700: facial expression recognition device
10: input unit
20: processor
30: memory

Claims (20)

Receiving, by the facial expression recognition apparatus, a plurality of training images related to the facial expressions, extracting training features, and training a classification model of an artificial neural network including a loss function; And
And receiving, by the facial expression recognition apparatus, a target image including a face, extracting a target feature, and classifying the target feature using a learned classification model of the artificial neural network to identify the facial expression of the target image. and,
The loss function is,
And a first loss function that increases intra-class variation and a second loss function that increases inter-class difference. The method of claim 1,
The artificial neural network is a convolutional neural network (CNN),
Wherein the second loss function is a Center Dispserion Loss function that maximizes the distance between different classification centers by calculating a mean squared difference between the centers of the respective classifications. The method of claim 2,
Wherein the first loss function is a Softmax Loss function. The method of claim 1,
The loss function is,
And calculating the linear expression by linearly combining the first loss function and the second loss function using a hyperparameter set in consideration of the balance between the first loss function and the second loss function. The method of claim 1,
Training the classification model of the artificial neural network,
Extracting training features by receiving a plurality of training images collected from a lab environment and a wild environment with respect to a facial expression:
Generating a detection result through a landmark detector on the extracted training feature;
Training a deep network structure composed of a plurality of convolution layers using the generated detection result; And
Fine-tuning the classification model of the artificial neural network by re-learning a pre-training model through transfer learning. The method of claim 5,
The landmark detector includes at least one of a dlib face detector (Multi-view HOG), HOG + Adaboost and Haar + Adaboost,
The training image is configured by combining at least two or more of the data set of SFEW, FER2013, CK + and KDEF,
The classification model is a VGGFace model, facial expression recognition method. Receiving, by the facial expression recognition apparatus, a plurality of training images related to the facial expressions, extracting training features, and training a classification model of an artificial neural network including a loss function; And
And receiving, by the facial expression recognition apparatus, a target image including a face, extracting a target feature, and classifying the target feature using a learned classification model of the artificial neural network to identify the facial expression of the target image. and,
The loss function is,
A first loss function that increases intra-class variation, a second loss function that increases inter-class difference, and a third loss function that reduces change in classification. , Facial expression recognition method. The method of claim 7, wherein
The artificial neural network is a convolutional neural network (CNN),
The second loss function is a Center Dispserion Loss function that maximizes the distance between different classification centers by calculating a mean square difference between the centers of respective classifications,
And the third loss function is a center loss function that minimizes the distance between the depth feature of each classification and the center of the classification by calculating the sum of the squared distances between the sample and the corresponding center in the feature space. The method of claim 8,
Wherein the first loss function is a Softmax Loss function. The method of claim 7, wherein
The loss function is,
Calculating a combined loss function by linearly combining the second loss function and the third loss function using a first hyperparameter set in consideration of the balance between the second loss function and the third loss function;
And calculating the linear expression by linearly combining the first loss function and the combined loss function using a second hyperparameter set in consideration of the balance of the first loss function and the combined loss function. The method of claim 7, wherein
Training the classification model of the artificial neural network,
Extracting training features by receiving a plurality of training images collected from a lab environment and a wild environment with respect to a facial expression:
Generating a detection result through a landmark detector on the extracted training feature;
Training a deep network structure composed of a plurality of convolution layers using the generated detection result; And
Fine-tuning the classification model of the artificial neural network by re-learning a pre-training model through transfer learning. The method of claim 11,
The landmark detector includes at least one of a dlib face detector (Multi-view HOG), HOG + Adaboost and Haar + Adaboost,
The training image is configured by combining at least two or more of the data set of SFEW, FER2013, CK + and KDEF,
The classification model is a VGGFace model, facial expression recognition method. A computer-readable recording medium having recorded thereon a program for executing the method of any one of claims 1 to 12. An input unit configured to receive a plurality of training images related to facial expressions and a target image including a face;
A memory for learning a facial expression and storing a program for recognizing the facial expression of the target image using the learned model; And
A processor for executing a program stored in the memory;
The program stored in the memory,
The training feature is extracted from the input training image to train the classification model of the artificial neural network including the loss function, the target image is extracted by receiving the target image including the face, and the classified model of the trained artificial neural network And classifying the target feature to identify a facial expression of the target image.
The loss function is,
And a first loss function that increases intra-class variation and a second loss function that increases inter-class difference. The method of claim 14,
The artificial neural network is a convolutional neural network (CNN),
The first loss function is a Softmax Loss function,
And the second loss function is a center dispserion loss function that maximizes the distance between different classification centers by calculating a mean square difference between the centers of respective classifications. The method of claim 14,
The loss function is,
And a facial expression recognition apparatus calculated by linearly combining the first loss function and the second loss function using a hyperparameter set in consideration of the balance between the first loss function and the second loss function. The method of claim 14,
The loss function is,
And a third loss function for reducing the change in classification. The method of claim 17,
And the third loss function is a center loss function that minimizes the distance between the depth feature of each classification and the corresponding classification center by calculating the sum of the squared distances between the sample and the corresponding center in the feature space. The method of claim 17,
The loss function is,
Calculating a combined loss function by linearly combining the second loss function and the third loss function using a first hyperparameter set in consideration of the balance between the second loss function and the third loss function;
And a facial expression recognition apparatus calculated by linearly combining the first loss function and the combined loss function using a second hyperparameter set in consideration of the balance of the first loss function and the combined loss function. The method of claim 14,
The program stored in the memory,
The training feature is extracted from a plurality of training images collected from a lab environment and a wild environment with respect to a facial expression, and a detection result is generated through a landmark detector on the extracted training feature. The deep network structure composed of a plurality of convolution layers is trained using the generated detection result, and the pre-training model is completed through transfer learning. And fine-tuning the classification model of the artificial neural network by relearning, thereby performing a command for learning the classification model of the artificial neural network.

Images