KR102060719B1 - System and method for face detection and emotion recognition based deep-learning - Google Patents

Abstract

The present invention relates to a deep learning-based face detection and emotion recognition system and method, wherein the system detects a face by receiving a plurality of images, and detects a face and recognizes the emotion expressed on the face. And a probability calculator configured to derive a current emotion by calculating a emotion based on a probability of emotion transfer based on a first emotion, a second emotion recognized in the second image, and a result value related to a previously derived emotion.

Description

Deep learning based face detection and emotion recognition system and method {SYSTEM AND METHOD FOR FACE DETECTION AND EMOTION RECOGNITION BASED DEEP-LEARNING}

The present invention relates to a face detection and emotion recognition system and method, and more particularly to a deep learning based face detection and emotion recognition system and method.

The conventional emotion recognition system illustrated in FIG. 1 may recognize individual emotion states of each face from an input image including a plurality of faces. The emotion recognition system includes an input unit 10, a face feature extractor 20, a first classifier 30, an emotion feature extractor 50, and a second classifier 60.

The input unit 10 receives an input image including a plurality of faces, and the face feature extractor 20 generates a face feature value from the input image. The first classifier 30 outputs the first classification result 40, that is, the detected face image from the input image using the training data. Meanwhile, when a plurality of faces are present in the input image and a plurality of faces are detected, a crop image is output for each detected face.

Conventionally, the feature used for initial face detection was the intensity of the face in the image. However, as performance depends on race, lighting, etc., irrelevant features are needed, and features such as Haar-like feature, Local Binary Pattern (LBP), and Modified Census Transform are proposed.

Meanwhile, a neural network based classifier, an AdaBoost classifier, or the like may be used as a classifier for face detection.

The emotion feature extractor 50 generates an emotion feature value from the detected face crop image. The second classifier 60 outputs the second classification result 70, that is, the emotional state of each detected face, from the face crop image using the training data.

As a conventional technique for emotion recognition, facial expressions are distinguished as shown in Table 1 using eyebrows, eyes, nose, and mouth as feature elements. Emotion recognition methods using such facial feature elements include optical flow analysis and holistic analysis.

TABLE 1

However, the prior art approach has problems and limitations in the following aspects.

First, subjective feature point selection by classifier designer. Since the classifier is heavily influenced by the recognition object and the surrounding environment, feature point selection and extraction are generally very difficult problems and greatly affect the recognition performance according to the feature point selection result. In addition, there is no measure to determine that the feature point selected for the classifier is optimal.

Second, two sorters (face detection classifier, emotion recognition classifier) are required because the two-step sequential process of emotion recognition is performed after face detection. Accordingly, two types of learning data, feature point selection and extraction process, and learning process are required.

Third, when a plurality of faces are detected, an emotion recognition classifier operation equal to the number of faces is required to recognize emotions for each face. Therefore, as the number of detected faces increases, computation time may increase significantly.

Fourth, the existing emotion recognition technology is performed on a still image, but the actual human emotion needs to be recognized on a continuous line in consideration of the passage of time.

Published Patent Publication 10-2016-0096460

Therefore, the problem to be solved by the present invention is to overcome the problems and limitations of the prior art, using the deep learning architecture and probability model including one classifier can perform face detection and emotion recognition at the same time, The present invention provides a face detection and emotion recognition system and method having higher accuracy in consideration of emotion continuity.

According to an embodiment of the present invention, a deep learning operation unit detects a face by receiving a plurality of images and recognizes an emotion expressed on the detected face, and a first emotion recognized in the first image by the deep learning operation unit. And a probability calculator configured to derive a current emotion by calculating a second emotion recognized in the second image, which is the next input image of the first image, and a result value related to the previously derived emotion, based on the emotion transfer probability. .

The emotion transfer probability may be higher as the distance is closer and lower as the distance is greater in the 3-axis emotion model.

The deep learning calculator has at least one bounding box information for face detection and emotion estimate information for a face in the at least one bounding box for each grid cell in order to perform the face detection and emotion recognition with one classifier. There may be.

The probability calculating unit is the following equation

According to the present emotion can be derived.

The apparatus may further include a second deep learning calculator configured to determine an identity of the detected face.

According to another aspect of the present invention, there is provided a face detection and emotion recognition method comprising: receiving a plurality of images, detecting a face from the image, recognizing the emotion expressed on the detected face, and the first step in the recognition step. The current emotion is calculated based on the probability of emotion transition based on a first emotion recognized in the image, a second emotion recognized in the second image that is the next input image of the first image, and a result value related to a previously derived emotion. Deriving a step.

According to the present invention, by using the deep learning architecture and the probabilistic model, face detection and emotion recognition can be simultaneously performed using only one classifier, and the face for a plurality of faces simultaneously without increasing the computation time in proportion to the number of detected faces. Since detection and emotion recognition can be performed, real-time calculation is possible, and since continuity of emotion is considered, face detection and emotion recognition can be performed with higher accuracy.

1 is a block diagram illustrating a conventional face detection and emotion recognition system.
2 is a diagram illustrating a face detection and emotion recognition system according to an embodiment of the present invention.
3 is a block diagram illustrating the deep learning operation unit illustrated in FIG. 2.
FIG. 4 is a diagram illustrating a tensor reconstructed by the output node reconstructor shown in FIG. 3.
5 is a diagram illustrating a result of face detection and emotion recognition for each grid cell.
6 is a diagram illustrating emotion recognition classification probabilities of each bounding box.
7 is a diagram illustrating a three-axis emotion model.
8 is a diagram illustrating a state transition model.
9 is a flowchart illustrating a face detection and emotion recognition method according to another embodiment of the present invention.
10 is a block diagram illustrating a face detection and emotion recognition system according to another embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention.

2 is a diagram illustrating a face detection and emotion recognition system according to an embodiment of the present invention.

2, a face detection and emotion recognition system according to an embodiment of the present invention includes an input unit 100, a deep learning operator 200, a probability operator 300, and an output unit 400.

The input unit 100 receives an input image including a plurality of faces, and the deep learning operation unit 200 detects a plurality of human faces inside the input image in real time based on an object recognition technology using deep learning. Emotion recognition is simultaneously performed, and the probability calculating unit 300 performs a continuous emotion recognition in time by combining a probability model with respect to the detected face and the recognized emotion as a result of the deep learning operation unit 200. The output unit 400 displays or exports the emotion calculated by the probability calculator 300 with respect to the detected face in various ways.

The deep learning operation unit 200 may be implemented using a Faster RCNN algorithm. Since the Faster RCNN algorithm is based on the Convolutional Neural Network (CNN), since the feature point selection and extraction process is performed through a convolutional operation that can be learned, the deep learning operation unit 200 is a designer by a designer who was a problem of the prior art. No subjective feature point selection is required. In addition, since convolution operation parameters are learned through a backpropagation algorithm so that recognition errors are minimized, feature selection and extraction processes can be optimized.

In addition, the deep learning operation unit 200 uses a Faster RCNN algorithm to determine a localization problem of finding an object in an input image and a classification problem of finding an object. Problem) can be redefined. Therefore, unlike the prior art in which two kinds of classifiers are required, face detection and emotion recognition may be simultaneously performed using one classifier.

The deep learning operation unit 200 divides the input image into N * N grid cells and generates bounding boxes of a predetermined size and number for each cell. In addition, after selecting a plurality of bounding boxes in which there are objects to be detected, a plurality of objects are recognized since the objects in each bounding box are recognized.

Therefore, even when a plurality of faces are detected, each individual emotional state can be recognized with respect to the bounding box for the plurality of faces detected without the need for an additional face recognition classifier. That is, the deep learning algorithm for object location and object recognition can be used for face location and facial expression recognition.

Next, the deep learning operation unit 200 will be described in more detail with reference to FIGS. 3 to 6. 3 is a block diagram illustrating the deep learning operation unit illustrated in FIG. 2, and FIG. 4 is a diagram illustrating a tensor reconfigured by the output node reconstruction unit illustrated in FIG. 3. FIG. 5 is a diagram illustrating face detection and emotion recognition results for each grid cell, and FIG. 6 is a diagram illustrating emotion recognition classification probabilities of each bounding box.

The deep learning calculator 200 includes an image controller 210, a feature extractor 220, a classifier 230, an output node reconstructor 240, and a result calculator 250.

The deep learning operation unit 200 detects a face from the input image and recognizes seven basic emotions (happy, angry, sad, disgusted, neutral, fearful, and surprised) expressed on the face.

The image controller 210 adjusts the size of the input image. In this embodiment, the size of the input image is adjusted to 448 * 448 (= 200,704). The image controller 210 may perform image preprocessing, such as image correction or noise removal, as necessary.

The feature extracting unit 220 receives the adjusted image and automatically extracts the feature while passing through 24 convolutional layers.

The classifier 230 passes through three fully connected layers and generates a total of 833 output node values at the output. Here, the number of nodes related to the face detection bounding box is 490 and the number of nodes related to the emotion recognition result is 343.

The output node reconstructor 240 reconstructs 833 output nodes into 7 * 7 * 17 tensors as shown in FIG. Tensor means here a multidimensional matrix chunk. A 7 * 7 * 17 tensor is divided into 7 * 7 grid cells, each of which consists of 17 values:

① Randomly generate two candidate bounding boxes containing faces for each grid cell, the first five of the 17 values represent the information of the first bounding box, and the next five represent the information of the second bounding box. .

Each bounding box information consists of five predicted values of position (x, y), size (w, h), and probability c of a face in the bounding box.

③ The last seven values represent the estimate (probability P) for each of the seven emotions for the face in the bounding box.

Therefore, as a result of face detection and emotion recognition for each grid cell, the 17 * 1 * 1 tensor for each grid cell may be represented as shown in FIG. 5.

As shown in FIG. 6, the result calculator 250 calculates an emotion recognition classification probability of each bounding box by multiplying the probability c of each bounding box by the estimated value P for each of the seven emotions.

Since there are two bounding boxes per 49 grid cells (= 7 * 7), it is possible to obtain the emotion recognition classification probabilities of a total of 98 bounding boxes. At this time, the value of the emotion recognition classification probability is smaller than the reference value (for example, 0.2) is changed to zero.

The result calculator 250 sorts the bounding box in descending order for each emotion based on the emotion recognition classification probability. To remove the overlapping bounding box for each emotion, calculate the IoU (Intersection of Union) based on the largest value of the emotion recognition classification probability, and if the classification value is smaller than the reference value (for example, 0.5), Change it to zero.

Through this process, the emotion class having the highest value of the emotion recognition classification probability is the emotion recognition result, and the bounding box to which the value belongs indicates the face detection result.

If the largest value is 0, it means that there is no face in the grid cell. If two emotion classes are detected in one grid cell, it means that the faces overlap.

On the other hand, in the conventional emotion recognition technology, most of the emotion recognition for the still image is performed, it is common to recognize the human emotion on the continuous line in consideration of the passage of time. And the probability of emotional transition from one state of emotion to another on the time axis varies from emotion to emotion.

보다는 화남 상태에서 슬픔으로 변화될 확률

이 일반적으로 더 크다.For example, the probability of being sad from sad

Rather than being angry and sad

This is usually larger.

FIG. 7 illustrates a three-axis emotion model, which illustrates a direct relationship between specific combinations of the neurotransmitters dopamine, noradrenaline and serotonin and eight basic emotions. In other words, as the neurotransmitter on each axis increases, the emotion associated with that axis increases. For example, when dopamine is maximal and noradrenaline and serotonin are zero, the fear emotion is maximal. Conversely, when dopamine is zero and noradrenaline and serotonin are maximal, the surprise emotion is maximal.

Since neurotransmitters are less likely to change rapidly in the human body, the closer the distance is, the higher the probability of emotional transfer and the lower the distance, the lower the probability of emotional transfer.

The probability operation unit 300 of the face detection and emotion recognition system according to an embodiment of the present invention uses the state transition model of FIG. 8 having seven emotions as a state and the following [Equation 1]: Use it to derive the current emotion.

[Equation 1]

That is, the probability calculator 300 targets the emotions for each frame recognized by the deep learning calculator 200 and the result values related to previous emotions that the probability calculator 300 previously calculated and derived, and based on the emotion transfer probability. The current emotion is derived by calculating with.

Therefore, according to the deep learning operator 200 and the probability operator 300 according to the exemplary embodiment of the present disclosure, emotions may be continuously recognized in consideration of the passage of time. Therefore, when face detection and emotion recognition are performed on a continuous image such as a video, even when the deep learning operation 200 incorrectly recognizes a partial frame, the probability operation unit 300 may correct the correct emotion. Can be.

For the convenience of explanation, it has been described that emotion recognition is performed on seven emotions, but emotion recognition may be performed on eight or more emotions or six or less emotions.

As described above, according to the face detection and emotion recognition system according to the embodiment of the present invention based on the deep learning technology, the computation time does not increase proportionally to the plurality of face detection and emotion recognition while maintaining a high recognition rate of 97% or more, thereby real-time calculation. It is possible and commercialization possibility is high.

Next, a face detection and emotion recognition method according to an embodiment of the present invention will be described with reference to FIG. 9. 9 is a flowchart illustrating a face detection and emotion recognition method according to another embodiment of the present invention.

The method illustrated in FIG. 9 is a method performed by the face detection and emotion recognition system illustrated in FIGS. 2 to 8, and the deep learning operation unit 200 performs deep learning-based face detection and emotion recognition (S810). In addition, the probability calculating unit 300 derives the current emotion using the emotional transition probability in consideration of the continuous emotional state (S820).

Since steps S810 and S820 have been described in detail in the foregoing embodiment, the present embodiment is adopted as it is, and description thereof is omitted to avoid duplication.

Next, with reference to FIG. 10, a face detection and emotion recognition system for identifying an identity of a detected face as well as face detection and emotion recognition will be described. 10 is a block diagram illustrating a face detection and emotion recognition system according to another embodiment of the present invention.

The face detection and emotion recognition system according to another exemplary embodiment of the present invention includes an input unit 910, a first deep learning calculator 920, a probability calculator 930, a second deep learning calculator 940, and an output unit 950. It includes.

The input unit 910, the first deep learning operation unit 920, the probability operation unit 930, and the output unit 950 according to the present embodiment are the input unit 100, the deep learning operation unit 200, and the probability operation unit ( 300 and the output unit 400 are substantially the same, so a detailed description thereof will be omitted and only differences will be described.

The first deep learning calculator 920 provides the cropped or combined image of the detected face to the second deep learning calculator 940.

Like the first deep learning calculator 920, the second deep learning calculator 940 may determine the identity of the detected face by using a Faster RCNN algorithm or a general CNN algorithm.

Therefore, the face detection and emotion recognition system of the present embodiment can simultaneously recognize not only face recognition in the input image, but also the identity and individual emotions of the plurality of detected faces in real time.

Embodiments of the invention include a computer readable medium containing program instructions for performing various computer-implemented operations. This medium records a program for executing the face detection and emotion recognition methods described so far. The media may include, alone or in combination with the program instructions, data files, data structures, and the like. Examples of such media include, but are not limited to, magnetic media such as hard disks, floppy disks, and magnetic tape, optical recording media such as CDs and DVDs, floppy disks and program commands such as magnetic, optical media, ROM, RAM, flash memory, and the like. Hardware devices configured to store and perform such operations. Alternatively, the medium may be a transmission medium such as an optical or metal wire, a waveguide, or the like including a carrier wave for transmitting a signal specifying a program command, a data structure, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

Although the preferred embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concepts of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

100, 910: input unit, 200: deep learning operation unit,
300, 930: probability operation unit, 400, 950: output unit,
920: first deep learning operator, 940: second deep learning operator,
210: image control unit, 220: feature extraction unit,
230: classifier, 240: output node reconstruction unit,
250: result calculation unit

Claims (10)

delete delete delete A deep learning operation unit which detects a face by receiving a plurality of images and recognizes the emotion expressed on the detected face, and
A result value related to a first emotion recognized in the first image by the deep learning operation unit, a second emotion recognized in a second image which is the next input image that is temporally continuous with respect to the first image, and a result value related to a previously derived emotion Probability calculation unit that calculates the current emotion based on the probability of emotional transition
Including,
The probability calculating unit is the following equation

According to the face detection and emotion recognition system for deriving the current emotion. delete delete delete delete Receiving a plurality of images,
Detecting a face from the image and recognizing the emotion expressed on the detected face; and
In the recognition step, the first emotion recognized in the first image, the second emotion recognized in the second image which is the next input image which is temporally continuous with respect to the first image, and a result value related to the previously derived emotion Deriving the current emotion by calculating based on the probability of the emotional transition
Including,
The current emotion derivation is

Face detection and emotion recognition method calculated according to. delete

Images