KR102174175B1 - Facial emotional recognition apparatus for Identify Emotion and method thereof - Google Patents

Abstract

The present invention relates to a facial emotion recognition apparatus and method for identifying emotion from an input facial image.
According to the present invention, in a facial emotion recognition apparatus for identifying emotion from an input face image, a face image to be subjected to emotion recognition is input, a feature of the entire face image is extracted from the input face image, and the extracted feature A facial feature extraction unit that transmits information on the upper two emotions among the six emotions obtained from, and calculates a difference in coordinates by comparing the input face image with the feature points respectively assigned to the basic face image for the same person, A geometric change amount extracting unit that calculates vector values for two upper emotions by applying the difference value of the coordinates by the CNN algorithm, and the emotion recognition value extracted from the facial feature extracting unit and the vector value calculated by the geometric change amount extracting unit. And an emotion classifying unit for classifying the emotion having the highest value among the result values calculated by assigning a weight to each as a final emotion.
The facial expression recognition apparatus according to the present invention improves emotion recognition errors that frequently occur in the upper two emotions obtained through a conventional algorithm, and a vector value calculated through a recognition value extracted through facial features and a geometric change amount. Each weight is assigned to each and the result value is calculated, thereby having the effect of determining the final emotion with improved accuracy.

Description

TECHNICAL FIELD [0001] Facial emotion recognition apparatus for identifying emotions and method thereof

The present invention relates to a facial emotion recognition device capable of identifying emotions and a method thereof, and more particularly, to a facial emotion recognition device capable of accurately identifying emotions from an input facial image using a learning algorithm and a method thereof. will be.

Recently, human and machine technologies for communication between humans and humans have been developed based on the five senses that play a major role in human interaction. In particular, with the development of artificial intelligence technology in recent years, Apple's siri, Amazon's alex, and SKT's NUGU, an artificial intelligence speaker, and Naver's Clova, manage people's daily schedules with technologies that recognize and respond to people's voices and languages. It is applied to artificial intelligence robots, such as playing music that is played, and has a close interaction in real life. However, in order to understand the complex and complex interactions of people, it is necessary to accept various senses in technology.

Among them, the necessity of technology for vision, which has a cognitive ratio of 50% or more of human senses, is important. The human face provides important information as a clue to understand the user's current state. Facial expression classification and recognition have been widely studied for the past 10 years, and with the recent increase in data and the development of deep learning technology, research on a face recognition system that more accurately recognizes emotions in real time has become active.

However, the conventional facial recognition system requires a large amount of data to be used for deep learning learning, and a problem of a lack of facial expression recognition data set with emotion labels is caused.

In addition, the facial expression recognition data set is mainly composed of the data set in a stationary state than the data set in which there is a change in sequence, so there is a problem that the accuracy is poor due to mainly using the spatial appearance.

The technology behind the present invention is disclosed in Korean Patent Application Publication No. 10-2014-00876007 (published on July 9, 2014).

An object of the present invention is to provide a facial emotion recognition apparatus and method capable of accurately identifying emotions from an input face image using a learning algorithm.

According to an embodiment of the present invention for achieving such a technical problem, in a facial emotion recognition apparatus for identifying emotion from an input face image, a face image to be subjected to emotion recognition is input, and the entire face image is received from the input face image. Facial feature extraction that extracts features for, obtains six emotions listed according to ranking from the extracted features, and delivers information on the top two emotions among the obtained six emotions to the geometrical variation extraction unit and the emotion classification unit, respectively. Second, the difference in coordinates is calculated by comparing the feature points assigned to the input face image and the basic face image without facial expressions for the same person, and the difference in coordinates is applied to the six emotions combined by the CNN algorithm. Thus, a geometrical change amount extracting unit that calculates vector values for the upper two emotions, and an emotion recognition value calculated from the upper two emotions transmitted from the facial feature extracting unit and a vector value calculated by the geometrical change amount extracting unit are weighted respectively. And an emotion classifying unit that calculates each result value for the upper two emotions by assigning to and classifies the emotion having the highest value among the calculated result values.

The emotions can be classified into six categories including happiness, sadness, fear, disgust, surprise and anger.

The facial feature extraction unit may include an image input module receiving an input of a face image subject to emotion recognition, a face region recognition module separating a face region from the input face image, and applying the separated face image to an autoencoder A face generation module that generates a basic face image from the input face image and transmits it to the geometric change amount extraction unit, and facial feature extraction that extracts main feature information of the face from the input face image using a local binary pattern (LBP) algorithm. The module and the extracted main feature information of the face are applied to the CNN algorithm to calculate six emotion results, and the emotion corresponding to the top two among the calculated six emotion results is determined by a geometric change amount extraction unit and an emotion classification unit. It may include a determination module that is transmitted to each.

The face generation module generates a dataset by learning an error arising from a difference in feature values between two face images using a facial image with an expression and a basic face image without an expression for the same person, and generates the data set. The basic face image may be generated from the face image that is the emotion recognition target by using an autoencoder to which the set is applied.

The facial feature extraction module may apply blurring to a face image in which only the face area is separated to remove noise, and then extract features mainly in main action units using an LBP algorithm.

The LBP algorithm may be calculated by the following equation.

,

,

는 입력된 얼굴 이미지에 선택된 현재의 픽셀이고,

는

의 이웃에 분포된 픽셀이고,

은 현재의 픽셀과 이웃의 픽셀을 비교하여 0 또는 1로 산출된 값이고, P는 8로, 비교하고자 하는 이웃 픽셀의 개수이다. 따라서 p는 0부터 7까지 현재의 픽셀의 왼쪽 상단에 위치한 이웃 픽셀부터 시계방향으로 부여된다.here,

Is the current pixel selected in the input face image,

Is

Are pixels distributed in the neighborhood of

Is a value calculated as 0 or 1 by comparing the current pixel with a neighboring pixel, and P is 8, which is the number of neighboring pixels to be compared. Therefore, p is given from 0 to 7 in a clockwise direction from a neighboring pixel located at the top left of the current pixel.

The determination module calculates six emotions by applying a plurality of convolutions to the input face image to which the LBP algorithm is applied, and classifies emotions corresponding to the top two among the calculated six emotions. can do.

The geometrical change amount extracting unit generates feature points based on action units on an image input module receiving an input of the face image to be subjected to emotion recognition, the input face image and a basic face image transmitted from the face generation module. A feature point generation module, a vector value calculation module that compares and determines the input face image and the basic face image using the generated feature point, and calculates a vector value using a coordinate difference for the feature point where the difference occurs, and An emotion result calculation module that corresponds to the upper two emotions received from the determination module, and selects emotions corresponding to the upper two among the six emotions combined by applying the vector value calculated by the vector value calculation module to the CNN algorithm It may include.

The vector value calculation module may calculate a vector value by the following equation.

,

)는 표정이 있을 때의 특징점에 대한 좌표값이고, (

,

)는 표정이 없을 때의 특징점에 대한 좌표값이다. here, (

,

) Is the coordinate value for the feature point when there is an expression, and (

,

) Is the coordinate value for the feature point when there is no expression.

The emotion classification unit includes an emotion receiving module for receiving information on the upper two emotions from the facial feature extraction unit, and a recognition value for calculating and normalizing the upper two emotions transmitted from the facial feature extraction unit as an emotion recognition value. The calculation module, and the recognition values for the upper two emotions of the facial feature extracting unit and the vector values for the upper two emotions transmitted from the geometrical variation extracting unit, are weighted to result in the upper two emotions. A value is calculated, and an emotion to which the highest value is assigned among the calculated result values may be determined as the emotion for the face image to be the emotion recognition target.

The emotion classification unit may calculate result values for the upper two emotions by the following equation.

는 얼굴 특징 추출부에서 추출된 감정에 대한 인식값이고,

는 기하학적 변화량 추출부에서 추출된 감정에 대한 벡터값이며,

는 가중치 값을 나타낸다. here,

Is the recognition value for the emotion extracted from the facial feature extraction unit,

Is a vector value for the emotion extracted from the geometric change amount extraction unit,

Represents the weight value.

The weight value may be set to a value between 0.7 and 0.8.

In addition, the present invention includes the steps of extracting features for the entire face image from the face image subject to emotion recognition, and extracting information on the upper two emotions among the six emotions obtained from the extracted facial features,

The input face image and the feature points assigned to the basic face image without expression for the same person are compared to calculate the difference in coordinates, and the calculated difference value of the coordinates is applied to the CNN algorithm for the upper two emotions. Calculating the vector value, and

Each result value for the upper two emotions is calculated by assigning a weight to each of the vector values for the upper two emotions calculated using the difference between the recognition values for the upper two emotions calculated from the facial features and the feature point, and And classifying the emotion having the highest value among the calculated result values as final emotions.

As described above, according to the present invention, it is possible to improve emotion recognition errors that frequently occur in the upper two emotions obtained through a conventional algorithm, and can be used both as features of a dynamic or static state, and extracted through facial features. A weight is applied to the vector value calculated through the geometric change of the recognition value and the feature point, and the result value is calculated, thereby achieving an effect of determining a final emotion with improved accuracy.

In addition, since the present invention can extract a basic face image from an input face image even without a separate data set, it is possible to solve the problem of lack of data set and to extract dynamic information from a static data set rather than a spatial data set. It has an effect.

In addition, the present invention has the effect of enabling a more detailed and comprehensive analysis by measuring the strength of a partial action unit and analyzing a high number of frames because the degree of change is small and has a short time in the emotional state to conceal the emotion. .

1 is a block diagram of an apparatus for recognizing facial expressions according to an embodiment of the present invention.
FIG. 2 is a configuration diagram of a facial feature extraction unit illustrated in FIG. 1.
3 is a configuration diagram of a geometric change amount extraction unit shown in FIG. 1.
FIG. 4 is a diagram schematically illustrating a feature point generated by the feature point generation module of FIG. 3.
5 is a diagram showing the configuration of the emotion classification unit shown in FIG.
6 is a flowchart schematically illustrating an emotion identification method using a facial emotion recognition apparatus according to an embodiment of the present invention.
FIG. 7 is a flow chart for explaining step S610 shown in FIG. 6.
FIG. 8 is a diagram illustrating an example of generating a basic face image from a face image through an autoencoder in step S613 shown in FIG. 7.
9 is a diagram illustrating a method of extracting a feature through the LBP algorithm in step S614 shown in FIG. 7.
10 is a diagram illustrating a method of extracting the upper two emotions from the determination module in step S615 shown in FIG. 7.
11 is a flow chart for explaining step S620 shown in FIG.
12 is a diagram schematically illustrating a difference between a face image and a basic image in step S623 illustrated in FIG. 11.
FIG. 13 is a flowchart for explaining step S630 shown in FIG. 6.
14 is a graph of a weight value assigned in step S634 shown in FIG. 8.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this process, the thickness of the lines or the size of components shown in the drawings may be exaggerated for clarity and convenience of description.

In addition, terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to the intention or custom of users or operators. Therefore, definitions of these terms should be made based on the contents throughout the present specification.

Hereinafter, an apparatus for recognizing facial expressions according to an embodiment of the present invention will be described with reference to FIG. 1.

1 is a block diagram of an apparatus for recognizing facial expressions according to an embodiment of the present invention.

As shown in FIG. 1, the facial expression recognition apparatus 100 according to an embodiment of the present invention includes a facial feature extraction unit 110, a geometric change amount extraction unit 120, and an emotion classification unit 130.

First, the facial feature extraction unit 110 receives a face image that is an emotion recognition target from a user, and extracts features of the entire face image from the input face image.

Then, the facial feature extraction unit 110 acquires six emotions listed according to ranking from the features extracted from the entire face image, and obtains information on the upper two emotions among the obtained six emotions, and the geometric change amount extraction unit 120 ) And the emotion classification unit 130, respectively.

The geometrical change amount extracting unit 120 calculates a difference in coordinates by comparing the input face image with the feature points respectively assigned to the basic face image without an expression for the same person. Then, the geometric change amount extraction unit 120 calculates vector values for the upper two emotions among the six emotions combined by applying the calculated difference value of the coordinates to the CNN algorithm.

The emotion classification unit 130 gives a weight to each of the emotion information for the upper two extracted by the facial feature extraction unit 110 and the vector values for the upper two emotions extracted by the geometric change amount extraction unit 120, Calculate the value. The emotion classifying unit 130 determines the emotion having the highest value among the two upper emotions as the final emotion by using the calculated result value.

Hereinafter, a facial feature extraction unit 110 according to an embodiment of the present invention will be described with reference to FIG. 2.

FIG. 2 is a configuration diagram of a facial feature extraction unit illustrated in FIG. 1.

The facial feature extraction unit 110 tracks a face area from an input facial image to be recognized for emotion, and extracts features from the tracked face area, and includes an image input module 111, a face area recognition module 112, A face generation module 113, a facial feature extraction module 114, and a determination module 115 are included.

First, the image input module 111 receives a face image input by a user. The received face image is an emotion recognition object, and the image input module 111 delivers the received face image to the region recognition module 112.

The face region recognition module 112 separates only the face region from the received face image.

At this time, the reason why the face region recognition module 112 separates only the face region is the accuracy of emotion recognition according to environmental variables such as the position of the person included in the image, the accessories worn by the person, sunlight and the clothes worn by the person. This is to prevent falling.

The face generation module 113 generates a basic face image from the face image in which only the received face area is separated. In addition, the face generation module 113 generates a data set by collecting both a face image and a basic face image for the same person.

The facial feature extraction module 114 extracts main features of the entire face by applying the face image separated from the face region recognition module 112 to a local binary pattern (LBP) algorithm. In other words, the facial feature extraction module 114 calculates the LBP for each pixel of the input face image in order to extract the main features of the entire face, and generates a face image expressed in a texture form according to the calculated LBP image result. It is transmitted to the determination module 115.

The determination module 115 calculates six emotion results by applying the received textured face image to the CNN algorithm.

The calculated information on the six emotions is arranged in order, and the determination module 115 transmits two emotion information having a high ranking among the information on the six emotions listed to the emotion classification unit 130.

Hereinafter, the geometrical change amount extraction unit 120 according to an embodiment of the present invention will be described in more detail with reference to FIGS. 3 and 4.

FIG. 3 is a configuration diagram of a geometric change amount extracting unit shown in FIG. 1, and FIG. 4 is a diagram schematically illustrating a feature point generated by the feature point generation module of FIG. 3.

The geometrical change amount extracting unit 120 extracts the amount of change using feature points respectively assigned to the face image and the basic face image, and extracts vector values for the upper two emotions by using the extracted amount of change. 121), a feature point generation module 122, a vector value calculation module 123, and an analysis result calculation module 124.

First, the image input module 121 receives a face image that is an emotion recognition target from a user.

The feature point generation module 122 receives a face image from the image input module 121 and generates a feature point based on the action unit on the received face image.

Action units refer to major muscles that move when making facial expressions. As shown in FIG. 4, the feature point generation module 122 generates approximately 18 feature points such as eyebrows, eyes, nose, and mouth. Position.

By generating the feature points as described above, it is possible to reduce the effect of lighting when identifying emotions through a face image, and it has an effect that it is easy to identify emotions through a difference between a face image and a basic face image.

Meanwhile, the feature point generation module 122 also generates feature points in the basic face image in order to track changes in the feature points generated in the face image. If there is no information on the basic face image, the face generation module 112 of the facial feature extraction unit 110 requests and receives the generated basic face image.

The vector value calculation module 123 receives the face image and the basic face image from which the feature point is generated from the feature point generation module 122, and compares the face image and the basic face image to determine a vector value for the feature point where the difference occurs. Calculate.

Hereinafter, the emotion classification unit 130 according to an embodiment of the present invention will be described in more detail with reference to FIG. 5.

5 is a diagram showing the configuration of the emotion classification unit shown in FIG.

As shown in FIG. 5, the emotion classification unit 130 includes information on the upper two emotions extracted by the facial feature extraction unit 110 and vector values for the upper two emotions extracted by the geometrical change amount extraction unit 120. Each weight is assigned to calculate a result value that can be determined as a final emotion, and includes an emotion receiving module 131, a recognition value calculation module 132, and a classification module 133.

The emotion receiving module 131 receives information on the upper two emotions from the facial feature extraction unit 110. At this time, the upper two emotion information received from the feature extraction unit 110 is not calculated as a value, but includes only information. Accordingly, the emotion receiving module 131 transmits the received emotion information to the recognition value calculation module 132.

The recognition value calculation module 132 calculates information on the received upper two emotions as a recognition value. Accordingly, the recognition value calculation module 132 calculates and normalizes information on emotion as a recognition value.

The classification module 133 receives the recognition values for the top two emotions received from the facial feature extraction unit 110 and the vector values for the top two emotions received from the geometric change amount extraction unit 120. . In addition, the classification module 133 determines an emotion for a face image that is an emotion recognition target by assigning a weight to the received recognition value and the vector value.

Hereinafter, a method of identifying an emotion using the facial emotion recognition apparatus 100 according to an embodiment of the present invention will be described with reference to FIG. 6.

6 is a flowchart schematically illustrating an emotion identification method using a facial emotion recognition apparatus according to an embodiment of the present invention.

As shown in FIG. 6, in the emotion identification method using the facial emotion recognition apparatus 100 according to an embodiment of the present invention, the step of extracting information on the upper two emotions from the facial feature extracting unit (S610), extracting the geometric change amount A step of extracting vector values for the upper two emotions from the part (S620) and a final emotion classification step (S630).

First, the facial feature extraction unit 110 extracts features for the entire face using the input face image, and extracts information on the upper two emotions from the extracted features (S610).

Then, the geometrical change amount extraction unit 120 generates feature points in the face image input by the user and the basic face image received from the facial feature extraction unit 110. The geometric change amount extracting unit 120 calculates vector values for the upper two emotions by using the difference values between the generated feature points (S620).

As described above, when the facial feature extraction unit 110 extracts information on the upper two emotions and the geometric change amount extraction unit 120 calculates vector values for the upper two emotions, the emotion classification unit 130 ) Calculates a result value by assigning a weight to each of the recognized value and the vector value, and determines a final emotion by using the calculated result value (S630).

In the following, each step described above will be described in more detail.

First, a step (S610) of extracting information on the upper two emotions from the facial feature extraction unit will be described in more detail using FIGS. 7 to 10.

7 is a flow chart for explaining step S610 shown in FIG. 6, and FIG. 8 is a diagram showing an example of generating a basic face image from a face image through an autoencoder in step S613 shown in FIG. FIG. 7 is a diagram illustrating a method of extracting features through the LBP algorithm in step S614 shown in FIG. 7, and FIG. 10 is a diagram illustrating a method of extracting the upper two emotions from the determination module in step S615 shown in FIG.

As shown in FIG. 7, the step of extracting information on the upper two emotions from the facial feature extraction unit (S610) includes a face image input step (S611), a face region separation step (S612), and a basic face generation step (S613). . It includes a facial feature extraction step (S614) and the upper two emotion information delivery step (S615) through the LBP algorithm.

First, the image input module 111 of the facial feature extraction unit 110 receives a face image to be recognized from a user (S611).

The image input module 111 transfers the received face image to the image input module 112, and the image input module 112 separates only the face area from the received face image (S612).

The separated face image is transmitted to the face generation module 113 and the face feature extraction module 114, respectively.

First, the face generation module 113 generates a basic face image from a face image that is an emotion recognition target based on the generated data set (S613).

To explain this in detail, first, as shown in FIG. 8, the face generation module 113 inputs a face image and a basic face image obtained from the same person into an autoencoder using a VGG19 network structure. And output respectively.

In this case, the autoencoder finds a difference value by changing the multidimensional input data into a low-dimensional code and then converting the low-dimensional code into the first input multidimensional data.

As described above, the face generation module 113 generates a data set by repeatedly calculating a difference value generated between the face image and the basic face image through an autoencoder.

Therefore, when the face image input from the user is one image in which emotions are unknown but expressions are generated, the face generation module 113 applies the difference value calculated by the deep-learned dataset to the input face image. Create a face image.

Meanwhile, the facial feature extraction module 114 receiving the separated face image extracts the entire face feature from the separated face image (S614).

At this time, for the entire face feature extracted by the facial feature extraction module 114, a feature value is calculated from a plurality of pixels included in the face image, and the face image that has been shown in a chromatic color according to the calculated feature value is expressed in an achromatic color, but the texture is It appears as if it exists.

Accordingly, the facial feature extraction module 114 calculates a feature value using a local binary pattern (LBP) algorithm.

The LBP (local binary pattern) algorithm efficiently expresses the texture of an image, and calculates a value obtained by coding the relative brightness change of the surrounding area of each pixel for all pixels of the image in binary numbers. That is, if it is brighter than the center pixel, it is coded as 1, and if it is dark, it is coded as 0, and a binary number connecting the coded values is used as the feature value of the pixel, and the LBP of the current pixel is calculated using Equation 1 below.

는 입력된 얼굴 이미지에 선택된 현재의 픽셀이고,

는

의 이웃에 분포된 픽셀이고,

은 현재의 픽셀과 이웃의 픽셀을 비교하여 0 또는 1로 산출된 값이다. 또한, P는 8로, 비교하고자 하는 이웃 픽셀의 개수이다. 따라서 p는 0부터 7까지 현재의 픽셀의 왼쪽 상단에 위치한 이웃 픽셀부터 시계방향으로 부여된다.here,

Is the current pixel selected in the input face image,

Is

Are pixels distributed in the neighborhood of

Is a value calculated as 0 or 1 by comparing the current pixel with a neighboring pixel. Also, P is 8, which is the number of neighboring pixels to be compared. Therefore, p is given from 0 to 7 in a clockwise direction from a neighboring pixel located at the top left of the current pixel.

)이 65이고, 주변 픽셀값(

)이 70,66,58,60,60,64,68,69값을 가진다고 가정한 상태에서,

)식을 대입하면, 5, 1, -7, -5, -5, -1, 3, 4가 산출된다. For example, as shown in FIG. 9, eight pixels adjacent to the current pixel are used. Current pixel value (

) Is 65, and the surrounding pixel value (

) Is assumed to have values of 70,66,58,60,60,64,68,69,

Substituting the equation) yields 5, 1, -7, -5, -5, -1, 3, 4.

값을 이진수로 변환하며, 이진수 변환 방법은 이웃의 픽셀이 현재의 픽셀보다 더 크거나 같은 경우 1로 변환하고, 현재의 픽셀이 더 작은 경우에는 0으로 변환한다. 따라서, 5, 1, -7, -5, -5, -1, 3, 4는 1,1,0,0,0,0,1,1로 변환한다. At this time, the calculated

The value is converted to a binary number. In the binary conversion method, when the neighboring pixel is greater than or equal to the current pixel, it is converted to 1, and when the current pixel is smaller, it is converted to 0. Thus, 5, 1, -7, -5, -5, -1, 3, 4 convert to 1,1,0,0,0,0,1,1.

The converted value is multiplied by a power of 2 and added again to calculate the LBP. That is, (1*1)+(1*2)+(0*4)+(0*8)+(0*16)+(0*32)+(1*64)+(1*128) 195 is calculated.

According to the above calculation method, the facial feature module 114 calculates the LBP for each pixel of the input face image, displays the received face image in a texture form, and transmits it to the determination module 115.

On the other hand, the facial feature extraction module 114 removes noise by applying blurring, a pre-processing process, before converting the face image separated by only the face region to the LBP algorithm, and then using the LBP algorithm to remove the noise. units) to extract features.

As described above, the determination module 115, which received information on the facial feature from the facial feature extraction module 114, applies the feature information to the CNN algorithm and calculates six emotion results.

Meanwhile, the CNN algorithm is an artificial neural network algorithm called a convolutional neural network, in which input data exists in a two-dimensional form, and includes a convolutional layer and a pooling layer. At this time, the convolutional layer and the pooling layer designate a region in the face image and perform a process of compressing a value.

Therefore, the determination module 115 inputs the received LBP image into the CNN algorithm, and proceeds with three convolutional layers and two pooling layers as shown in Table 1 below, so that the emotion information classified into seven in the last softmax layer is stored. Calculate.

At this time, the seven emotions include six emotions, including happiness, sadness, fear, disgust, surprise, and anger, and one innocence. However, in the present invention, it is classified into six emotions excluding no emotions.

As shown in FIG. 10, the determination module 115 calculating six kinds of emotion information as described above is arranged from 0 to 5 according to the rank, and is the highest level corresponding to 0 and 1 among the listed emotion information from 0 to 5. The two kinds of emotion information are transmitted to the geometrical change amount extraction unit 120 and the emotion classification unit 130, respectively (S615).

Hereinafter, the step (S620) of extracting vector values for the upper two emotions from the geometric change amount extracting unit will be described in more detail using FIGS. 11 and 12.

FIG. 11 is a flowchart illustrating step S620 shown in FIG. 6, and FIG. 12 is a diagram schematically illustrating a difference between a face image and a basic image in step S623 shown in FIG.

As shown in FIG. 11, the step of extracting the vector values for the upper two emotions from the geometric change amount extracting unit 120 (S620) is a face image input step (S621), a feature point generation step in the input face image and the basic face image. (S622), a vector value calculation step (S623) by using the feature point difference, and the upper two emotion vector values calculation step (S624).

First, a face image for identifying emotion is input from the user (S621),

In this case, the user may input only one face image, or both a face image and a basic face image for the same person.

As described above, the image input module 121 receiving the face image transmits the face image to the feature point generation module 122. The feature point generation module 122 receiving the face image generates feature points in the face image (S622).

At this time, the feature point generation module 122 generates feature points in both the face image and the basic face image, so that the difference between the feature points generated in the face image and the feature points generated in the basic face image can be compared. If a basic face image is not received from the user, as described above, a request is made to the face generation module 113 of the facial feature extraction unit 120 to receive the basic face image.

As described above, the feature point generation module 122 that generates the feature points in both the face image and the basic face image transmits the face image and the basic face image to the vector value calculation module 123.

The vector value calculation module 123 compares feature points generated in each of the received face image and the basic face image, and calculates a vector value using the generated coordinate difference (S623).

On the other hand, approximately 18 feature points are generated, and since they have (x,y) coordinates, approximately 36 vectors are generated.

Accordingly, as shown in FIG. 12, the vector value calculation module 123 compares the coordinates of the feature points generated in the face image and the coordinates of the feature points generated in the basic face image, and compares the coordinates of the feature points where the difference occurs. Is calculated as a vector value.

At this time, a vector value of length 36 is calculated by applying to Equation 2.

,

)는 표정이 있을 때의 특징점에 대한 좌표값이고, (

,

)는 표정이 없을 때의 특징점에 대한 좌표값이다. here, (

,

) Is the coordinate value for the feature point when there is an expression, and (

,

) Is the coordinate value for the feature point when there is no expression.

The vector value calculation module 123 that calculates the vector value as described above transfers the vector value to the analysis result calculation module 124.

The emotion result calculation module 124 applies the received vector value to the CNN algorithm configured using the VGG16 network, and selects vector values for the emotions corresponding to the upper two among the combined six emotions (S624).

The VGG16 network is a model consisting of 16 layers, and consists of a convolutional and fully-connected layer. Therefore, the CNN algorithm to which the VGG16 network is applied is shown in Table 2 below.

The CNN algorithm creates a total of 30 cases by combining 6 emotions. That is, six emotions are combined into a pair, and the combined emotions are calculated by selecting a model corresponding to the upper 2 emotions in the Softmax layer as shown in Table 2 below. do.

Accordingly, the emotion result calculation module 124 transmits the calculated vector values for the two uppermost emotions to the emotion classification unit 130.

Hereinafter, the final emotion classification step (S630) will be described in more detail using FIG. 13.

FIG. 13 is a flow chart for explaining step S630 shown in FIG. 6, and FIG. 14 is a graph of weight values assigned in step S634 shown in FIG. 8.

As shown in Fig. 13, the final emotion classification step (S630) includes the information receiving step (S631), the recognition value calculation step (S632), the vector value receiving step (S633), and the result using the recognition value and the vector value. It includes a value calculation step (S634) and a final emotion determination step (S634).

First, the emotion receiving module 131 of the emotion classifying unit 130 receives information on the upper two emotions from the facial feature extraction unit 110 and the geometric change amount extraction unit 120 (S931).

At this time, the emotion information received from the facial feature extraction unit 110 expresses two emotions among happiness, sadness, fear, disgust, surprise, and anger, and cannot be recognized as a value. Accordingly, the emotion receiving module 131 transmits information on the upper two emotions received from the facial feature extraction unit 110 to the recognition value calculating module 132.

Accordingly, the recognition value calculation module 132 calculates a recognition value by using the information on the received upper two emotions (S632).

At this time, the recognized value is calculated by Equation 3 below.

는 앞서 산출한 얼굴 특징 추출부(110)의 결과 중 상위 k번째 값에 해당한다. j는 0부터 1의 값을 가지며, 이는 상위 2개의 감정을 의미한다. 이 결과,

는 상위 2가지의 감정에 대해 정규화된 값이다.here,

Is the highest k-th value among the results of the facial feature extraction unit 110 calculated above. j has a value from 0 to 1, which means the top two emotions. As a result of this,

Is the normalized value for the top two emotions.

The recognition value calculation module 132 determines the sum of the upper two emotions received from the facial feature extraction unit as 1. That is, the sum of the recognition values for the emotions of the upper 1 and the emotions of the upper 2 becomes 1. For example, assuming that the upper 1 emotion is a smiling emotion and the upper 2 emotion is an angry emotion, assuming that the recognition value of the smiling emotion is 0.9, the angry emotion corresponding to the upper 2 is 0.1.

As described above, the recognition value calculation module 132 that calculates the emotion information as a recognition value transfers the recognition value to the classification module 133.

Meanwhile, the classification module 133 receives not only the recognized value, but also the vector value from the geometrical and geometrical variation extracting unit 120 (S633).

Then, the classification module 133 calculates result values for emotions corresponding to the top two by assigning weights to the received recognition values and vector values (S634).

At this time, the result value is calculated by the following equation 4.

는 얼굴 특징 추출부에서 추출된 감정에 대한 인식값이고,

는 기하학적 변화량 추출부에서 추출된 감정에 대한 인식값이며,

는 가중치 값을 나타낸다.here,

Is the recognition value for the emotion extracted from the facial feature extraction unit,

Is the recognition value for the emotion extracted from the geometric change amount extraction unit,

Represents the weight value.

At this time, the weight value is set to a value between 0.7 and 0.8.

That is, the recognition values for the top two emotions extracted from the facial feature extraction unit are assigned a weight of 0.7 to 0.8, and the vector values for the top two emotions extracted from the geometric change amount extraction unit are given a weight of 0.2 to 0.3. do.

Meanwhile, the classification module 133 generates a data set with respect to the weight assigned to obtain a result value, and calculates a weight value with high accuracy using the generated data set.

In other words, the classification module 133 generates a data set to obtain a weight value. The data set is generated by substituting the training data and test data in a ratio of 9 to 1, and the training data is the average of the recognized values or vector values by repeatedly calculating the recognition values or vector values for 6 kinds of emotion information. It is the data calculated according to. On the other hand, the test data is data input to match the training data.

The data set for the recognized value is represented by CK+, and the data set for the vector value is represented by JAFFE, and weights are assigned as shown in Table 3 below to calculate the result value for the accuracy trend.

weight (α)
dataset 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 CK+ 50.93 54.63 55.56 57.41 73.15 88.89 89.81 89.81 88.89 JAFFE 45.05 47.25 49.45 49.45 52.75 93.41 94.51 96.70 96.70 Average 47.99 50.94 52.50 53.43 62.95 91.15 92.16 93.26 92.80

When a table is substituted into a graph for calculation in order to change the accuracy of the weight, as shown in FIG. 14, the highest accuracy was measured when the weight value was 0.7 to 0.8.

Accordingly, the classification module 133 calculates a result value by assigning a weight value of 0.7 to 0.8 calculated as described above.

As described above, the classification module 133 calculates each result value for the two emotions, and determines the emotion to which the highest value is assigned among the two result values as the emotion for the face image to be subjected to emotion recognition (S635). ).

Therefore, the facial expression recognition apparatus according to an embodiment of the present invention improves the emotion recognition error that frequently occurs in the upper two emotions obtained through a conventional algorithm, and calculates the recognition value and geometric change amount extracted through the facial features. A weight is applied to each calculated vector value and the result value is calculated, thereby achieving an effect of determining a final emotion with improved accuracy.

In addition, according to the present invention, since the basic face image can be extracted from the input face image even without a separate data set, the problem of lack of data set is solved, and dynamic information is extracted from static data set rather than spatial data set. This has the possible effect.

In addition, according to the present invention, since the degree of change is small and has a short time in the emotional state to conceal the emotion, a more detailed and comprehensive analysis is possible by analyzing the strength of a partial action unit and a high number of frames Has.

The present invention has been described with reference to the embodiments shown in the drawings, but these are only exemplary, and those of ordinary skill in the art will understand that various modifications and equivalent other embodiments are possible therefrom. will be. Therefore, the true technical protection scope of the present invention should be determined by the technical idea of the following claims.

100: facial expression recognition device
110: facial feature extraction unit 111: image input module
112: face region recognition module 113: face generation module
114: facial feature extraction module 115: judgment module
120: geometric change amount extraction unit 121: image input module
122: feature point generation module 123: vector value calculation module
124: evaluation result calculation module
130: emotion classification unit 131: emotion receiving module
132: recognition value calculation module 133: classification module

Claims (24)

In the facial emotion recognition device for identifying emotion from the input face image,
A face image that is an emotion recognition target is input, features of the entire face image are extracted from the input face image, six emotions listed according to ranking are obtained from the extracted features, and the top two emotions are obtained. A facial feature extraction unit that transmits information on emotion to a geometric change amount extraction unit and an emotion classification unit, respectively,
The difference in coordinates is calculated by comparing the input face image with the feature points assigned to the basic face image without expression for the same person, and the difference value of the coordinates is applied to the six emotions combined by the CNN algorithm. A geometrical change amount extraction unit that calculates vector values for two emotions, and
Weights are assigned to the emotion recognition values calculated from the upper two emotions received from the facial feature extraction unit and the vector values calculated by the geometric change amount extraction unit to calculate each result value for the upper two emotions, and calculate It includes an emotion classifying unit for classifying the emotion having the highest value among the result values,
The facial feature extraction unit,
An image input module for receiving a face image to be the target of emotion recognition,
A face region recognition module that separates a face region from the input face image,
A face generation module that applies the separated face image to an autoencoder to generate a basic face image from the input face image and transmits it to a geometric change amount extraction unit,
A facial feature that removes noise by applying blurring to a face image in which only the face region is separated, and then extracts the main feature information of the face mainly in the main action units using a local binary pattern (LBP) algorithm. Extraction module, and
The extracted main feature information of the face is applied to the CNN algorithm to calculate six emotion results, and the emotions corresponding to the top two among the calculated six emotion results are transmitted to the geometric change amount extraction unit and the emotion classification unit, respectively. It includes a judgment module that
The emotion classification unit,
An emotion receiving module that receives information on the upper two emotions from the facial feature extraction unit,
A recognition value calculation module that calculates and normalizes the upper two emotions received from the facial feature extraction unit as emotion recognition values, and
A weight is assigned to the recognition values for the upper two emotions of the facial feature extraction unit and the vector values for the upper two emotions transmitted from the geometric change amount extraction unit to calculate a result value for the upper two emotions. And a classification module configured to determine an emotion to which the highest value is assigned among the calculated result values as an emotion for a face image to be subjected to emotion recognition. The method of claim 1,
The above emotion is,
Facial emotion recognition device classified into 6 categories including happiness, sadness, fear, disgust, surprise and anger. delete The method of claim 1,
The face generation module,
A dataset is created by learning an error arising from the difference in feature values between two face images using a face image with an expression and a basic face image without an expression for the same person.
A facial emotion recognition device that generates the basic face image from a face image that is a target of emotion recognition using an autoencoder to which the generated data set is applied. delete The method of claim 1,
The LBP algorithm,
Facial emotion recognition device calculated by the following equation:

,

here,

Is the current pixel selected in the input face image,

Is

Are pixels distributed in the neighborhood of

Is a value calculated as 0 or 1 by comparing the current pixel with a neighboring pixel, and P is 8, which is the number of neighboring pixels to be compared. Therefore, p is given from 0 to 7 in a clockwise direction from a neighboring pixel located at the top left of the current pixel. The method of claim 1,
The determination module,
Six emotions are calculated by applying a plurality of convolutions to the input face image to which the LBP algorithm is applied,
A facial emotion recognition device for classifying emotions corresponding to the top two among the calculated six emotions. The method of claim 1,
The geometric change amount extraction unit,
An image input module for receiving a face image to be the target of emotion recognition,
A feature point generation module for generating feature points based on action units in the input face image and the basic face image transmitted from the face generating module,
A vector value calculation module that compares and determines the input face image and the basic face image using the generated feature points, and calculates a vector value using a coordinate difference for the feature point where the difference occurs, and
An emotion result calculation module that corresponds to the upper two emotions received from the determination module, and selects emotions corresponding to the upper two among the six emotions combined by applying the vector value calculated by the vector value calculation module to the CNN algorithm Facial emotion recognition device comprising a. The method of claim 8,
The vector value calculation module,
Facial emotion recognition device for calculating a vector value by the following equation:

here, (

,

) Is the coordinate value for the feature point when there is an expression, and (

,

) Is the coordinate value for the feature point when there is no expression. delete The method of claim 1,
The emotion classification unit,
Facial emotion recognition apparatus for calculating result values for the upper two emotions by the following equation:

here,

Is the recognition value for the emotion extracted from the facial feature extraction unit,

Is a vector value for the emotion extracted from the geometric change amount extraction unit,

Represents the weight value. The method of claim 11,
The weight value is set to a value between 0.7 and 0.8. In a method of identifying emotion from an input face image using a facial emotion recognition device,
Extracting features for the entire face image from the face image subject to emotion recognition, and extracting information on the upper two emotions among the six emotions obtained from the extracted facial features,
The input face image and the feature points assigned to the basic face image without expression for the same person are compared to calculate the difference in coordinates, and the calculated difference value of the coordinates is applied to the CNN algorithm for the upper two emotions. Calculating the vector value, and
Each result value for the upper two emotions is calculated by assigning a weight to each of the vector values for the upper two emotions calculated using the difference between the recognition values for the upper two emotions calculated from the facial features and the feature point, and And classifying the emotion with the highest value among the calculated result values as the final emotion,
Extracting information on the upper two emotions by using the facial features,
Receiving a face image to be the target of the emotion recognition,
Separating a face area from the input face image
Generating a basic face image from the input face image by applying the separated face image to an autoencoder,
Applying blurring to the face image where only the face area is separated to remove noise, and then extracting main feature information of the face mainly in main action units using a local binary pattern (LBP) algorithm, And
Computing six emotion results by applying the extracted main feature information of the face to the CNN algorithm, and classifying emotions corresponding to the top two among the calculated six emotion results,
The step of classifying the final emotion,
Receiving information on the upper two emotions extracted from the facial features,
The step of calculating the received upper two emotions as recognition values, and
A weight is assigned to the vector values for the upper two emotions calculated by using the calculated recognition values for the upper two emotions and the difference between the feature points to calculate the result values for the upper two emotions, and the calculated A facial emotion recognition method comprising the step of determining an emotion to which the highest value is assigned among the result values as an emotion for a face image to be an emotion recognition target. The method of claim 13,
The above emotion is,
6 ways to recognize facial emotions, including happiness, sadness, fear, disgust, surprise and anger. delete The method of claim 13,
The step of generating the basic face image,
Creating a dataset by learning an error arising from the difference in feature values between two face images using a facial image with an expression and a basic facial image without an expression for the same person, and
And generating a basic face image from the face image that is the emotion recognition target by using an autoencoder to which the generated data set is applied. delete The method of claim 13,
The LBP algorithm,
Facial emotion recognition method calculated by the following equation:

,

here,

Is the current pixel selected in the input face image,

Is

Are pixels distributed in the neighborhood of

Is a value calculated as 0 or 1 by comparing the current pixel with a neighboring pixel, and P is 8, which is the number of neighboring pixels to be compared. Therefore, p is given from 0 to 7 in a clockwise direction from a neighboring pixel located at the top left of the current pixel. The method of claim 13,
The step of classifying the emotions corresponding to the top two,
Calculating six emotions by applying a plurality of convolutions to the input face image to which the LBP algorithm is applied,
A facial emotion recognition method comprising the step of classifying emotions corresponding to the upper two among the calculated six emotions. The method of claim 13,
The step of calculating vector values for the upper two emotions,
Receiving a face image to be the target of the emotion recognition,
Generating feature points based on action units in the input face image and the basic face image,
Comparing and determining the input face image and the basic face image using the generated feature point, and calculating a vector value using a coordinate difference for the feature point where the difference is generated, and
And applying the calculated vector value to a CNN algorithm to select an emotion corresponding to the upper two among the six emotions combined. The method of claim 20,
The step of calculating the vector value,
Facial emotion recognition method for calculating a vector value by the following equation:

here, (

,

) Is the coordinate value for the feature point when there is an expression, and (

,

) Is the coordinate value for the feature point when there is no expression. delete The method of claim 13,
The step of classifying the final emotion,
Facial emotion recognition method for calculating result values for the top two emotions by the following equation:

here,

Is the recognition value for the emotion extracted from the facial feature extraction unit,

Is a vector value for the emotion extracted from the geometric change amount extraction unit,

Represents the weight value. The method of claim 23,
The weight value is set to a value between 0.7 and 0.8.

Images