KR102187396B1 - Learning method and apparatus for facial expression recognition, facial expression recognition method using electromyogram data - Google Patents

Abstract

Disclosed is a learning method and apparatus for increasing a recognition success rate of facial expression recognition using EMG data. The disclosed learning method for facial expression recognition includes: extracting a first feature value from EMG data of a user's face portion for a plurality of facial expressions; And learning a facial expression class for the EMG data by using the first feature value and a second feature value obtained from reference EMG data for the facial expression, wherein the facial expression class for the EMG data The learning step includes: calculating a first average value and a first covariance value of the first feature value for the facial expression class; Generating a first parameter by weighting averaging a second average value and the first average value of the second feature value for the facial expression class; And generating a second parameter by a weighted average of the second covariance value and the first covariance value of the second feature value for the facial expression class, wherein the first and second parameters are LDA models. This parameter is for.

Description

Learning method and device for facial expression recognition, facial expression recognition method using EMG data {LEARNING METHOD AND APPARATUS FOR FACIAL EXPRESSION RECOGNITION, FACIAL EXPRESSION RECOGNITION METHOD USING ELECTROMYOGRAM DATA}

The present invention relates to a facial expression recognition method, and more particularly, to a learning method and apparatus for facial expression recognition, and a facial expression recognition method using EMG data.

Currently, facial expression recognition technology is being studied in two directions. One is to recognize facial expressions based on images, and the other is to recognize facial expressions based on an electromyogram signal of the face.

In the case of image-based facial expression recognition technology, since the camera must be positioned in front of the user's field of view, the user's field of view may be obscured by the camera, and if the user wears equipment such as an HMD, the performance of facial expression recognition may deteriorate. have. With the recent development of virtual reality technology, various virtual reality services are being provided, and there are attempts to fuse the facial expressions of users receiving virtual reality services into virtual reality services. Image-based facial expression recognition technology is not suitable for these services. not.

Therefore, in recent years, a lot of technologies for recognizing a user's facial expression based on an EMG signal by attaching an EMG measuring electrode to an HMD have been developed.

Related prior documents include Korean Patent Application Publication No. 2016-0024725 and Korean Patent Registration No. 10-1633057.

The present invention is to provide a learning method and apparatus for increasing a recognition success rate of facial expression recognition using EMG data.

In addition, the present invention is to provide a facial expression recognition method using EMG data for reducing misrecognition of facial expressions that occurs in a process of changing facial expressions by a user.

According to an embodiment of the present invention for achieving the above object, extracting a first feature value from EMG data of a user's face for a plurality of facial expressions; And learning a facial expression class for the EMG data by using the first feature value and a second feature value obtained from reference EMG data for the facial expression, wherein the facial expression class for the EMG data The learning step includes: calculating a first average value and a first covariance value of the first feature value for the facial expression class; Generating a first parameter by weighting averaging a second average value and the first average value of the second feature value for the facial expression class; And generating a second parameter by a weighted average of the second covariance value and the first covariance value of the second feature value for the facial expression class, wherein the first and second parameters are LDA models. A learning method for facial expression recognition, which is a parameter for, is provided.

In addition, according to another embodiment of the present invention for achieving the above object, a feature value extracting unit for extracting a first feature value from EMG data of a user's face for a plurality of facial expressions; And a facial expression learning unit for learning a facial expression class for the EMG data by using the first feature value and a second characteristic value obtained from reference EMG data for the facial expression, wherein the facial expression learning unit An operation unit that calculates a first average value and a first covariance value of the first feature value for an expression class; And a weighted average of the second average value and the first average value of the second feature value for the facial expression class to generate a first parameter, and a second covariance value of the second feature value for the facial expression class, and A learning apparatus for recognizing facial expressions is provided, comprising a parameter generator configured to generate a second parameter by weighting the first covariance value, wherein the first and second parameters are parameters for an LDA model.

In addition, according to another embodiment of the present invention for achieving the above object, the step of receiving EMG data of the user's face; Recognizing facial expressions for the EMG data at a preset recognition period using the learning data; And outputting the successively recognized results as a current final recognition result when the results continuously recognized by a preset number are the same, a facial expression recognition method using EMG data.

According to the present invention, the facial expression recognition rate may be improved by additionally performing learning by using training data indicating a facial expression recognition rate equal to or greater than a threshold value.

In addition, according to the present invention, by outputting the continuous facial expression recognition result as the final recognition result when the continuous facial recognition result is equal to or greater than a threshold value, it is possible to reduce misrecognition of facial expressions occurring in the facial expression transfer process.

1 is a view for explaining a learning device for facial expression recognition according to an embodiment of the present invention.
2 is a view for explaining a learning method for facial expression recognition according to an embodiment of the present invention.
3 is a diagram showing a facial expression recognition rate when reference EMG data is used and when the reference EMG data is not used.
4 is a diagram for describing a facial expression recognition method using EMG data according to an embodiment of the present invention.
5 is a view for explaining an example of a facial expression recognition result according to an embodiment of the present invention.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to a specific embodiment, it is to be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention. In describing each drawing, similar reference numerals have been used for similar elements.

The present invention relates to a facial expression recognition method using an EMG signal of a facial part, and recognizes a user's facial expression using machine learning.

Recognizing a user's facial expression has been applied in various fields, and recently, it has been used in the virtual reality field to provide a service for generating an avatar in a virtual space reflecting the user's expression.

As training data for learning facial expressions, an EMG signal of a user's face for a plurality of previously given facial expressions may be used. Since the EMG signals measured by the facial part show different features according to facial expressions, the EMG signals can be used as training data. In addition, the greater the difference in the features of the EMG signal according to the facial expression, the greater the learning effect, and the difference in the features of the EMG signal according to the facial expression may be maximized as the difference between the facial expressions increases. However, it may be difficult for the user to make facial expressions so that the difference in EMG characteristics according to facial expressions increases, unless various facial expressions are practiced like an actor, which leads to a decrease in the learning effect.

Accordingly, the present invention proposes a learning method capable of increasing the recognition rate of facial expressions by using both training data of a user who has sufficiently practiced various facial expressions and training data that is not.

Meanwhile, as described above, the learning method for facial expressions uses EMG data for facial expressions given in advance. Therefore, in the facial expression recognition process, one of the previously given facial expressions is output as a recognition result. However, since the expression in the process of changing the facial expression by the user is not included in the facial expression given in advance, the result of the user's expression being misrecognized may occur in the process of changing the facial expression by the user.

For example, when the user's expression changes from a smiling expression to a surprise expression, an incorrect expression recognition result may be output for an intermediate expression in the process of changing the expression.

Accordingly, the present invention proposes a facial expression recognition method capable of reducing facial expression misrecognition that may occur when a user's expression is changed, that is, when a user's expression is transferred.

The learning method and the facial expression recognition method according to the present invention may be performed in a computing device including a processor. Depending on the embodiment, the learning method and the facial expression recognition method may be performed by one computing device, or may be performed by a separate learning device and a facial expression recognition device.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a view for explaining a learning device for facial expression recognition according to an embodiment of the present invention.

Referring to FIG. 1, the learning apparatus according to the present invention includes a feature value extracting unit 110 and a facial expression learning unit 210.

The feature value extracting unit 110 extracts a first feature value from EMG data of the user's face for a plurality of facial expressions. As an embodiment, EMG data may be provided from an HMD worn by a user, and an electrode for acquiring an EMG signal of a user's face may be attached to the HMD.

The feature value extraction unit 110 may extract a feature value by pre-processing the provided EMG data, and various values capable of well representing the features of facial expressions may be used as the feature value. As an embodiment, the feature value may be a sample entropy, a root mean square (RMS), a wave length, or a cepstral coefficient for an EMG signal.

The facial expression learning unit 120 learns a facial expression class for EMG data by using a first feature value and a second feature value obtained from reference EMG data for a plurality of facial expressions. That is, the facial expression learning unit 120 learns a facial expression class for the user's EMG data by using the reference EMG data as well as training data called EMG data of the user's face.

The reference EMG data is training data indicating a facial expression recognition rate equal to or greater than a threshold value, and may be data obtained from a facial portion of a subject on which facial expression training has been performed. In other words, the reference EMG data may be training data indicating that a recognition rate when facial expression recognition is performed from a learning result using the reference EMG data is greater than or equal to a threshold value.

According to the present invention, the facial expression recognition rate may be improved by additionally performing learning by using training data indicating a facial expression recognition rate equal to or greater than a threshold value.

The facial expression learning unit 120 more specifically includes an operation unit 121 and a parameter generation unit 123.

The operation unit 121 calculates a first average value and a first covariance value of the first feature values for various facial expression classes such as a smiling expression, a crying expression, and an angry expression. That is, the first average value of the first feature value extracted from the EMG data obtained when the user makes a smiling expression and the first average value of the first feature value extracted from the EMG data obtained when the user makes a crying expression may be calculated. In addition, the first covariance value may be calculated by calculating the covariance for the first feature value of each facial expression class and averaging the covariance calculated for each facial expression class. In addition, according to an embodiment, the second average value and the second covariance value of the second feature value for the facial expression class may be calculated, and the method of calculating the second average value and the second covariance value includes the first average value and the first covariance value. It is the same as how to calculate.

The parameter generator 123 generates a first parameter by weighted average of the second average value and the first average value of the second feature value for each facial expression class, and the second covariance value and the first covariance of the second feature value The values are weighted averaged to generate a second parameter. That is, first and second parameters are generated for each facial expression class, and the first and second parameters are parameters for a linear discriminant analysis (LDA) model.

LDA, as a method of machine learning, is a method used to classify given data into several classes. As parameters of the LDA model, the average and covariance values of data included in each class are used. When the average and covariance values are determined, a straight line that divides the data into several classes is determined. In the LDA-based learning method, learning the facial expression class for the EMG data can be said to determine the mean value and the covariance value, which are parameters of the LDA model.

As described above, the first parameter is the weighted average result of the first and second average values, corresponding to the average value for the LDA model, and the second parameter is the weighted average result of the first and second covariance values, for the LDA model. Corresponds to the covariance value.

In this way, when a parameter for the LDA model is determined, a facial expression class for the user's target EMG data may be determined using the learned LDA model, and the facial expression class may be output as the user's facial expression.

2 is a diagram for explaining a learning method for facial expression recognition according to an embodiment of the present invention, and FIG. 3 is a diagram illustrating a facial expression recognition rate when reference EMG data is used and when reference EMG data is not used. In FIG. 2, a learning method of a learning device is described as an embodiment.

The learning apparatus according to the present invention extracts a first feature value from EMG data of a user's face for a plurality of facial expressions (S210), and according to an embodiment, a second feature value from reference EMG data for a plurality of facial expressions. It can also be extracted. In this case, the learning apparatus may extract feature values (including first and second feature values) by pre-processing the EMG data to reduce deviation of the EMG data.

As an example, the learning apparatus first performs band pass filtering on an EMG signal provided from an HMD electrode in a frequency band of 20 to 450 Hz, moves a window of a preset size, and extracts EMG data. In addition, feature values are extracted from the result of applying the Riemann-based kernel to the EMG data extracted corresponding to each window size in the time domain. The Riemann-based kernel is a kind of filter that is widely used in the field of BCI (Brain Computer Interface).

Thereafter, the learning apparatus learns a facial expression class for the EMG data by using the first feature value and the second feature value, and, as an embodiment, the learning may be performed using the LDA model.

In more detail, the learning apparatus calculates a first average value and a first covariance value of the first feature value for the facial expression class (S220). Then, the second average value and the first average value of the second feature value for the facial expression class are weighted to generate a first parameter (S230), and the second covariance value and the first value of the second feature value for the facial expression class are The weighted average of the covariance values is performed to generate a second parameter (S240). The first and second parameters are parameters for the LDA model.

As an example, the learning device may generate first and second parameters as shown in [Equation 1].

는 제2평균값을 나타내며,

는 제1평균값을 나타낸다. 그리고

는 제2공분산값을 나타내며,

는 제2평균값을 나타낸다. α,β는 각각 제1 및 제2파라미터를 위한 가중치로서, 0보다 크고 1보다 작은 범위에서 결정될 수 있다.here,

Represents the second average value,

Represents the first average value. And

Represents the second covariance value,

Represents the second average value. α and β are weights for the first and second parameters, respectively, and may be determined in a range greater than 0 and less than 1.

For example, if there are a total of 11 facial expression classes, the first and second average value vectors including 11 average value elements can be calculated, and the first parameter can also be calculated as one vector including 11 weighted average value elements. I can. In addition, the first and second covariance values may be calculated in the form of a matrix having a size of 11×11, and a second parameter representing the weighted averaged covariance value may also be calculated in the form of a matrix having a size of 11×11.

Referring to FIG. 3, it can be seen that the case of additionally using reference EMG data, which is training data representing the recognition rate of facial expressions equal to or greater than a threshold value, has a higher recognition rate than the case of using only EMG data for a user. As an example, when using only EMG data for a user, the recognition rate is 83.73%, but when α and β are 0.7 and 0.8, respectively, the recognition rate is 86.70%, and when reference EMG data is additionally used, the recognition rate is better.

In FIG. 3, a red bar represents a recognition rate when reference EMG data is additionally used, and a blue bar represents a recognition rate when only EMG data for a user is used. In FIG. 3, the x-axis represents various weights for the first and second parameters, and the y-axis represents the recognition rate.

When the weights (1-α, 1-β) applied to the first average value and the first covariance value are smaller than the weights (α, β) applied to the second average value and the second covariance value, the recognition rate increases, and the second When the weight (β) applied to the covariance value is greater than the weight (α) applied to the second average value, the recognition rate tends to increase, so the learning device can generate the first and second parameters by applying the weight in this way. I can.

4 is a diagram for explaining a facial expression recognition method using EMG data according to an embodiment of the present invention, and FIG. 5 is a view for explaining an example of a facial expression recognition result according to an embodiment of the present invention. In FIG. 4, a method of recognizing a facial expression by an apparatus for recognizing a facial expression is described as an embodiment.

The facial expression recognition apparatus according to the present invention may recognize a facial expression for the user's EMG data by using the LDA model to which the first and second parameters generated as shown in FIG. 2 are applied.

To this end, the facial expression recognition apparatus first receives (S410), that is, the EMG data of the user's face, and recognizes the facial expression on the EMG data at a preset recognition period using the learning data (S420). Here, the training data is the LDA model described above.

Further, the facial expression recognition apparatus does not output the recognition result in step S420 as the final recognition result as it is, but outputs the final recognition result according to the identity of the continuously recognized results by a preset number. That is, the facial expression recognition apparatus outputs one of the recognition results in step S420 as the final recognition result according to the sameness of the recognition result at the current recognition point and the recognition result from the current recognition point to the past recognition point corresponding to a preset number. do.

The apparatus for recognizing facial expressions is an embodiment, and when the result of consecutive recognition as many as a preset number is the same, outputs the continuously recognized result as the current final recognition result (S430). If the results continuously recognized by the preset number are not the same, the facial expression recognition apparatus outputs the previous final recognition result as the current final recognition result.

For example, in a process in which the user changes a facial expression from a neutral expression to a surprise expression, the facial expression recognition apparatus may recognize the facial expression as shown in FIG. 5 in step S420. In FIG. 5, 1 represents a neutral expression, 2 represents a sad expression, and 5 represents a surprise expression. Although the user changes the facial expression from the expressionless expression (1) to the surprise expression (5), in the process 510 of changing the facial expression, a sad expression may be recognized. Although the user did not make a sad expression in the process of changing facial expressions, the facial expression corresponding to the expression in the expression transfer process was not learned, so the expression in the expression transfer process was misrecognized as a sad expression.

Since the facial expression transfer process cannot be omitted, it is not possible to learn all the various expressions that may occur in the facial expression transfer process, and the facial expression in the facial expression transfer process does not need to be recognized. In order to prevent, as an example, when the recognition results in step S420 are the same as four consecutively, the results continuously recognized in step S430 are output as the current final recognition results.

Therefore, referring to FIG. 5, due to the user's expression in the process 510 of changing the facial expression, in a state in which the four consecutively recognized results in step S420 are not the same, the facial expression recognition apparatus performs the previous final recognition result. The human expression (1) is output in step S430 as the current final recognition result. After the process 510 of changing the facial expression, since all four results continuously recognized in step S420 are the expressions of surprise 5, the facial expression recognition apparatus outputs the expression of surprise 5 in step S430 as the final recognition result.

As a result, the recognition result finally output in step S430 may consist of only the expressionless expression (1) and the surprise expression (5) without a sad expression, and a sad expression may be prevented from being misrecognized.

In the experimental environment in which the user was instructed to make a sad expression while making a expressionless expression, the false recognition rate in step S420 was 3.13%, but the false recognition rate in step S430 was reduced to 1.72%.

As a result, according to the present invention, by outputting the continuous facial expression recognition result as the final recognition result when the continuous facial recognition result is equal to or greater than the threshold value, it is possible to reduce misrecognition of facial expressions occurring in the facial expression transfer process.

Meanwhile, the preset number in step S430 may be determined according to the user's expression change speed. The preset number may be set to increase as the user's expression change speed is slower, and the preset number may be set to decrease as the user's expression change speed increases.

In addition, the user's expression change speed may be estimated from the recognition result in step S420 or may be directly set by the user in consideration of individual characteristics.

The facial expression recognition apparatus may count results that are continuously recognized, and determine the speed of change of the user's facial expressions by using the number of recognition results counted to be less than the number of counts before and after the current counting time. Here, counting means counting the results that are continuously recognized. As an example, the facial expression recognition apparatus may determine an inverse number of the number of recognition results counted to be less than the number of counting before and after, as the speed of change of the user's expression.

As shown in FIG. 5, the number of recognition results counted in the process 510 of changing the facial expression is 3, which is less than the number of counting 4 recognized as a previous expressionless expression and the number 4 recognized as a sad expression afterwards. Accordingly, the facial expression recognition apparatus may determine the user's expression change speed using the number of recognition results in the process 510 of changing the facial expression, and determine 1/3 as the user's expression change speed.

The technical details described above may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiments, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, in the present invention, specific matters such as specific components, etc., and limited embodiments and drawings have been described, but this is provided only to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , If a person of ordinary skill in the field to which the present invention belongs, various modifications and variations are possible from these descriptions. Therefore, the spirit of the present invention is limited to the described embodiments and should not be defined, and all things that are equivalent or equivalent to the claims as well as the claims to be described later fall within the scope of the spirit of the present invention. .

Claims (9)

Extracting a first feature value from EMG data of a user's face part for a plurality of facial expressions of a user of the facial expression recognition device; And
Using the first feature value and a second feature value obtained from reference EMG data for the facial expression, learning a facial expression class for the EMG data,
Learning the facial expression class for the EMG data
Calculating a first average value and a first covariance value of the first feature value for the facial expression class;
Generating a first parameter by a weighted average of the second average value and the first average value of the second feature value for the facial expression class; And
And generating a second parameter by weighting averaging a second covariance value of the second feature value and the first covariance value for the facial expression class,
The first and second parameters are parameters for the LDA model,
The reference EMG data is training data indicating a facial expression recognition rate equal to or greater than a threshold value,
Learning method for facial expression recognition.
The method of claim 1,
The weight applied to the first average value and the first covariance value is
A weight that is smaller than the weight applied to the second average value and the second covariance value
Learning method for facial expression recognition.
The method of claim 2,
The weight applied to the second covariance value is
A weight greater than the weight applied to the second average value
Learning method for facial expression recognition.
delete delete Receiving EMG data of the user's face;
Recognizing facial expressions for the EMG data at a preset recognition period using the learning data; And
In the case where the results continuously recognized as much as a preset number are the same, outputting the continuously recognized results as a current final recognition result,
The step of outputting the continuously recognized result as a final recognition result
If the results continuously recognized by the preset number are not the same, outputting the previous final recognition result as the current final recognition result
Facial expression recognition method using EMG data.
delete The method of claim 6,
The preset number is
Determined according to the user's expression change speed
Facial expression recognition method using EMG data.
The method of claim 8,
Counting the successively recognized results; And
Determining the user's expression change speed using the number of recognition results counted less than the number of counting before and after the current counting point
Facial expression recognition method using EMG data further comprising a.

Images