KR102216360B1 - Emotion recognition method and device using electromyogram signal - Google Patents

Abstract

Disclosed is a method and apparatus for recognizing emotions using an EMG signal, capable of inferring a user's emotion from a user's facial expression. An emotion recognition method using the disclosed EMG signal includes the steps of receiving an EMG signal for a user's face; Performing preprocessing on the EMG signal; Extracting a feature value from the preprocessed EMG signal; And recognizing an emotion for the user by using the feature value, wherein the feature value includes two or more of sample entropy, RMS, wave length, and cepstral coefficient. .

Description

Emotion recognition method and apparatus using an electromyogram signal TECHNICAL FIELD [EMOTION RECOGNITION METHOD AND DEVICE USING ELECTROMYOGRAM SIGNAL]

The present invention relates to an emotion recognition method and device, and more particularly, to an emotion recognition method and device using an EMG signal of a user's face.

Currently, facial expression recognition technology is being studied in two directions. One is to recognize facial expressions based on images, and the other is to recognize facial expressions based on an electromyogram (EMG) signal of the face.

In the case of image-based facial expression recognition technology, since the camera must be positioned in front of the user's field of view, the user's field of view may be obscured by the camera, and if the user wears equipment such as an HMD, the performance of facial expression recognition may deteriorate. have. With the recent development of virtual reality technology, various virtual reality services are being provided, and there are attempts to fuse the facial expressions of users receiving virtual reality services into virtual reality services. Image-based facial expression recognition technology is not suitable for these services. not.

Therefore, in recent years, a lot of technologies for recognizing a user's facial expression based on an EMG signal by attaching an EMG measuring electrode to an HMD have been developed. Since the degree of activation of facial muscles differs according to facial expressions (laughter, anger, sadness, etc.), the user's facial expression can be recognized using an EMG signal representing such a different pattern.

Related prior documents include Korean Patent Application Publication No. 2016-0024725 and Korean Patent Registration No. 10-1633057.

An object of the present invention is to provide a method and apparatus for recognizing emotions using an EMG signal capable of inferring a user's emotion from a user's facial expression.

According to an embodiment of the present invention for achieving the above object, receiving an EMG signal for a user's face; Performing preprocessing on the EMG signal; Extracting a feature value from the preprocessed EMG signal; And recognizing an emotion for the user by using the feature value, wherein the feature value includes at least two of sample entropy, RMS, wave length, and cepstral coefficient. An emotion recognition method using an EMG signal is provided.

In addition, according to another embodiment of the present invention for achieving the above object, by using the first EMG signal of the user's face part, recognizing emotions for the user; Analyzing the accuracy of the recognized emotion; Determining a candidate emotion group including an emotion selected according to the accuracy from among the plurality of recognized emotions; And recognizing one of the emotions included in the candidate emotion group as an emotion for the user by using the second EMG signal of the user's face.

In addition, according to another embodiment of the present invention for achieving the above object, there is provided a wearable device type emotion recognition apparatus using an EMG signal, comprising: a plurality of electrodes for measuring an EMG signal of a user's face; An emotion recognition unit for extracting a feature value from the EMG signal and recognizing an emotion for a user using the feature value; And there is provided an emotion recognition apparatus using an EMG signal including an image processing unit that generates an avatar for the user based on the recognized emotion.

According to the present invention, by using the EMG signal to the user's face, it is possible to infer an emotion according to a change in the user's facial expression.

In addition, according to the present invention, the accuracy of emotion recognition may be improved by analyzing the accuracy of the emotion recognized in advance and recognizing the emotion of the user based on the analyzed result.

In addition, according to the present invention, by adaptively determining the emotion used for machine learning according to the accuracy of emotion recognition and individual emotion patterns, it is possible to reduce the time required for emotion learning.

1 is a diagram for describing an emotion recognition apparatus using an EMG signal according to an embodiment of the present invention.
2 is a view for explaining an emotion recognition method using an EMG signal according to an embodiment of the present invention.
3 is a diagram for explaining an emotion recognition method using an EMG signal according to another embodiment of the present invention.
4 is a diagram showing a confusion matrix for eight emotions.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to a specific embodiment, it is to be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention. In describing each drawing, similar reference numerals have been used for similar elements.

The present invention relates to a method of recognizing a user's emotion from a user's facial expression, and proposes a method of recognizing a user's emotion by using an EMG signal to a user's face.

Since facial expressions vary according to emotions, emotions for the user can be inferred from facial expressions. In addition, since the degree of activation of the facial muscles differs according to such facial expressions, the EMG signal for the facial part changes according to the facial expression, and as a result, emotions for the user can be inferred from the EMG signals for the user's facial part.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram for describing an emotion recognition apparatus using an EMG signal according to an embodiment of the present invention.

The emotion recognition device according to the present invention is a wearable type device, and may be a head mounted display (HMD) type worn on a user's face as an embodiment.

The emotion recognition apparatus according to the present invention includes a plurality of electrodes 110, an emotion recognition unit 120, and an image processing unit 130.

The plurality of electrodes 110 may measure a muscle level signal of the user's face, and may measure an EMG signal by contacting the user's eyeball periphery. The number and position of electrodes may be variously determined according to exemplary embodiments.

The emotion recognition unit 120 extracts a feature value from the EMG signal, and recognizes an emotion for a user using the extracted feature value. In an embodiment, two or more of sample entropy, root mean square (RMS), wavelength, and cepstral coefficient may be used as feature values.

As an embodiment, the emotion recognition unit 120 may recognize emotions for a user using learning data. Machine learning may be performed using EMG signals obtained from a user's reference facial expressions for various emotions, and learning data obtained from machine learning may be used for emotion recognition. If the user makes facial expressions similar to the reference facial expressions used in machine learning, the EMG signals will also be similar, so that emotions corresponding to the reference facial expressions may be recognized as emotions toward the user.

The image processing unit 130 generates an avatar for the user based on the recognized emotion. That is, the image processing unit 130 generates an avatar indicating the current emotional state of the user.

Most virtual reality devices are provided in the form of an HMD, and the image processing unit 130 may generate an avatar representing a user's emotions in a virtual reality space. A user can use various virtual reality services using such an avatar.

2 is a view for explaining an emotion recognition method using an EMG signal according to an embodiment of the present invention.

The emotion recognition method according to the present invention may be performed in a computing device including a processor. Hereinafter, the emotion recognition method of the emotion recognition apparatus described in FIG. 1 will be described, but the emotion recognition apparatus in which the emotion recognition method according to the present invention is performed does not need to be limited to a wearable type.

The emotion recognition apparatus according to the present invention receives an EMG signal for a user's face (S210), and performs preprocessing on the EMG signal (S220).

The EMG signal may be a signal measured by a wearable device worn on the user's face or may be provided from a separate EMG measurement device. The emotion recognition apparatus may perform preprocessing on the input EMG signal using a band pass filter of 20 to 450 Hz.

The emotion recognition apparatus according to the present invention extracts a feature value from the preprocessed EMG signal (S230), and, as described above, may extract two or more of sample entropy, RMS, wave length, and capstral coefficient as a feature value. .

In step S230, the emotion recognition apparatus moves a window of a preset size with respect to the EMG signal measured for a preset time, and extracts a feature value for each window. For example, the EMG signal for the user's face may be a signal measured for 3 seconds, and a feature value is extracted for a 0.1 second window. In other words, feature values are extracted for the EMG signal for 0.1 second, and feature values are extracted for each window section.

In this case, the emotion recognition apparatus may extract a feature value by modeling the pre-processed EMG signal with autoregressive (AR). As an example, autoregressive modeling may be performed as shown in [Equation 1].

는 현재 근전도 신호,

는 이전 근전도 신호를 나타낸다. 그리고 C는 상수이며, a_i는 모델링에서 계산되는 계수이다.

는 잡음 성분을 나타낸다.[Equation 1] represents the fourth-order autoregressive modeling, where,

Is the current EMG signal,

Represents the previous EMG signal. And C is a constant, and a _i is a coefficient calculated in modeling.

Represents the noise component.

From the equation modeled as above, the capstral coefficient can be calculated as in [Equation 2]. EMG signals are modeled for each window, and the capstral coefficients (c ₁ , c _p ) for each window are calculated through [Equation 2].

범위의 정수이다.Where a _p corresponds to the coefficients of the fourth-order autoregressive model, and p is

It is an integer in the range.

In addition, RMS, sample entropy, and wave length may be calculated by sampling the EMG signals in each window section without additional modeling. RMS represents the effective value for the signal value in the window section, and the wave length corresponds to the length of the signal in the window section. And sample entropy is a kind of statistical parameter used to characterize a continuous signal (Richman, JS; Moorman, JR, Physiological time-series analysis using approximate entropy and sample entropy, Journal of Physiology.Heart and Circulatory Physiology, 2000) ).

The emotion recognition apparatus according to the present invention recognizes the emotion of the user by using the extracted feature value (S240).

As described above, the emotion recognition apparatus may infer a user's emotion through machine learning, and as an example, a linear discriminant analysis (LDA) algorithm may be used.

The emotion recognition apparatus recognizes emotions for the user from feature values for each window, and may determine a final emotion for the user according to the number of recognized emotions. For example, if sadness is the most among the emotions recognized for 30 windows, the emotion recognition apparatus may determine the final emotion for the user as sadness.

After all, according to the present invention, by using the EMG signal for the user's face, it is possible to infer emotions according to the user's facial expression changes.

Meanwhile, the emotion recognition apparatus according to the present invention can increase the accuracy of the emotion recognition by analyzing the accuracy of the recognized emotion and using the analysis result, which will be described in more detail with reference to FIG. 3.

3 is a diagram for explaining an emotion recognition method using an EMG signal according to another embodiment of the present invention, and FIG. 4 is a diagram illustrating a confusion matrix for eight emotions.

The emotion recognition apparatus according to the present invention recognizes the emotion of the user by using the first EMG signal of the user's face (S310). As described in FIG. 2, emotions may be recognized using feature values of the first EMG signal and machine learning.

In addition, the emotion recognition apparatus analyzes the accuracy of the emotion recognized through the first EMG signal (S320). Accuracy may be analyzed by comparing the actual emotion of the user with the result recognized in step S310, and accuracy may be analyzed through a confusion matrix as shown in FIG. 4.

4 is a diagram showing a confusion matrix obtained by comparing eight emotions of a user with an emotion recognition result according to the present invention, and shows the accuracy of the recognized emotions. The confusion matrix represents the number of output classes of the user's real emotion (target class).

4 shows a total of 6840 emotion recognition results for each of the eight emotions. For example, the number of times the user's actual emotion, anger, is recognized as anger is 4263 times, and the number of times that is recognized as ridicule is 75 times. In addition, the number of times the user's actual feelings of ridicule are recognized as ridicule is 6179 times, and the number of times it is recognized as disgust is 156 times. In FIG. 4, the green value (%) in the bottom row indicates the accuracy of emotion recognition, and the red value (%) indicates the error rate of emotion recognition.

4, the accuracy of emotion recognition for happiness (93.4%), ridicule (90.3%), and surprise (75.6%) is higher than 75%, sadness (67.6%), expressionlessness (67.5%), It can be seen that the accuracy of emotion recognition for disgust (66.6%), anger (62.3%), and fear (61.1%) is relatively low. Depending on the type of emotion, since there may be emotions with similar facial expressions, emotions with low recognition accuracy may exist.

The emotion recognition apparatus determines a candidate emotion group including an emotion selected according to accuracy from among a plurality of emotions recognized in step S310 (S330), and uses the second EMG signal of the user's face to determine the emotion included in the candidate emotion group. One of them is recognized as an emotion toward the user (S340).

As an embodiment, the emotion recognition apparatus may determine a candidate emotion group including ridicule, happiness, and surprise classified as having high accuracy in FIG. 4.

Alternatively, the emotion recognition apparatus may determine a candidate emotion group in consideration of not only emotion recognition accuracy, but also emotions that the user often feels in real life, that is, the user's emotion pattern. For example, if the emotions that the user frequently feels in real life are happiness, surprise, sadness, anger, and fear, among them, a candidate emotion group including a preset number of emotions may be determined in the order of high accuracy. If the preset number is 4, four emotions of happiness, surprise, sadness, and anger may be included in the candidate emotion group. The user's emotional pattern may be set by the user.

Learning may be performed using feature values of EMG signals for emotions included in the candidate emotion group, and the emotion recognition device recognizes one of the emotions included in the candidate emotion group as the user's emotion based on the learning data. can do. Like in step S310, the emotion recognition apparatus may recognize an emotion toward the user by using the feature value extracted from the second EMG signal in step S340.

[Table 1] is a table showing the accuracy of emotion recognition results targeting the eight emotions of FIG. 4, and [Table 2] is an emotion using a candidate emotion group including four emotions: happiness, surprise, sadness, and anger. This is a table showing the accuracy of the recognition results. In the case of using the candidate emotion group, it can be seen that the accuracy has increased by 10% or more.

Consequently, according to the present invention, the accuracy of emotion recognition may be improved by analyzing the accuracy of the emotion recognized in advance and recognizing the emotion of the user based on the analyzed result.

In addition, according to the present invention, by adaptively determining the emotion used for machine learning according to the accuracy of emotion recognition and individual emotion patterns, it is possible to reduce the time required for emotion learning.

Meanwhile, according to an embodiment, the feature value extracted from the second EMG signal may be determined according to the accuracy of the candidate emotion group. For example, if the accuracy of the emotion included in the candidate emotion group is generally low, the accuracy of emotion recognition may be improved by extracting all four feature values. On the other hand, if the accuracy of emotions included in the candidate emotion group is generally high, by extracting three or less feature values, it is possible to reduce the amount of computation required for emotion recognition while maintaining the recognition performance to an acceptable level.

The technical details described above may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiments, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, in the present invention, specific matters such as specific components, etc., and limited embodiments and drawings have been described, but this is provided only to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , If a person of ordinary skill in the field to which the present invention belongs, various modifications and variations are possible from these descriptions. Therefore, the spirit of the present invention is limited to the described embodiments and should not be defined, and all things that are equivalent or equivalent to the claims as well as the claims to be described later fall within the scope of the spirit of the present invention. .

Claims (10)

delete delete delete delete delete In the emotion recognition method using an EMG signal performed in a computing device,
Recognizing an emotion toward the user by using the first EMG signal on the user's face;
Analyzing the accuracy of the recognized emotion;
Determining a candidate emotion group including an emotion selected according to a user's emotion pattern and the accuracy from among the plurality of recognized emotions; And
Recognizing one of the emotions included in the candidate emotion group as an emotion toward the user using a second EMG signal of the user's face,
Recognizing one of the emotions included in the candidate emotion group as an emotion for the user
Using the feature value extracted from the second EMG signal, to recognize the emotion of the user,
The feature value includes at least two of sample entropy, RMS, wave length, and cepstral coefficient,
The number of feature values extracted from the second EMG signal is
Determined according to the accuracy of the emotion included in the candidate emotion group
An emotion recognition method using EMG signals.
delete delete The method of claim 6,
The first and second EMG signals are
The signal measured by the wearable device worn on the user's face
An emotion recognition method using EMG signals.
delete

Images