KR102334595B1 - Emotion recongnition method and device - Google Patents

Abstract

Embodiments of the present invention provide techniques for improving emotion recognition performance by reducing influence of an individual character in an emotion recognition system based on brain waves. Disclosed are an emotion recognition method and device. According to an embodiment of the present invention, emotion recognition method comprises: a step of acquiring a multi-channel EEG signal; a step of extracting correlation features between channels of the EEG signal; and a step of classifying emotion corresponding to the EEG signal based on the correlation feature.

Description

EMOTION RECONGNITION METHOD AND DEVICE

The present disclosure relates to an emotion detection method and apparatus.

Biosignals, such as electroencephalogram (EEG), electromyogram (EMG), and electrocardiogram (ECG), can be highly correlated information for understanding a human's psychological state. Through this, an emotional state monitoring technology using a wearable device such as a smart watch capable of detecting a biosignal is being developed.

EEG-based emotion recognition technology is greatly affected by individual differences according to personality and gender, and even the same person can be greatly affected by various conditions such as health, mood, and surrounding environment.

As a related prior art, there is Korean Patent Registration No. 10-2175997 (Title of the Invention: EEG-based User Intention Recognition Method and Apparatus).

The following embodiments provide techniques for improving emotion recognition performance by reducing the influence of individual characteristics in an emotion recognition system based on brain waves.

However, the technical problems are not limited to the above-described technical problems, and other technical problems may exist.

The emotion recognition method according to an embodiment includes: obtaining a multi-channel EEG signal; extracting a correlation feature between channels of the EEG signal; and corresponding to the EEG signal based on the correlation characteristic. It includes the step of classifying the emotion.

The extracting may include calculating an average value of the EEG signal at regular time intervals, and calculating the correlation feature based on the average value.

Each component of the correlation characteristic may be a Pearson correlation coefficient of the average value between two channels.

The classifying may include inputting the correlation feature into a neural network.

The neural network may be a long short-term memory (LSTM) stacked in two layers.

The classifying may include classifying the emotion by filtering individual features from the correlation features.

A method for learning a neural network for emotion recognition according to an embodiment includes: obtaining a multi-channel EEG signal; extracting a correlation feature between channels of the EEG signal; to update the parameters of the neural network.

The extracting may include calculating an average value of the EEG signal at regular time intervals, and calculating the correlation feature based on the average value.

Each component of the correlation characteristic may be a Pearson correlation coefficient of the average value between two channels.

The neural network may be a long short-term memory (LSTM) stacked in two layers.

The emotion recognition apparatus according to an embodiment includes a memory including instructions and a processor for executing the instructions, and when the instructions are executed by the processor, the processor obtains a multi-channel EEG signal and , extracts a correlation feature between channels of the EEG signal, and classifies an emotion corresponding to the EEG signal based on the correlation feature.

The processor may calculate an average value of the EEG signal at regular time intervals, and calculate the correlation feature based on the average value.

Each component of the correlation characteristic may be a Pearson correlation coefficient of the average value between two channels.

The processor may input the correlation feature to the neural network.

The neural network may be a long short-term memory (LSTM) stacked in two layers.

The processor may classify the emotion by filtering individual characteristics from the correlation characteristics.

1 illustrates an emotion recognition system according to an embodiment.
FIG. 2 is a diagram for explaining an operation of the emotion recognition apparatus shown in FIG. 1 .
3 shows the visualized correlation features.
4 is a flowchart illustrating an emotion recognition method according to an exemplary embodiment.
5 is a flowchart illustrating a method of learning a neural network for emotion recognition according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for purposes of illustration only, and may be changed and implemented in various forms. Accordingly, the actual implemented form is not limited to the specific embodiments disclosed, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Although terms such as first or second may be used to describe various components, these terms should be interpreted only for the purpose of distinguishing one component from another. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected to” another component, it may be directly connected or connected to the other component, but it should be understood that another component may exist in between.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

1 illustrates an emotion recognition system according to an embodiment.

The emotion recognition system 10 may recognize an emotion based on a person's electroencephalogram (EEG). The emotion recognition system 10 may recognize an emotion with a high recognition rate in consideration of the correlation between the EEG channels in order to reduce the influence of the EEG difference between individuals.

The emotion recognition system 10 may include an emotion recognition apparatus 100 and an EEG meter 200 . The emotion recognition apparatus 100 may recognize the emotion of the measurement target based on the EEG measured by the EEG meter 200 .

The emotion recognition apparatus 100 may include a processor 130 and a memory 150 . The memory 150 may store instructions (or programs) executable by the processor 130 . For example, the instructions may include instructions for executing an operation of a processor and/or an operation of each component of the processor.

The memory 150 may be implemented as a volatile memory device or a nonvolatile memory device. The volatile memory device may be implemented as dynamic random access memory (DRAM), static random access memory (SRAM), thyristor RAM (T-RAM), zero capacitor RAM (Z-RAM), or twin transistor RAM (TTRAM), Nonvolatile memory devices include EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, MRAM (Magnetic RAM), CBRAM (Conductive Bridging RAM), FeRAM (Ferroelectric RAM), PRAM (Phase change RAM), and RRAM. (Resistive RAM) can be implemented.

The processor 130 may process data stored in the memory 150 . The processor 130 may execute computer readable code (eg, software) stored in the memory 150 and instructions induced by the processor 130 .

The processor 130 may be a hardware-implemented data processing device having a circuit having a physical structure for executing desired operations. For example, desired operations may include code or instructions included in a program.

The processor 130 includes a central processing unit, a graphics processing unit, a neural processing unit, a multi-core processor, a multiprocessor, and an ASIC. (Application-Specific Integrated Circuit), may include a Field Programmable Gate Array (FPGA).

It should be understood that operations of the emotion recognition apparatus 100 are performed by the processor 130 .

The EEG measuring device 200 may detect a human EEG signal. The EEG measuring device 200 may measure multi-channel EEG signals. For example, the EEG measuring device 200 may measure an EEG signal of 32 channels at a sampling rate of 128 Hz. Channels measured by the EEG meter 200 may be Fp1, AF3, F3, O2, or the like.

The processor 130 may extract a correlation feature between channels of the EEG signal based on the multi-channel EEG signal. The processor 130 may extract a correlation feature by calculating a Pearson correlation coefficient between channels.

The processor 130 may classify the emotion corresponding to the EEG signal based on the correlation feature. The processor 130 may obtain the emotion classification result by inputting the correlation feature into the neural network. A neural network may be a long short-term memory (LSTM) stacked in two layers.

FIG. 2 is a diagram for explaining an operation of the emotion recognition apparatus shown in FIG. 1 .

)라고 지칭한다(c는 C 이하 자연수이고, n은 N이하 자연수이고, k는 K이하 자연수이다).The processor 130 may acquire an EEG signal (Raw EEG; E) having C channels. The EEG signal E may be divided at regular time intervals. The EEG signal E may be divided into N first segments, and each of the N first segments may be divided into K second segments. The finally divided K×N segments may each include L data (C, N, K, and L are natural numbers). Hereinafter, the k-th second segment included in the n-th first segment in the c-th channel is referred to as the (k,n)-th segment (

) (c is a natural number less than or equal to C, n is a natural number less than or equal to N, and k is a natural number less than or equal to K).

)에 포함된 L개의 데이터의 평균 값을 계산하여 싱글 채널 특징(single-channel features;

)를 획득할 수 있다. 프로세서(130)는 싱글 채널 특징(

)을 계산함으로써 뇌파 신호의 노이즈를 제거하고 전체 데이터 크기를 줄일 수 있다. 싱글 채널 특징(

)은 수학식 1과 같이 표현될 수 있다.The processor 130 is the (k, n) th segment (

) by calculating the average value of L data included in single-channel features;

) can be obtained. The processor 130 has a single channel feature (

), the noise of the EEG signal can be removed and the overall data size can be reduced. single channel feature (

) can be expressed as in Equation 1.

)의 채널 사이의 피어슨 상관 계수를 계산하여 채널 사이의 상관 관계 특징(Channel-wise feature;

)를 계산할 수 있다. 상관 관계 특징(

)은 수학식 2에 기초하여 계산될 수 있다.The processor 130 has a single channel feature (

) by calculating the Pearson correlation coefficient between channels of a correlation feature between channels (Channel-wise feature;

) can be calculated. Correlation features (

) can be calculated based on Equation (2).

는 i번째 제1 세그먼트에 대한 상관 관계 특징의 (x,y) 성분을 의미하고,

및

는 각각 i번째 제1 세그먼트에 대해 계산된 싱글 채널 특징의 x번째 채널 및 y번째 채널 데이터를 의미한다. 각각의 상관 관계 특징(

)은 C×C 크기의 행렬로 표현될 수 있다.in Equation 2

denotes the (x,y) component of the correlation feature for the i-th first segment,

and

denotes x-th channel and y-th channel data of the single channel feature calculated for the i-th first segment, respectively. Each correlation feature (

) can be expressed as a matrix of size C×C.

)을 뉴럴 네트워크(300)에 입력하여 뇌파 신호에 대응하는 감정을 분류할 수 있다. 예를 들어, 프로세서(130)는 뇌파 신호에 대응하는 감정을 긍정(positive), 중립(neutral) 및 부정(negative) 중 하나로 분류하거나, HVHA(High Valence-High Arousa), HVLA(High Valence-Low Arousa), LVHA(Low Valence-High Arousa) 및 LVLA(Low Valence-Low Arousa) 중 하나로 분류할 수 있다.The processor 130 determines the correlation feature (

) can be input to the neural network 300 to classify emotions corresponding to EEG signals. For example, the processor 130 classifies the emotion corresponding to the EEG signal into one of positive, neutral, and negative, or High Valence-High Arousa (HVHA), High Valence-Low (HVLA). Arousa), LVHA (Low Valence-High Arousa), and LVLA (Low Valence-Low Arousa) can be classified into one.

)을 이용할 수 있다. 프로세서(130)는 상관 관계 특징(

) 순차적으로 두 개의 층으로 쌓은 LSTM(Long Short-Term Memory; 300)에 입력할 수 있다. 이때, 프로세서(130)는 상관 관계 특징(

)을

크기로 플래튼(flatten)하여 LSTM(300)에 입력할 수 있다. 플래튼(flatten) 동작은 상관 관계 특징(

)에 대각선으로 중복되는 상부 삼각형 성분을 제거하여 재배열하는 동작일 수 있다.The processor 130 has a plurality of successive correlation features (

) can be used. The processor 130 determines the correlation feature (

) can be input to a Long Short-Term Memory (LSTM) 300 sequentially stacked in two layers. At this time, the processor 130 is a correlation feature (

)second

It can be input to the LSTM 300 by flattening it to a size. The flatten behavior is characterized by the correlation (

) may be an operation of removing and rearranging the upper triangular component that overlaps diagonally.

)이 입력될 수 있고, 제1 LSTM(310)의 은닉층이 제2 LSTM(330)에 입력될 수 있다. 이때, 은닉층 벡터의 차원은 예를 들어, 256일 수 있다. MLP(350)는 제2 LSTM(330)의 출력에 기초하여 감정 분류 결과를 출력할 수 있다. LSTM(300)의 파라미터는 학습을 통해 결정될 수 있으며, 예를 들어, SEED(SJTU Emotion EEG Dataset) 또는 DEAP(Database for Emotion Analysis using Physiological Signals)을 이용하여 학습될 수 있다.The two-layered Long Short-Term Memory (LSTM) 300 may include a first LSTM 310 , a second LSTM 330 , and a multi-layer perceptron (MLP) 350 . The first LSTM 310 has a platen correlation feature (

) may be input, and the hidden layer of the first LSTM 310 may be input to the second LSTM 330 . In this case, the dimension of the hidden layer vector may be, for example, 256. The MLP 350 may output the emotion classification result based on the output of the second LSTM 330 . The parameters of the LSTM 300 may be determined through learning, for example, may be learned using SJTU Emotion EEG Dataset (SEED) or Database for Emotion Analysis using Physiological Signals (DEAP).

2 shows the visualized correlation features.

Looking at the correlation features extracted based on SEED (SJTU Emotion EEG Dataset) and DEAP (Database for Emotion Analysis using Physiological Signals), even if the same person feels various emotions in the EEG emotional data set, the correlation features appear similar and many people Even if you feel these emotions, you can confirm that other characteristics appear.

) 중 개인의 특징을 필터링하여 감정 특징을 추출할 수 있다. 상관 관계 특징(

) 중 개인의 특징을 필터링 함으로써 프로세서(130)는 높은 인식률로 감정을 분류할 수 있다.The processor 130 uses the LSTM 300 to generate a correlation feature (

), the emotional characteristics can be extracted by filtering the characteristics of the intermediary. Correlation features (

) by filtering the intermediary's characteristics, the processor 130 can classify the emotion with a high recognition rate.

4 is a flowchart illustrating an emotion recognition method according to an exemplary embodiment.

The emotion recognition method 400 may include an operation 410 of acquiring a multi-channel EEG signal. The EEG signal has 32 channels and can be measured at a sampling rate of 128 Hz.

The emotion recognition method 400 may include an operation 420 of extracting correlation features for each channel of the EEG signal. The correlation feature may be extracted based on an average value of EEG signals calculated at regular time intervals. The EEG signal may be divided at regular time intervals, an average value of data in the segmented segment may be calculated to obtain a single channel characteristic, and a correlation characteristic may be calculated based on the single channel characteristic. For example, the correlation feature may be calculated based on a Pierce correlation coefficient of data between two channels.

The emotion recognition method 400 may include an operation 430 of detecting an emotion corresponding to the EEG signal based on the correlation feature. Emotions may be classified by filtering individual characteristics from correlation characteristics. The correlation feature may be input to the neural network, and an emotion classification result corresponding to the EEG signal may be output from the neural network. A neural network may be an LSTM stacked in two layers.

Since the operations of the emotion recognition apparatus 100 described with reference to FIGS. 1 to 3 are applied to the operations shown in FIG. 4 as they are, a more detailed description will be omitted.

5 is a flowchart illustrating a method of learning a neural network for emotion recognition according to an embodiment.

The learning method 500 may include an operation 510 of acquiring a multi-channel EEG signal. The multi-channel EEG signal may be obtained from an EEG signal and a data set in which an emotion classification corresponding to the EEG signal is labeled. For example, the EEG signal may be obtained from SJTU Emotion EEG Dataset (SEED) or Database for Emotion Analysis using Physiological Signals (DEAP).

The learning method 500 may include an operation 520 of extracting correlation features for each channel of the EEG signal. Since operation 520 may be the same as operation 420 of the emotion recognition method 400 shown in FIG. 4 , a redundant description will be omitted.

The learning method 500 may include an operation 530 of updating a parameter of the neural network based on the correlation feature. The neural network may be an LSTM stacked in two layers, and parameters of the neural network may be updated based on the output of the neural network and the ground truth labeled in the EEG signal.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using a general purpose computer or special purpose computer. The processing device may execute an operating system (OS) and a software application running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination, and the program instructions recorded on the medium are specially designed and configured for the embodiment, or are known and available to those skilled in the art of computer software. may be Examples of the computer readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

The hardware devices described above may be configured to operate as one or a plurality of software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited drawings, a person skilled in the art may apply various technical modifications and variations based thereon. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (17)

In the emotion recognition method performed through one or more processors,
acquiring multi-channel EEG signals;
extracting correlation features between channels of the EEG signal; and
Classifying the emotion corresponding to the EEG signal by inputting the correlation feature into a long short-term memory (LSTM) stacked in two layers
including,
The extraction step is
dividing the EEG signal into segments of a predetermined time interval;
obtaining a single-channel feature by calculating an average value of data included in the segmented segment; and
Calculating a Pearson correlation coefficient of the average value of the EEG signal between the two channels, each component of the correlation feature, based on the covariance and standard deviation of the two channel data of the single channel feature
Including, emotion recognition method.
delete delete delete delete According to claim 1,
The classification step is
Classifying the emotion by filtering individual features from the correlation features
Including, emotion recognition method.
In the learning method of a neural network for emotion recognition performed through a processor,
acquiring multi-channel EEG signals;
extracting correlation features between channels of the EEG signal; and
updating a parameter of a long short-term memory (LSTM) stacked in two layers based on the correlation characteristic
including,
The extraction step is
dividing the EEG signal into segments of a predetermined time interval;
obtaining a single-channel feature by calculating an average value of data included in the segmented segment; and
Calculating a Pearson correlation coefficient of the average value of the EEG signal between the two channels, each component of the correlation feature, based on the covariance and standard deviation of the two channel data of the single channel feature
A learning method comprising:
delete delete delete A computer program stored in a computer-readable recording medium in combination with hardware to execute the method of any one of claims 1, 6 and 7.
a memory containing instructions; and
a processor for executing the instructions
including,
When the instructions are executed by the processor, the processor
Acquire multi-channel EEG signals,
segmenting the EEG signal into segments at regular time intervals;
Obtain a single-channel feature by calculating the average value of the data included in the segmented segment,
Extracting a correlation feature that is a Pearson correlation coefficient of the average value of the EEG signal between two channels based on the covariance and standard deviation of the two channel data of the single channel feature,
An emotion recognition apparatus for classifying an emotion corresponding to the EEG signal by inputting the correlation feature into a long short-term memory (LSTM) stacked in two layers.
delete delete delete delete 13. The method of claim 12,
The processor is
An emotion recognition device for classifying the emotion by filtering individual features from the correlation feature.

Images