KR20220098991A - Method and apparatus for recognizing emtions based on speech signal - Google Patents

Abstract

An emotion recognition device for recognizing the emotions of a speaker based on voice signals according to one aspect of the present invention includes a memory storing a voice-based emotion recognition program, and a processor executing the program stored in the memory. The voice-based emotion recognition program receives the speaker's voice data and classifies the speaker's emotions by inputting the received voice data into an emotion classification model. The emotion classification model includes a local feature extraction unit that extracts local features of voice data through ConvLSTM, and a global feature extraction unit that extracts global features of voice data through GRU (gated recurrent unit), and classifies the speaker's emotions based on the extracted local and global features.

Description

Apparatus and method for emotion recognition based on voice signals

The present invention relates to a speech-based emotion recognition apparatus and method capable of recognizing a speaker's emotion from a speech signal through a machine learning model.

Research on a technology for recognizing a speaker's emotions through a human voice is in progress. In particular, Speech Emotion Recognition (SER) technology, which uses artificial intelligence technology or machine learning technology, is known as a new field of digital audio signal processing, and is a human-computer interaction (HCI) technology. It is expected to play an important role in many applications related to

Existing research has introduced a number of deep neural networks (DNNs) to model emotion recognition in voice data. For example, a DNN model for detecting important signals in original audio samples has been proposed, or a technique has been proposed to provide an input to the model using a specific representation of an audio recording.

In particular, researchers are extracting hidden signals through various types of convolution operations and recognizing lines, curves, points, shapes and colors. For example, mid-level end-to-end models including convolution neural networks (CNN), recurrent neural networks (RNN), long short-term memory (LSTM), and deep belief networks (DBN) are utilized. However, there is a problem that the accuracy level and recognition rate are low because the configuration of these various artificial neural network models is still poor. In the case of a model using CNN, the role of increasing the accuracy of emotion recognition is insufficient.

In addition, RNN and LSTM are used to learn long-term change factors in time and to recognize emotions, but there is a problem in that it increases the computation and training time of the entire model without significantly improving the accuracy. As such, there is a need to provide an efficient and important framework for recognizing spatial emotional signals and sequential signals.

Japanese Patent Publication No. 6732703 (Title of the invention: emotion interaction model learning device, emotion recognition device, emotion interaction model learning method, emotion recognition method, and program)

The present invention is to solve the problems of the prior art, and it is an object of the present invention to provide a voice-based emotion recognition apparatus and method capable of classifying a speaker's emotions from a voice by utilizing both spatial and temporal characteristics of a voice signal. .

However, the technical task to be achieved by the present embodiment is not limited to the above-described technical task, and other technical tasks may exist.

As a technical means for solving the above technical problem, an emotion recognition apparatus for recognizing a speaker's emotion based on a speech signal according to an aspect of the present invention includes: a memory in which a speech-based emotion recognition program is stored; and a processor executing the program stored in the memory, wherein the speech-based emotion recognition program receives the speaker's voice data and inputs the received voice data into an emotion classification model to classify the speaker's emotions. At this time, the emotion classification model includes a local feature extractor that extracts local features of voice data through ConvLSTM, and a global feature extractor that extracts global features of voice data through a gated recurren unit (GRU), and the local features and global features Classify the speaker's emotions based on their characteristics.

In addition, the structure emotion recognition method using a voice-based emotion recognition apparatus according to another aspect of the present invention includes the steps of receiving voice data of a speaker, and inputting the received voice data into an emotion classification model to classify the emotions of the speaker Including, wherein the emotion classification model includes a local feature extractor for extracting local features of voice data through ConvLSTM, and a global feature extractor for extracting global features of voice data through a gated recurring unit (GRU), the local features and classifying the speaker's emotions based on global characteristics.

According to the above-described problem solving means of the present application, by effectively extracting temporal and spatial characteristics included in voice data, it is possible to automatically classify the speaker's emotions.

In particular, as the ConvLSTM model is used in the process of extracting local features of voice data, it is possible to easily recognize a continuous sequence of a voice signal and to extract connected emotion information from the recognized sequence. In addition, since emotional information that is temporally separated from each other can be considered together through the GRU, the prediction performance of the SER system can be improved.

1 is a block diagram illustrating a configuration of a voice-based emotion recognition apparatus according to an embodiment of the present invention.
2 is a conceptual diagram illustrating a voice-based emotion recognition apparatus according to an embodiment of the present invention.
3 is a flowchart illustrating a voice-based emotion recognition method according to an embodiment of the present invention.
4 is a flowchart illustrating a process of constructing an emotion classification model used in a voice-based emotion recognition method according to an embodiment of the present invention.
5 is a diagram illustrating a detailed configuration of a ConvLSTM layer according to an embodiment of the present invention.
6 is a diagram illustrating a configuration of a GRU used in a global feature extraction unit according to an embodiment of the present invention.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present application pertains can easily implement them. However, the present application may be implemented in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is said to be "connected" with another part, it includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. do.

Throughout this specification, when a member is said to be located “on” another member, this includes not only a case in which a member is in contact with another member but also a case in which another member is present between the two members.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a configuration of a voice-based emotion recognition apparatus according to an embodiment of the present invention.

As shown, the voice-based emotion recognition apparatus 100 may include a communication module 110 , a memory 120 , a processor 130 , and a database 140 . In addition, the voice-based emotion recognition apparatus 100 may have a built-in microphone or the like, and it is also possible to directly generate voice data through this.

The communication module 110 receives voice data of a speaker from an external device, and may receive voice data input through a microphone connected to various smart terminals through the communication network 300 . In addition, the communication module 110 may receive update information, such as a voice-based emotion recognition program, from various external devices (servers or terminals) and transmit the received information to the processor 130 .

The communication module 110 may be a device including hardware and software necessary for transmitting and receiving signals such as control signals or data signals through wired/wireless connection with other network devices.

The memory 120 stores a voice-based emotion recognition program for classifying the speaker's emotions based on the speaker's voice. Various types of data generated during the execution of an operating system for driving the voice-based emotion recognition apparatus 100 or a voice-based emotion recognition program are stored in the memory 120 .

In this case, the voice-based emotion recognition program receives the speaker's voice data through the communication module 110, and inputs the received voice data to the emotion classification model to classify the speaker's emotions. In this case, the emotion classification model includes a local feature extractor that extracts local features of voice data through ConvLSTM, and a global feature extractor that extracts global features of voice data through a gated recurren unit (GRU), and additionally through a loss function. It may include a loss function unit for updating the emotion classification model. The specific contents of the voice-based emotion recognition program will be described later.

In this case, the memory 120 collectively refers to a non-volatile storage device that continuously maintains stored information even when power is not supplied, and a volatile storage device that requires power to maintain the stored information.

In addition, the memory 120 may perform a function of temporarily or permanently storing data processed by the processor 130 . Here, the memory 120 may include magnetic storage media or flash storage media in addition to the volatile storage device requiring power to maintain stored information, but the scope of the present invention is limited thereto. it's not going to be

The processor 130 executes the program stored in the memory 120, but according to the execution of the speech-based emotion recognition program, classifies the speaker's emotion based on the voice through the construction process of the emotion classification model and the constructed emotion classification model. do the work

The processor 130 may include any type of device capable of processing data. For example, it may refer to a data processing device embedded in hardware having a physically structured circuit to perform a function expressed as a code or an instruction included in a program. As an example of the data processing device embedded in the hardware as described above, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated (ASIC) circuit) and a processing device such as a field programmable gate array (FPGA), but the scope of the present invention is not limited thereto.

The database 140 stores or provides data necessary for the voice-based emotion recognition apparatus 100 under the control of the processor 130 . The database 140 may be included as a component separate from the memory 120 , or may be built in a part of the memory 120 .

On the other hand, the voice-based emotion recognition apparatus 100 may directly generate voice data by recording the speaker's voice signal through a microphone built into or connected to the device 100, and may perform emotion recognition. have.

2 is a conceptual diagram for explaining a voice-based emotion recognition apparatus according to an embodiment of the present invention, FIG. 3 is a flowchart for explaining a voice-based emotion recognition method according to an embodiment of the present invention, and FIG. 4 is this It is a flowchart for explaining a process of constructing an emotion classification model used in a voice-based emotion recognition method according to an embodiment of the present invention.

A speech-based emotion recognition method performed by a speech-based emotion recognition program stored in the memory 140 will be described.

First, the voice-based emotion recognition program installed in the voice-based emotion recognition apparatus 100 receives recorded voice data from a microphone through a microphone or the like, or receives voice data through the communication module 110 ( S310 ). The voice data is digital data, and may be divided into voice segments divided by a predetermined time unit and input to the emotion classification model. In this way, speech segments in a continuous relationship with each other have a strong temporal correlation, and emotion recognition is performed using this characteristic.

Next, the speech-based emotion recognition program classifies the speaker's emotion by inputting the received speech data into the emotion classification model (S320). For example, as voice data is input to the emotion classification model, the emotional state of the speaker may be classified as 'angry', 'sad', 'happiness', 'normal', etc. as the output.

At this time, the emotion classification model includes a local feature extractor that extracts local features of voice data through ConvLSTM, and a global feature extractor that extracts global features of voice data through a gated recurren unit (GRU). The construction process will be described in more detail with reference to FIGS. 2 and 4 to 6 .

Speech data processed in the present invention is data having temporally continuous features, and features are typically extracted using a model such as LSTM, but the model has a problem of increasing calculation and learning time.

Accordingly, the emotion classification model of the present invention merges a CNN model useful for extracting and learning spatial features and an LSTM model useful for extracting and learning temporal features, a plurality of local function learning blocks (LFLB, local It includes a local feature extraction unit consisting of features learning blocks). In the present invention, by utilizing spatial features, not only temporal intervals between voice segments, but also features separated at intervals in frequency bands are extracted and used to classify emotions. Through this, it is possible to properly utilize the characteristics of emotions expressed over a long time as well as a short time, thereby contributing to the improvement of the performance of emotion recognition.

The figure shows a structure in which four local function learning blocks (LFLBs) are sequentially connected, but this is an exemplary configuration and the present invention is not limited thereto.

In this case, each local function learning block has a structure in which a ConvLSTM layer, a BN layer, and a pooling layer are sequentially connected as shown in FIG. 2 . And, through a structure in which each of a plurality of local function learning blocks are sequentially connected, input-state and state-state correlations between voice segments may be found. That is, in the process of processing sequentially input speech segments, the correlation of each speech segment is captured, and emotions are recognized through this.

The ConvLSTM layer was used for hidden step-by-step prediction to keep sequential information in an internal state, to optimize the sequence and find spatiotemporal correlations between speech segments.

Referring to FIG. 4 , a process of constructing an emotion classification model will be described.

First, the voice-based emotion recognition program installed in the voice-based emotion recognition apparatus 100 receives recorded voice data from a microphone through a microphone or the like, or receives voice data through the communication module 110 ( S410 ).

Next, voice data is input to the local feature extraction unit based on ConvLSTM (S420).

5 is a diagram illustrating a detailed configuration of a ConvLSTM layer according to an embodiment of the present invention.

The illustrated ConvLSTM layer calculates a weight using the following equation.

[Equation 1]

σ denotes a sigmoid function, * denotes a convolution operation, ⓒ denotes an element wise operation, tanh denotes a hyperbolic tangent function, w denotes a weight for each variable, and b denotes an element wise operation. A bias value, t is the number of operation repetitions, x _t is input data, c _t is a cell state, and h _t is a hidden state. As described above, in the ConvLSTM layer, a part of the operation in which the multiplication between matrices is performed is replaced by the convolution operation.

And, in FIG. 5, it is an input gate (input gate), f _t is a forget gate, o _t is an output gate (output gate), g _t is an input modulation gate ₍ input modulation gate), respectively, This is the same as the configuration of a general LSTM.

Meanwhile, in ConvLSTM, data processed by each input gate, input data (x _t ), cell state (c _t ), and hidden state (h _t ) are all expressed as three-dimensional tensors. In this case, in the input tensor, the first dimension represents temporal information, the second dimension represents size information, and the third dimension represents spatial information. As such, ConvLSTM has technical characteristics in extracting spatiotemporal features during state-to-state transition.

Referring again to FIG. 4, voice data is input to the GRU-based global feature extraction unit (S430).

First, as shown in FIG. 2 , the global feature extraction unit includes a Global Feature Learning Block (GFLB). GFLB includes a gated recurrent unit (GRU) to learn global feature information from speech data and recognize long-term context dependencies.

6 is a diagram illustrating a configuration of a GRU used in a global feature extraction unit according to an embodiment of the present invention.

The GRU is a kind of LSTM framework to which a gate mechanism is applied, and includes an update gate and a reset gate as shown in (a). The update gate performs the same operations as the forget gate and input gate in the LSTM, and the reset gate performs the same operation as the reset gate in the LSTM.

를 출력한다.The reset gate properly resets past information, and uses a sigmoid function, as shown in the following equation.

to output

[Equation 2]

를 출력한다.The update gate determines the update rate of past and present information, and uses a sigmoid function, as shown in the following equation

to output

[Equation 3]

)을 산출하는데, 이때 리셋 게이트의 결과를 이용한다.In addition, the update gate is the current information candidate group (

), which uses the result of the reset gate.

[Equation 4]

)의 결과는 수학식 3과 수학식 4에 의해서 결정되는 업데이트 게이트의 출력을 결합하여 수학식 5에 의해 결정된다.Finally, the final hidden state (

) is determined by Equation 5 by combining the outputs of the update gate determined by Equations 3 and 4.

[Equation 5]

In this case, σ denotes a sigmoid function, * denotes element wise multiplication, tanh denotes a hyperbolic tangent function, W and U denote a weight for each variable, t denotes the number of repetitions of operations, x _t is the input data, and h _t is the hidden state.

With this configuration, the short-term dependency included in the voice segment is activated by the reset gate, and the previous state of the voice segment is controlled by the update gate, which also serves to control long-term context information.

Meanwhile, in the present invention, as shown in (b), it is possible to adjust the weight for the global feature by arranging a plurality of unit layers in which two GRUs are stacked.

In addition, a fully connected layer is coupled to the output terminal of the global feature extraction unit, and through this, the speaker's emotions are classified, and then updated based on the result of the combined fusion loss function.

Referring back to FIG. 4 , an operation of updating the emotion classification model using the loss function is performed ( S440 ).

In the present invention, the loss of the emotion classification model is calculated using a center loss function and a soft max loss function. The predictive performance of the model using the soft max loss function is somewhat lower due to the large distance within the class.

In the present invention, the minimum distance within a class is calculated using the central loss function, and the maximum distance between classes is calculated through the soft max loss function, and the specific equation is as follows.

[Equation 6]: Soft max loss function

[Equation 7]: Center loss function

n is the number of classes, m is the minimum batch size, and c _yi is the center of class y _i .

At this time, in order to calculate the minimum distance required to prevent misclassification in a real-time scenario, a fusion loss function that reflects both the soft max loss function and the central loss function using the λ symbol for the center loss is used as in Equation 8 did.

[Equation 8]

The emotion classification model calculates the loss for the output of the local feature extractor and the global feature extractor through the fusion loss function based on the central loss function and the soft max loss function, and performs weight updates in the direction of minimizing the loss.

In order to evaluate the effect of the emotion classification model of the present invention constructed as described above, IEMOCAP and RAVDESS, which are open databases widely used in academic experiments in the same field, were used, and these include two standard corpus data including emotional language corpus, respectively. . For the IEMOCAP and RAVDESS corpus according to the present invention, a recognition rate of 75% and a recognition rate of 80% were secured, respectively, which correspond to the highest figures as of the end of 2020.

An embodiment of the present invention may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

Although the methods and systems of the present invention have been described with reference to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general purpose hardware architecture.

The above description of the present application is for illustration, and those of ordinary skill in the art to which the present application pertains will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may also be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present application.

100: voice-based emotion recognition device
110: communication module
120: memory
130: processor
140: database

Claims (8)

An emotion recognition apparatus for recognizing a speaker's emotion based on a voice signal, comprising:
a memory in which a speech-based emotion recognition program is stored; and
A processor for executing the program stored in the memory;
The speech-based emotion recognition program receives the speaker's voice data, and classifies the speaker's emotions by inputting the received voice data into an emotion classification model,
The emotion classification model includes a local feature extractor that extracts local features of voice data through ConvLSTM, and a global feature extractor that extracts global features of voice data through a gated recurren unit (GRU), wherein the local features and global features A voice-based emotion recognition device that classifies the speaker's emotions based on the According to claim 1,
The local feature extraction unit has a structure in which a plurality of local function learning blocks are sequentially connected, and each local function learning block has a structure in which a ConvLSTM layer, a BN layer and a pooling layer are sequentially connected. According to claim 1,
The global feature extraction unit will include a plurality of unit layers each including two stacked gated recurrent units (GRUs), the speech-based emotion recognition apparatus. The method of claim 1,
The emotion classification model calculates the loss for the outputs of the local feature extraction unit and the global feature extraction unit through a fusion loss function based on the central loss function and the soft max loss function, and updates the weights in a direction to minimize the loss. A voice-based emotion recognition device that performs. In the emotion recognition method using a voice-based emotion recognition device,
receiving voice data of the speaker; and
Including the step of inputting the received voice data into an emotion classification model to classify the speaker's emotions,
The emotion classification model includes a local feature extractor that extracts local features of voice data through ConvLSTM, and a global feature extractor that extracts global features of voice data through a gated recurren unit (GRU), wherein the local features and global features A voice-based emotion recognition method that classifies the speaker's emotions based on 6. The method of claim 5,
The local feature extraction unit has a structure in which a plurality of local function learning blocks are sequentially connected, and each local function learning block has a structure in which a ConvLSTM layer, a BN layer, and a pooling layer are sequentially connected. 6. The method of claim 5,
The global feature extraction unit will include a plurality of unit layers each including two stacked gated recurrent units (GRUs), the speech-based emotion recognition method. 6. The method of claim 5,
Further comprising the step of updating the emotion classification model,
Calculating the loss for the outputs of the local feature extracting unit and the global feature extracting unit through a fusion loss function based on the central loss function and the soft max loss function, and performing weight updates in a direction to minimize the loss, Speech-based emotion recognition method.

Images