KR20230007781A - Method and apparatus for recognizing speaker’s emotions based on speech signal - Google Patents

Abstract

In accordance with one aspect of the present invention, an emotion recognition apparatus recognizing emotions of a speaker based on a voice signal, includes: a memory storing a voice-based emotion recognition program; and a processor executing the program stored in the memory. The voice-based emotion recognition program receives a voice signal of a speaker, and inputs the received voice signal into an emotion classification model to classify emotions of the speaker, wherein the emotion classification model includes: a first channel module extracting spectral features from the spectrum of the voice signal; a second channel module extracting spatial features from the spectrogram of the voice signal; a fusion layer module generating a joint spatial-spectral feature vector from the spectral features outputted from the first channel module and the spatial features outputted from the second channel module; an optimal feature selection module selecting optimal features from the fusion layer module; and a classifier module performing emotion classification with respect to the selected optimal features. Therefore, the present invention is capable of removing redundancy and selecting optimal features.

Description

Apparatus and method for recognizing speaker's emotion based on voice signal

The present invention relates to a voice-based emotion recognition apparatus and method capable of recognizing a speaker's emotion from a spectrum and a spectrogram of a voice signal through a machine learning model.

Research on a technology for recognizing a speaker's emotion through a human voice is being conducted. In particular, smart speech emotion recognition (SER, Speech Emotion Recognition) technology using artificial intelligence technology or machine learning technology is known as a new field of digital audio signal processing, and human-computer interaction (HCI, Human Computer Interface) technology It is expected to play an important role in many applications related to

Existing studies have introduced various numbers of deep neural networks (DNNs) to model emotion recognition in voice signals. For example, DNN models have been proposed that detect important signals in raw audio samples, or techniques that provide input to models using specific representations of audio recordings have been proposed.

In particular, many previous studies have attempted emotion recognition using various input types of voice signals such as voice spectrogram, original voice signal, and log-Mel spectrogram for an efficient voice-based emotion recognition system. However, these methods are models that use only partial information for emotion recognition, and provide incomplete information to the system. In order to create an accurate voice-based emotion recognition system, it is important to use various voice-related information in an integrated manner, and the present invention intends to provide an emotion recognition system that utilizes various voice-related information.

Republic of Korea Patent Registration No. 10-1564176 (Title of Invention: Emotion Recognition System and Control Method)

The present invention is to solve the above-mentioned problems of the prior art, and provides a voice-based emotion recognition apparatus and method capable of classifying a speaker's emotion from a voice by utilizing feature information extracted from a voice spectrum and a spectrogram of a voice signal. has a purpose to

However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

As a technical means for solving the above technical problem, an emotion recognition device for recognizing a speaker's emotion based on a voice signal according to an aspect of the present invention includes a memory in which a voice-based emotion recognition program is stored; and a processor executing a program stored in the memory. A voice-based emotion recognition program receives a voice signal of a speaker, inputs the received voice signal to an emotion classification model to classify the speaker's emotion, and the emotion classification model extracts a spectral feature from the spectrum of the voice signal. A first channel module, a second channel module for extracting spatial features from the spectrogram of the voice signal, and a joint spatial spectrum feature vector from the spectral features output from the first channel module and the spatial features output from the second channel module. It includes a convergence layer module to generate, an optimal feature selection module to select an optimal feature from the output of the convergence layer module, and a classifier module to perform emotion classification on the selected optimal feature.

In addition, an emotion recognition method using a voice-based emotion recognition device according to another aspect of the present invention includes the steps of receiving a voice signal of a speaker and inputting the received voice signal to an emotion classification model to determine the speaker's emotion It includes the step of classifying. At this time, the emotion classification model extracts a spectrum feature from the spectrum of the voice signal; extracting spatial features from the spectrogram of the speech signal; generating a joint spatial spectral feature vector from the spectral features and the spatial features; selecting optimal features from joint spatial spectral feature vectors; and performing emotion classification on the selected optimal feature.

According to the above-mentioned problem solving means of the present application, it is possible to automatically classify a speaker's emotion by effectively extracting spectral features and spatial features included in a voice signal.

In the conventional case, since emotion is classified only with a single feature extracted from a spectrum or spectrogram, it suffers from loss of language information. Since the present invention uses both spectrum analysis and spatial feature analysis, emotion recognition can be performed by maximally utilizing information of a voice signal.

In addition, since a joint spatial spectral feature vector is generated based on the spectral and spatial features extracted from the voice signal and an algorithm for selecting an optimal feature is performed therefrom, it is possible to remove redundancy and select an optimal feature.

1 is a block diagram showing the configuration of a voice-based emotion recognition apparatus according to an embodiment of the present invention.
2 is a block diagram illustrating the configuration of an emotion classification model included in a voice-based emotion recognition program according to an embodiment of the present invention.
3 is a flowchart illustrating a specific configuration of an emotion classification model according to an embodiment of the present invention.
4 is a flowchart illustrating an emotion recognition method according to an embodiment of the present invention.
5 illustrates performance evaluation results of an emotion classification model according to an embodiment of the present invention.

Hereinafter, embodiments of the present application will be described in detail so that those skilled in the art can easily practice with reference to the accompanying drawings. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. And in order to clearly describe the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is said to be "connected" to another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element in between. do.

Throughout the present specification, when a member is said to be located “on” another member, this includes not only a case where a member is in contact with another member, but also a case where another member exists between the two members.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram showing the configuration of a voice-based emotion recognition device according to an embodiment of the present invention, and FIG. 2 shows the configuration of an emotion classification model included in a voice-based emotion recognition program according to an embodiment of the present invention. It is a block diagram shown.

As shown, the voice-based emotion recognition device 100 may include a communication module 110, a memory 120, a processor 130, and a database 140. In addition, the voice-based emotion recognition apparatus 100 may have a built-in microphone, and it is also possible to directly generate a voice signal through this.

The voice-based emotion recognition apparatus 100 may operate as a server that receives voice signals from each user terminal and provides emotion classification results therefrom. In this case, the voice-based emotion recognition device 100 may operate in a cloud computing service model such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS). In addition, the voice-based emotion recognition device 100 may be built in the form of a private cloud, a public cloud, or a hybrid cloud.

The communication module 110 receives a voice signal of a talker from an external device, and may receive a voice signal input through a microphone connected to various smart terminals through the communication network 300 . In addition, the communication module 110 may receive update information such as a voice-based emotion recognition program from various external devices (servers or terminals) and transmit the received update information to the processor 130 .

The communication module 110 may be a device including hardware and software necessary for transmitting and receiving signals such as control signals or data signals with other network devices through wired or wireless connections.

The memory 120 stores a voice-based emotion recognition program for classifying the speaker's emotion based on the speaker's voice. The memory 120 stores various types of data generated during the execution of an operating system for driving the voice-based emotion recognition device 100 or a voice-based emotion recognition program.

At this time, the voice-based emotion recognition program receives the speaker's voice signal through the communication module 110 and inputs the received voice signal to the emotion classification model 200 to classify the speaker's emotion. At this time, the emotion classification model 200 includes a first channel module 220 that extracts spectral features from the voice signal, a second channel module 230 that extracts spatial features from the spectrogram of the voice signal, and a first channel module 220 From the spectral features output from ) and the spatial features output from the second channel module 230, the convergence layer module 240 that generates a joint spatial spectrum feature vector, and the optimal feature is selected from the output of the convergence layer module 240. and a classifier module 260 that performs emotion classification on the selected optimal feature. Details of the voice-based emotion recognition program will be described later.

At this time, the memory 120 collectively refers to a non-volatile storage device that continuously retains stored information even when power is not supplied and a volatile storage device that requires power to maintain stored information.

Also, the memory 120 may temporarily or permanently store data processed by the processor 130 . Here, the memory 120 may include magnetic storage media or flash storage media in addition to a volatile storage device that requires power to maintain stored information, but the scope of the present invention is limited thereto it is not going to be

The processor 130 executes the program stored in the memory 120, and classifies the speaker's emotion based on the voice through the process of building the emotion classification model and the built emotion classification model according to the execution of the voice-based emotion recognition program. Do the work.

The processor 130 may include any type of device capable of processing data. For example, it may refer to a data processing device embedded in hardware having a physically structured circuit to perform a function expressed as a code or command included in a program. As an example of such a data processing device built into hardware, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated (ASIC) circuit), field programmable gate array (FPGA), etc., but the scope of the present invention is not limited thereto.

The database 140 stores or provides data necessary for the voice-based emotion recognition device 100 under the control of the processor 130 . The database 140 may be included as a component separate from the memory 120 or may be built in a partial area of the memory 120 .

On the other hand, the voice-based emotion recognition device 100 may perform an operation of directly generating a voice signal by recording a speaker's voice signal through a microphone built into or connected to the device 100, and for this, emotion recognition can be performed.

3 is a flowchart illustrating a specific configuration of an emotion classification model according to an embodiment of the present invention.

Referring to FIGS. 2 and 3 together, when a voice signal is received, preprocessing is performed to generate a spectrum and a spectrogram of the voice signal, respectively, through the voice signal preprocessing module 210 .

First, a voice signal is used as an input and normalization is performed through a Root Mean Square (RMS) function defined in Equation 1 below.

[Equation 1]

The overall result of RMS is denoted by "R", the scaling factor of the speech signal is denoted by "f", and the amplitude change of the speech signal is directly performed by "s".

The speech segments normalized via the RMS function are sent to the first channel module 220 for spectral analysis.

In addition, the pre-processing module 210 transforms the normalized speech segment into a spectrogram using a short-time Fourier transform (STFT), and transmits it to the second channel module 230. A spectrogram is a visual representation of a voice signal in the form of a two-dimensional image, and a 2D CNN model is most suitable for extracting high-level features to recognize various emotions.

The first channel module 220 extracts spectral features from the spectral speech signal received from the preprocessing module 210 . In the present invention, a convolutional neural network (CNN) is used as the first channel module 220.

The first channel module 220 may use a dilated CNN (CNN). The first channel module 220 receives the spectrum for the n-th segment (Rn) of the voice signal as an input and extracts the spectrum feature (F1(Rn)) for the segment. To this end, Equation 2 below is used. can be used

[Equation 2]

은

특징맵에 대하여 카테고리적으로 연결된 네트워크 레이어(L-1)에서의 특징맵을 나타내고,

은

에 대한 컨볼루션 커널을 나타내고,

는 바이어스, *는 컨볼루션 연산자,

는 Relu나 시그모이드와 같은 비선형 활성함수를 나타낸다.At this time,

silver

Indicates a feature map in a network layer (L-1) connected categorically with respect to the feature map,

silver

Represents the convolution kernel for

is the bias, * is the convolution operator,

represents a non-linear activation function such as Relu or sigmoid.

Since the input tensor Rn input to the first channel module 220 is a 1D signal, 1D calculation is performed using a 1D convolution and pooling layer, and spectral features thereof are extracted.

The second channel module 230 extracts spatial features from the spectrogram of the voice signal received from the preprocessing module 210 . In the present invention, a 2D CNN is used as the second channel module 230.

The second channel module 230 may use a dilated CNN (CNN). The second channel module 230 receives the input spectrogram Cn and extracts a spatial feature F2(Cn). At this time, as shown, the second channel module 230 may include a configuration in which a plurality of extended CNN layers and BN (Batch Normalization) layers are alternately arranged and a pooling layer.

The fusion layer module 240 generates a common spatial spectrum representing spatial & sepctral fused features from the spectral features output from the first channel module 220 and the spatial features output from the second channel module 230. Create feature vectors. For this purpose, Equation 3 below can be used.

[Equation 3]

는 결합 연산자로서, 공간 특징과 스펙트럼 특징을 모두 연결하며, 공동 공간 스펙트럼 특징 벡터는

으로 정의된다.W ₁ and W ₂ denote weight matrices, and b ₁ and b ₂ denote biases in connected layers during operation.

is a combinational operator, concatenating both spatial and spectral features, and the joint spatial spectral feature vector is

is defined as

Optimal feature selection module 250 selects the optimal feature from the output of the fusion layer module. Optimal feature selection module 250 may be implemented by various feature selection algorithms.

For example, a Neighbor Component Analysis (NCA) method is known. NCA is a supervised learning method that classifies multivariate data into several classes. It is a distance-based method and is widely used as a method for selecting positive weight features. . To solve this problem, we proposed Iterative Neighbor Component Analysis (INCA), which automatically removes redundancies and selects the optimal number of features.

INCA selects the length of the feature vector according to the task, which utilizes the NCA feature selection method, and performs the following steps.

First, the common space spectral feature vector is normalized by applying a minimum-maximum normalization method that effectively uses NCA according to Equation 4.

[Equation 4]

In this case, i represents a natural number from 1 to n, and each feature is individually normalized and stored in array X. After normalization, indexes are sorted according to NCA using Equation 5.

[Equation 5]

Using this equation, the indices are ordered by the length of the normalized feature "x" and the target represents the actual output. Iterative feature selection using indices has high computational complexity due to undefined feature ranges. To reduce the cost, the number of features is limited like “x = {1, 2, 3…1000}” and then the minimum error rate is found from the selected features. Finally, we use the index value to select the optimal feature and perform an additional process for final prediction.

The classifier module 260 performs emotion classification on the selected optimal feature. For example, the speaker's emotional state may be classified as 'angry', 'sad', 'happy', or 'normal' as an output of the optimal feature. To this end, the classifier module 260 is iteratively learned using a plurality of training data, and using a soft max loss function defined by Equation 6 as follows, for the output of the optimal feature selection module 250 The loss is calculated, and the weight update is performed in a direction that minimizes the loss.

[Equation 6]

은 C⁽ⁿ⁾의 k 번째 요소를 나타내는 것으로, n번째 훈련 데이터에 대한 k 번째 감정의 확률을 나타낸다. θ 는 커널 및 편향 값을 나타낸다. 1 은 노출 함수(revealing function )로서 괄호 안의 조건이 충족되면 값이 1이되고 그렇지 않으면 0이되는 조건을 갖는다.N represents the total number of training data, L ⁽ⁿ⁾ represents the ground truth label of the nth training data,

represents the k-th element of C ⁽ⁿ⁾ , and represents the probability of the k-th emotion for the n-th training data. θ represents the kernel and bias values. 1 is the revealing function, which has a condition that the value becomes 1 if the conditions in parentheses are met and 0 otherwise.

Through such a loss function, it is possible to optimize the softmax layer for final prediction using the stochastic gradient descent method for θ.

For reference, according to an embodiment of the present invention, all kernels of the convolution operation are initialized using a standard variance of 0.05 and a mean using a Gaussian random distribution during model learning. And, we used a fixed learning rate of 0.0001 for all datasets with 100 training epochs. We chose a batch size of 64 for the entire training process and achieved high accuracy with a loss of 0.215 in training and 0.346 in validation.

Deep learning models need sufficient training data to ensure high recognition results. However, in the SER field, the actual labeled data is limited and is not sufficient for model training. To improve model performance with limited training data, training was performed by dividing each utterance into several segments and providing similar true labels containing all segments of the same utterance during training.

Meanwhile, for the evaluation of the emotion classification model according to the present invention, three standard databases were used, which are known as EMO-DB, SAVEE, and RAVDESS, respectively.

All of these databases are written in scripts, and the data is collected by reading prepared sentences with various emotions such as fear, anger, sadness, surprise, happiness, and calm. Each database includes voice files recorded by a plurality of male or female actors, which are recorded according to a predetermined sampling rate.

In the present invention, an emotion classification model was learned and evaluated using such a standard database. More specifically, each data was split at a ratio of 80:20 in each fold, and the system performance was evaluated using a 10-fold cross-validation technique.

4 is a flowchart illustrating an emotion recognition method according to an embodiment of the present invention.

First, the emotion recognition apparatus 100 receives the speaker's voice signal (S410).

Next, the received voice signal is input to the emotion classification model to classify the speaker's emotion, and the following steps are sequentially performed.

A preprocessing step is performed to generate a spectrum and a spectrogram for the audio signal. As described above, in order to generate a spectrum, a normalization process through an RMS function or a short-time Fourier transform can be used in order to generate a spectrogram.

Next, spectral features are extracted from the spectrum of the voice signal (S420). To this end, spectral features are extracted using a 1D CNN through the first channel module 220.

Next, spatial features are extracted from the spectrogram of the audio signal (S430). To this end, spatial features are extracted using a 2D CNN through the second channel module 230.

)를 생성한다.Next, a joint spatial spectral feature vector is generated from the spectral feature and the spatial feature (S440). To this end, using Equation 3 described above, the joint space spectral feature vector (

) to create

Next, an optimal feature is selected from the common space spectral feature vector (S450). To this end, optimal features are selected according to the Iterative Neighbor Component Analysis (INCA) method described above.

Next, emotion classification is performed on the selected optimal feature (S460). To this end, an operation of calculating a loss for the output of the optimal feature selection module through a soft max loss function and performing weight update in a direction that minimizes the loss is repeatedly performed. In this way, with respect to the emotion classification model whose weights have been updated, a voice signal to be classified is input, and an inference process of classifying emotions is performed.

5 illustrates performance evaluation results of an emotion classification model according to an embodiment of the present invention.

As a result of graphing the overall prediction results for the proposed model, it can be confirmed that recognition rates of 95%, 82%, and 85% can be secured for the EMO-DB, SAVEE, and RAVDESS data sets, respectively.

An embodiment of the present invention may be implemented in the form of a recording medium including instructions executable by a computer, such as program modules executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

Although the methods and systems of the present invention have been described with reference to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general-purpose hardware architecture.

The above description of the present application is for illustrative purposes, and those skilled in the art will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts thereof should be construed as being included in the scope of the present application.

100: voice-based emotion recognition device
110: communication module
120: memory
130: processor
140: database
200: emotion classification model
210: voice signal pre-processing module
220: first channel module
230: second channel module
240: fusion layer module
250: optimal feature selection module
260: classifier module

Claims (12)

An emotion recognition device for recognizing a speaker's emotion based on a voice signal,
a memory in which a voice-based emotion recognition program is stored; and
A processor for executing a program stored in the memory;
The voice-based emotion recognition program receives a voice signal of a speaker and inputs the received voice signal to an emotion classification model to classify the speaker's emotion,
The emotion classification model includes a first channel module for extracting a spectrum feature from the spectrum of the voice signal, a second channel module for extracting a spatial feature from the spectrogram of the voice signal, a spectrum feature output from the first channel module and the A fusion layer module for generating a joint spatial spectrum feature vector from the spatial features output from the second channel module, an optimal feature selection module for selecting an optimal feature from the output of the fusion layer module, and emotion classification for the selected optimal feature Emotion recognition device comprising a classifier module that performs. According to claim 1,
The emotion classification model further comprises a pre-processing module for generating a spectrum and a spectrogram of the voice signal. According to claim 1,
The first channel module extracts the spectral features using a one-dimensional convolutional neural network (CNN);
Wherein the second channel module extracts the spatial feature using a two-dimensional CNN. According to claim 1,
Wherein the fusion layer module generates the joint spatial spectrum feature vector according to the following equation.
[Equation 1]

F ₁ (R _n ) denotes a spectral feature, F ₂ (C _n ) denotes a spatial feature, W ₁ and W ₂ denote weight matrices, and b ₁ and b ₂ denote biases in connected layers during operation. ,

is a combinational operator, concatenating both spatial and spectral features, and the joint spatial spectral feature vector is

defined as According to claim 1,
The optimal feature selection module selects the optimal feature according to the Iterative Neighbor Component Analysis (INCA) method, normalizes the joint spatial spectrum feature vector according to Equation 2, and sorts the index according to Equation 3 That is, an emotion recognition device.
[Equation 2]

i represents a natural number from 1 to n, and each feature is individually normalized and stored in array X
[Equation 3]

index is ordered by the length of the normalized feature "x", and target represents the actual output According to claim 1,
wherein the emotion classification model calculates a loss for the output of the optimal feature selection module through a soft max loss function, and performs weight update in a direction that minimizes the loss. In the emotion recognition method using a voice-based emotion recognition device,
Receiving a speaker's voice signal; and
Classifying the speaker's emotion by inputting the received voice signal into an emotion classification model,
The emotion classification model is
(a) extracting spectral features from the spectrum of the speech signal;
(b) extracting spatial features from the spectrogram of the speech signal;
(c) generating a joint spatial spectral feature vector from the spectral feature and the spatial feature;
(d) selecting optimal features from the common spatial spectral feature vectors; and
(e) performing emotion classification on the selected optimal feature, the emotion recognition method using a voice-based emotion recognition device. According to claim 7,
Wherein the emotion classification model performs a preprocessing step of generating a spectrum and a spectrogram for the voice signal before performing step (a). According to claim 7,
The step (a) extracts the spectral features using a 1-dimensional CNN,
Wherein step (b) is to extract the spatial features using a two-dimensional CNN. According to claim 7,
Wherein step (c) is to generate the joint spatial spectrum feature vector according to the following equation.
[Equation 1]

F ₁ (R _n ) denotes a spectral feature, F ₂ (C _n ) denotes a spatial feature, W ₁ and W ₂ denote weight matrices, and b ₁ and b ₂ denote biases in connected layers during operation. ,

is a combinational operator, concatenating both spatial and spectral features, and the joint spatial spectral feature vector is

defined as According to claim 7,
Step (d) selects the optimal feature according to the Iterative Neighbor Component Analysis (INCA) method, normalizes the joint spatial spectrum feature vector according to Equation 2, and sorts the index according to Equation 3 That is, an emotion recognition method.
[Equation 2]

i represents a natural number from 1 to n, and each feature is individually normalized and stored in array X
[Equation 3]

index is ordered by the length of the normalized feature "x", and target represents the actual output According to claim 7,
Wherein step (e) calculates a loss for the output of the optimal feature selection module through a soft max loss function, and performs weight update in a direction that minimizes the loss.

Images