KR102106700B1 - A method of simultaneous recognition for emotion, age, and gender based on users' voice signal - Google Patents

Abstract

Disclosed is a method and system for simultaneously recognizing emotion, age, and gender based on a user's voice signal. In the method for recognizing user characteristic information executed by a computer, a step of classifying frames according to a plurality of predetermined characteristics based on an input data set corresponding to a voice signal, according to the distinguished characteristics Based on a convolutional neural network, performing learning based on a loss function for each of the characteristics based on a convolutional neural network, and input voice signals based on a learning model generated through the learning. And recognizing a plurality of different user characteristic information.

Description

A method of simultaneous recognition for emotion, age, and gender based on users' voice signal}

Embodiments of the present invention relates to a technique for recognizing a user's emotion, age, and gender, and more particularly, a technique for recognizing a user's emotion, age, and gender based on deep learning.

As interest in interface technologies between a user and a machine, such as artificial intelligence assistant and biometrics-based security, has increased, techniques for recognizing characteristics of human emotions, such as voice and facial expressions, have been actively studied.

Emotion recognition technology using voice signals can be used in various fields. For example, if the user is in an emotional state that is expected to be angry, an intelligent response to connect with the corresponding service is possible by allowing the service to be proposed in a tone of calming calm emotions, calm speech, and the like. In addition, when the user's emotional state is predicted as sadness, intelligent response is possible with services such as suggesting music such as a sad ballad.

In the process of using a smartphone-based service such as a customer center, telephone consultation, and telephone education, knowing information about a user's emotion, age, and gender makes it possible to provide a service suitable for the user's condition.

Korean Patent Publication No. 10-2011-0011969 relates to a method for constructing a voice emotion recognition model using a loss function and a maximum margin technique based on WTM, and the geometric distance between emotion groups of WTM (Watson-Tellegen Emotional Model). Using to quantify the difference between each emotion, to obtain the value of the loss function (loss function) based on the set value, and to build an emotion recognition model to obtain the parameters of each voice emotion model based on the obtained loss function Technology is disclosed.

The present invention relates to a technique for recognizing a user's emotion, age, and gender from a user's voice signal based on deep learning.

In addition, it relates to a technology for providing a service suitable for the recognized user's emotion, age and gender.

In the method for recognizing user characteristic information executed by a computer, a step of classifying frames according to a plurality of predetermined characteristics based on an input data set corresponding to a voice signal, according to the distinguished characteristics Based on a convolutional neural network, performing learning based on a loss function for each of the characteristics based on a convolutional neural network, and input voice signals based on a learning model generated through the learning. And recognizing a plurality of different user characteristic information.

According to one aspect, the step of performing the learning may include calculating a loss for each of the plurality of characteristics based on a softmax function, and calculating the loss for each of the plurality of characteristics. And setting a standard for final learning.

According to another aspect, the step of classifying a frame for each of the plurality of characteristics includes, for the input data set, the user's emotion, age, and gender. It can be divided into frames for.

According to another aspect, the step of performing the learning may include performing max pooling to perform sampling on each convolutional layer based on the convolutional neural network. have.

According to another aspect, the step of recognizing the user characteristic information may simultaneously recognize a user's emotion, age, and gender from the input voice signal.

According to another aspect, in the step of recognizing the user characteristic information, when the user characteristic information corresponds to emotion, the emotional state of the user from the input voice signal is neutral, joy, sadness, anger, disgust, surprise, You can recognize whether you are at least one of the fears.

According to another aspect, the method may further include providing a service corresponding to the current state of the user based on the recognized user characteristic information.

In the user characteristic information recognition system, targeting an input data set corresponding to a voice signal, a frame dividing unit for dividing a frame for a plurality of predetermined characteristics, and targeting the frame for each of the characteristics The learning control unit performs learning based on the loss function for each of the plurality of characteristics based on a convolutional neural network, and corresponds to an input voice signal based on a learning model generated through the learning. And a user characteristic recognition unit that recognizes a plurality of different user characteristic information.

According to one aspect, the learning control unit calculates a loss for each of the plurality of characteristics based on a softmax function, and sets a criterion for final learning based on the calculated sum of the losses for each of the plurality of characteristics. Can be set.

According to another aspect, the frame division unit may be divided into frames for the user's emotion, age, and gender for the input data set.

According to another aspect, the learning control unit may perform max pooling, which performs sampling on each convolutional layer based on the convolutional neural network.

According to another aspect, the user characteristic recognition unit may simultaneously recognize a user's emotion, age, and gender from the input voice signal.

According to another aspect, when the user characteristic information corresponds to emotion, the user characteristic recognition unit may have at least one of a neutral, joy, sadness, anger, disgust, surprise, and fear from the input voice signal. You can recognize whether or not.

According to another aspect, it may further include a service providing unit for providing a service corresponding to the current state of the user based on the recognized user characteristic information.

The present invention can simultaneously recognize a user's emotion, age, and gender from a user's voice signal using a learning based on a convolutional neural network among deep learning-based learning algorithms.

In addition, by recognizing the user's emotion along with the user's age and gender, it is possible to more accurately recognize the user's emotional state by reflecting changes in different ranges, tones, and speeds according to gender and age.

1 is a block diagram illustrating an internal configuration of a user characteristic information recognition system in an embodiment of the present invention.
2 is a flowchart illustrating a method for recognizing user characteristic information in an embodiment of the present invention.
3 is a diagram illustrating a convolutional neural network structure in an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The present embodiments relate to a technique of simultaneously recognizing a user's emotion, age, and gender based on a user's voice signal, and in particular, input using a learning model generated by learning based on a convolutional neural network among deep learning techniques The present invention relates to a technique of simultaneously recognizing a user's current emotional state, age, and gender based on a user's voice signal.

In the present embodiments, "user characteristic information" may indicate a user's state, such as a user's emotion, age, and gender.

In the present embodiments, "convolutional neural network-based learning" may indicate that learning progresses such that the joint loss converges to a minimum value.

1 is a block diagram illustrating an internal configuration of a user characteristic information recognition system in an embodiment of the present invention, and FIG. 2 is a flowchart illustrating a method for recognizing user characteristic information in an embodiment of the present invention.

The user characteristic information recognition system 100 according to the present embodiment may include a processor 110, a bus 120, a network interface 130, and a memory 140. The memory 140 may include an operating system 141 and a service provision routine 142. The processor 110 may include a frame classification unit 111, a learning control unit 112, a user characteristic recognition unit 113, and a service providing unit 114. In other embodiments, the user characteristic information recognition system 100 may include more components than those in FIG. 1. However, there is no need to clearly show most prior art components. For example, the user characteristic information recognition system 100 may include other components such as a display or a transceiver.

The memory 140 is a computer-readable recording medium and may include a non-permanent mass storage device such as random access memory (RAM), read only memory (ROM), and a disk drive. Also, program codes for the operating system 141 and the service providing routine 142 may be stored in the memory 140. These software components may be loaded from a computer-readable recording medium separate from the memory 140 using a drive mechanism (not shown). Such a separate computer-readable recording medium may include a computer-readable recording medium (not shown) such as a floppy drive, disk, tape, DVD / CD-ROM drive, and memory card. In other embodiments, software components may be loaded into memory 140 via network interface 130 rather than a computer-readable recording medium.

The bus 120 may enable communication and data transmission between components of the user characteristic information recognition system 100. The bus 120 may be constructed using a high-speed serial bus, parallel bus, storage area network (SAN), and / or other suitable communication technology.

The network interface 130 may be a computer hardware component for connecting the user characteristic information recognition system 100 to a computer network. The network interface 130 may connect the user characteristic information recognition system 100 to a computer network through a wireless or wired connection.

The user characteristic information recognition system 100 is a user terminal connected to the server, and recognizes user characteristic information such as the user's emotion, age, and gender based on a user's voice signal, and provides a platform corresponding to the recognized characteristic. It may be implemented in the form of (platform), or an application (application, that is, a service app) that recognizes user characteristic information such as user's emotion, age, and gender based on a voice signal input through a speaker provided in the user terminal. It may be implemented as. In this case, when implemented in the form of an application on the user terminal, the user characteristic information recognized by the user terminal may be transmitted to the service provider terminal as a server through the application, and the service provider terminal may apply a service corresponding to the received user characteristic information. Through it can be provided to the user terminal.

The processor 110 may be configured to process instructions of a computer program by performing input / output operations of the basic arithmetic, logic, and user characteristic information recognition system 100. Instructions may be provided to processor 110 by memory 140 or network interface 130, and via bus 120. The processor 110 may be configured to execute program codes for the frame division unit 111, the learning control unit 112, the user characteristic recognition unit 113, and the service provision unit 114. Such program code may be stored in a recording device such as memory 140.

The frame division unit 111, the learning control unit 112, the user characteristic recognition unit 113, and the service provision unit 114 may be configured to perform the steps 210 to 240 of FIG. 2.

In step 210, the frame division unit 111 may classify frames according to a plurality of predetermined characteristics for an input data set corresponding to a voice signal. For example, in order to generate a learning model for simultaneously recognizing a user's emotion, age, and gender, the frame division unit 111 may input a set of input data corresponding to voice signals of different users collected in advance. It can be divided into a frame corresponding to emotion, a frame corresponding to age, and a frame corresponding to gender.

For example, the frame division unit 111 may extract a feature corresponding to a valid sound that is previously specified from an input data set corresponding to a voice signal based on a MFC (Mel Frequency Cepstral Coefficient). At this time, the frame division unit 111 does not extract features for the entire set of input data, but divides them by a certain section, that is, by a certain frame, and extracts features through spectrum analysis for each frame. Can be. For example, since the voice signal is continuously changed in the time domain, it can be assumed that the voice signal does not change much within a predetermined short time in order to extract features for the changing sound. That is, it can be interpreted that there is virtually no change in the audio signal within the error range. Then, the frame division unit 111 may calculate the power spectrum (ie, frequency) for each frame. Here, the power spectrum calculated for each frame may be extracted as a feature, that is, a feature vector. As described above, when the neural frequency frequency is calculated, it is possible to know how much energy exists in each section.

In step 220, the learning control unit 112 may perform learning based on a loss function for a plurality of characteristics based on a convolutional neural network, targeting the frames according to the distinguished characteristics. That is, the frames to which the power spectrum calculated by the frame classification unit 111 belongs may be set as an input value of an input layer of the convolutional neural network. For example, 25 frames per characteristic (gender, age, emotion, etc.) may be set in the input layer for learning. As described above, learning can be performed by receiving the input power spectrum for each frame, that is, feature vectors corresponding to each characteristic (gender, age, emotion). At this time, the learning control unit 112 is a convolutional neural network structure composed of an input layer, a hidden layer, and an output layer, so that learning is performed at a desired value with respect to a value suggested by the output layer. Can be controlled. Here, each layer may be connected to a weight value intersecting each other, and the learning control unit 112 controls to perform learning by adjusting individual weights for each layer so that a desired value for the same input layer is output. can do.

Specifically, the learning control unit 112 may perform max pooling, which performs sampling on each convolutional layer (CNN_ReLU) according to characteristics belonging to at least one hidden layer. For example, when at least 25 frames are set for each characteristic, the learning control unit 112 determines the largest value among the multiplied weights of feature vectors corresponding to the 25 frames by characteristic (eg, by gender, age, and emotion). Max pooling can be performed.

When the max pooling is performed, the learning control unit 112 may calculate the loss for each of the plurality of characteristics based on the softmax function, and the criteria for the final learning based on the calculated sum of the losses for each of the plurality of characteristics. You can set That is, the learning control unit 112 may calculate a loss corresponding to each emotion, age, and gender, and calculate the joint loss by combining the calculated loss based on Equation 1 below. And, the joint loss can be set as a criterion for final learning. In other words, learning may be performed such that loss corresponding to each of emotion, age, and gender is calculated so that learning is performed to minimize joint loss.

[Equation 1]

In Equation 1, L _gender may represent a loss corresponding to gender calculated based on a softmax function, L _age may be a loss corresponding to age, and L _emotion may represent a loss corresponding to _emotion .

As described above, the loss corresponding to each of emotion, age, and gender is calculated using the softmax function, and joint loss is calculated as the sum of the calculated losses, and then set as a criterion for final learning. Multi-task learning may be performed to simultaneously recognize a user's emotion, age, and gender corresponding to various tasks on a voice signal.

According to Equation 1, the learning control unit 112 may set the weighted sum as a criterion for the final learning by simply training the variables instead of simply adding weights for each task (gender, emotion, age).

In operation 230, the user characteristic recognition unit 113 may recognize a plurality of different user characteristic information corresponding to the input voice signal simultaneously based on the learning model generated through learning. For example, the joint loss calculated based on Equation 1 may be set as a criterion for final learning, and a voice signal input through a speaker of a user terminal may be set as an input to the generated learning model. And, as the output of the learning model, the age and gender of the user recognized as the target of the input voice signal may be output. At this time, the current emotional state of the user along with the age and gender may be simultaneously recognized and output.

For example, the range and intensity of voice may be different depending on the age and gender of the user. For example, a female's transliteration band shows a difference in a higher frequency band than a male, and a younger child may have a relatively higher difference in frequency band than an adult. In this way, while the gender and age of the user are recognized in the learning model, the current emotional state of the user may be recognized and output in detail. There are changes in tone, speed, tremor, etc. according to the state of emotion, and feature vectors representing tone, speed, tremor, etc., and corresponding age of the gender based on feature vectors corresponding to the range and intensity of the voice signal It may be recognized whether the user's emotional state corresponds to a plurality of predetermined emotional states. For example, it may be recognized whether the user's emotional state corresponds to at least one of seven emotions such as neutral, joy, sadness, anger, disgust, surprise, and fear.

For example, in the case of a voice signal of a woman in her twenties or a child under the age of 7, the frequency corresponding to the extracted feature vector corresponds to joy, but in the case of a man in his 40s, they may be different from each other, such as anger and surprise. . Accordingly, by simultaneously recognizing three characteristics of the user's age, gender, and emotion on a single voice signal input to the learning model, the user's actual emotional state is reflected by physiological voice characteristics that vary depending on the user's age and gender. Can be recognized better. Here, the case corresponding to joy has been described as an example, but it may be recognized as corresponding to two or more emotional states. For example, it may be possible to be recognized as an emotional state corresponding to surprise and fear, an emotional state of surprise, fear and anger, and the like.

In step 240, when the user's age, gender, and emotion are recognized, the service provider 114 may provide a service suitable for the user's condition based on the recognized age, gender, and emotion.

For example, in the case of providing a music service, if the user's emotion corresponds to joy and a girl under 7 years of age, a happy birthday song may be provided. If the user's emotion corresponds to joy and the boy under 7 years of age, a popular cartoon ending song or opening song may be provided at a predetermined age. When the user's emotion corresponds to joy and to a man in his twenties, it is possible to provide a popular KPOP, etc. at a predetermined age. Similarly, in the case of providing a movie service, the service provider 114 may provide a service in consideration of the user's emotion, age, and gender.

In addition, when the customer center, telephone consultation, when the user's emotion, age and gender are recognized based on the voice signal input through the connected counterpart terminal, the service providing unit 114 displays the recognized user in the predefined customer response manual. A response method corresponding to emotion, age, and gender may be provided to a customer center or a telephone counselor's terminal. At this time, emotion, age, and gender information of the recognized user may be provided together. Then, a customer center representative or a telephone counselor can help to effectively respond to the user's request according to the user's current state by referring to the user's emotion, age and gender information, and the provided response method. For example, when a user calls a customer center due to product failure or misdelivery, there may be a user with an emotional state or a neutral user. At this time, the service providing unit 114 may provide a corresponding response method along with the recognized user's age, gender, and emotional state, so that the customer center representative can respond appropriately to the user. For example, if you are an elderly male and your emotions are in anger, you can make them quickly respond to your needs without spending a lot of time or raising your anger gauge, and at this time, the older male calls with a speech and speed that is easy to recognize. It is possible to provide a response method to respond to the.

3 is a diagram illustrating a convolutional neural network structure in an embodiment of the present invention.

Referring to FIG. 3, the convolutional neural network 300 may include an input layer 310, a hidden layer 320 and an output layer 330. The hidden layer 320 may be composed of a plurality of layers.

In FIG. 3, 25 40-dimensional mel spectrograms may be set as inputs of the input layer. At this time, for each characteristic (emotion, age, gender), 25 frames (ie, mel spectrogram * 25) may be set as input. Mel spectrogram is a 2D expression method optimized for human auditory perception, and can represent a speech signal expression technique that preserves the most important information by compressing a Short Time Fourier Transform (STFT) on the frequency axis.

As described above, when a plurality of frames are set for each characteristic in the input layer, the weighting is multiplied for each feature vector corresponding to each frame, and max pooling may be performed to collect the maximum value among the multiplied values by characteristics. .

For each of the max pooled values, a loss for each characteristic is calculated based on a soft max function, and the calculated loss is summed based on Equation 1 to simultaneously recognize emotion, gender, and age in a voice signal. You can calculate the joint loss that creates the training model. As described above, when the joint loss is calculated, the final learning criterion is set, and learning may be performed so that the joint loss converges to a minimum value for an input data set. Then, when a specific speech signal is input, based on a learning model generated as the learning is performed, a feature vector extracted from the input specific speech signal is set as an input of the learning model, and corresponds to the speech signal. The user's age, gender and emotion can be recognized and output as output values.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded in the medium may be specially designed and configured for the embodiments or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. -Hardware devices specifically configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler.

As described above, although the embodiments have been described by a limited embodiment and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques are performed in a different order than the described method, and / or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if substituted or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (14)

A method for recognizing user characteristic information executed by a computer, the method comprising:
Classifying a frame according to a plurality of predetermined characteristics based on an input data set corresponding to a voice signal;
Performing learning based on the loss function for each of the plurality of characteristics based on a convolutional neural network for the divided frames for each characteristic; And
Recognizing a plurality of different user characteristic information corresponding to the input voice signal based on the learning model generated through the learning.
Including,
The step of dividing the frame for each of the plurality of characteristics,
Extracting a power spectrum calculated through spectrum analysis for each frame divided for a certain section from an input data set corresponding to a speech signal based on a Mel Frequency Cepstral Coefficient (MFC) as a feature vector
Including,
The step of performing the learning,
The frames to which the calculated power spectrum for each frame belongs are set as an input value of an input layer of a convolutional neural network, and learning is performed by receiving feature vectors corresponding to the power spectrum for each frame as input.
Including,
Recognizing the user characteristic information,
When max pooling is performed for sampling of each convolution layer based on the convolutional neural network, and when the max pooling is performed, loss is performed according to a plurality of characteristics based on a soft max function (loss) is calculated, and the equation 1 (

), Calculates the joint loss, sets the calculated joint loss as a reference for the final learning, inputs a speech signal into the generated learning model, and outputs the learning model as the output of the learning model. Steps to recognize age and gender at the same time
Including,
In Equation 1, L _gender is the loss corresponding to the gender calculated based on the softmax function, L _age is the loss corresponding to the age, L _emotion is the loss corresponding to the _emotion
How to recognize user property information. delete According to claim 1,
The step of classifying a frame for each of the plurality of characteristics may include:
Targeting the input data set, and dividing it into frames for the user's emotion, age, and gender
A method for recognizing user characteristic information, characterized in that. delete delete According to claim 1,
Recognizing the user characteristic information,
When the user characteristic information corresponds to emotion, recognizing whether the user's emotional state corresponds to any one of neutral, joy, sadness, anger, disgust, surprise, and fear from the input voice signal
A method for recognizing user characteristic information, characterized in that. According to claim 1,
Providing a service corresponding to the current state of the user based on the recognized user characteristic information
A method for recognizing user characteristic information further comprising a. In the user characteristic information recognition system,
A frame dividing unit for classifying frames according to a plurality of predetermined characteristics for an input data set corresponding to a voice signal;
A learning control unit that performs learning based on the loss function for each of the plurality of characteristics based on a convolutional neural network, targeting the divided frames for each characteristic; And
A user characteristic recognition unit that recognizes a plurality of different user characteristic information corresponding to the input voice signal based on the learning model generated through the learning.
Including,
The division,
And extracting a power spectrum calculated through spectrum analysis for each frame divided for a predetermined section from an input data set corresponding to a speech signal based on a Mel Frequency Cepstral Coefficient (MFC), as a feature vector,
The learning control unit,
Frames to which the calculated power spectrum for each frame belongs are set as input values of an input layer of a convolutional neural network, and receiving feature vectors corresponding to the power spectrum for each frame as input, and performing learning. ,
The user characteristic recognition unit,
When max pooling is performed for sampling of each convolution layer based on the convolutional neural network, and when the max pooling is performed, loss is performed according to a plurality of characteristics based on a soft max function (loss) is calculated, and the equation 1 (

), Calculates the joint loss, sets the calculated joint loss as a reference for the final learning, inputs a speech signal into the generated learning model, and outputs the learning model as the output of the learning model. Including simultaneously recognizing age and gender,
In Equation 1, L _gender is the loss corresponding to the gender calculated based on the softmax function, L _age is the loss corresponding to the age, L _emotion is the loss corresponding to the _emotion
User characteristic information recognition system. delete The method of claim 8,
The frame division unit,
Targeting the input data set, and dividing it into frames for the user's emotion, age, and gender
User characteristic information recognition system, characterized by. delete delete The method of claim 8,
The user characteristic recognition unit,
When the user characteristic information corresponds to emotion, recognizing whether the user's emotional state corresponds to any one of neutral, joy, sadness, anger, disgust, surprise, and fear from the input voice signal
User characteristic information recognition system, characterized by. The method of claim 8,
A service providing unit that provides a service corresponding to a user's current state based on the recognized user characteristic information
User characteristic information recognition system further comprising a.

Images