JP6732703B2 - Emotion interaction model learning device, emotion recognition device, emotion interaction model learning method, emotion recognition method, and program - Google Patents

Description

The present invention relates to a technique for recognizing a feeling of a speaker using context information included in a dialogue.

In dialogue, it is important to recognize the emotion of the speaker. For example, by performing emotion recognition during counseling, the emotions of anxiety and sadness of the patient can be visualized, and it is expected that the counselor can deepen understanding and improve the quality of instruction. In addition, by recognizing human emotions in human-machine dialogue, it is possible to build a more friendly dialogue system, such as when people are happy, they are happy and when they are sad, they are encouraged. Hereinafter, the discussion between the two speakers is called "dialogue". In addition, a speaker who has made a utterance that is a target of emotion recognition among speakers who interact with each other is called a “target speaker”, and a speaker other than the target speaker is called a “partner speaker”. For example, in emotion recognition for counseling, the patient is the target speaker and the counselor is the other speaker.

Non-Patent Document 1 proposes an emotion recognition technique in dialogue. In general, emotion recognition technology often performs emotion recognition independently for each utterance (for example, Non-Patent Document 2). On the other hand, in the technique described in Non-Patent Document 1, focusing on the context information included in the dialogue, in addition to the characteristics of the current utterance, the current target speaker is also based on the past and future feelings of the target speaker itself. By recognizing emotions, the accuracy of emotion recognition in dialogue is improved. This is considered to be because emotions have continuity and relevance.

Martin Wollmer, Angeliki Metallinou, Florian Eyben, Bjorn Schuller, Shrikanth Narayanan, “Context-Sensitive Multimodal Emotion Recognition from Speech and Facial Expression using Bidirectional LSTM Modeling,” in Interspeech 2010, 2010. Che-Wei Huang, Shrikanth Narayanan, “Attention Assisted Discovery of Sub-Utterance Structure in Speech Emotion Recognition,” in Interspeech 2016, 2016.

The context information included in the dialogue includes a lot of information in addition to the emotional information of the target speaker used in the technique described in Non-Patent Document 1. For example, it is information on the emotion of the other speaker. Such information is also considered to be effective in emotion recognition of the target speaker, but the technique described in Non-Patent Document 1 uses only the emotion information of the target speaker among the context information. Therefore, there may be room for improving the accuracy of emotion recognition in dialogue.

In view of the above points, an object of the present invention is to improve not only the emotion information of the target speaker himself but also the context information included in the dialogue to improve the recognition accuracy of the emotion of the target speaker. Is.

In order to solve the above problems, the emotion interaction model learning device according to the first aspect of the present invention is a dialogue voice recording a dialogue consisting of a plurality of utterances of a target speaker and a plurality of utterances of a partner speaker, A learning data storage unit that stores learning data consisting of emotional correct values for each utterance included in the dialogue, and an emotional sequence for each utterance of the target speaker by recognizing each utterance emotion for each utterance extracted from the dialogue voice. The target speaker using the emotion recognition unit for each utterance, which generates the emotional sequence for each utterance of the other speaker, the correct value of the emotion, the emotional sequence for each utterance of the target speaker, and the emotional sequence for each utterance of the other speaker. A model learning unit that learns an emotion interaction model that re-estimates the emotion of the target utterance by inputting the utterance-by-utterance emotion of the target utterance that is the utterance of ,including.

In order to solve the above problems, an emotion recognition device according to a second aspect of the present invention is a model storage unit that stores an emotion interaction model learned by the emotion interaction model learning device according to the first aspect, and a target speaker Recognizing the emotions of each utterance for each utterance included in a dialogue consisting of multiple utterances and multiple utterances of the other speaker, and generating an emotional sequence for each utterance of the target speaker and an emotional sequence for each utterance of the other speaker. The utterance-by-utterance emotion recognition unit, the utterance-by-utterance emotion of the target utterance that is the utterance of the target speaker, and the utterance-by-utterance emotion of the immediately preceding utterance performed by the other speaker immediately before the target utterance are input to the emotion interaction model to input the target utterance And an emotion re-estimation unit that re-estimates the emotion.

According to the present invention, the recognition accuracy of the emotion of the target speaker is improved by using not only the emotion information of the target speaker itself but also the context information included in the dialogue.

FIG. 1 is a diagram for explaining an example in which emotions before and after a target speaker or a partner speaker influence the emotion of the target speaker. FIG. 2 is a diagram for explaining the emotion interaction model. FIG. 3 is a diagram illustrating a functional configuration of the emotion interaction model learning device. FIG. 4 is a diagram illustrating a processing procedure of the emotion interaction model learning method. FIG. 5 is a diagram for explaining emotion recognition using the emotion interaction model. FIG. 6 is a diagram illustrating a functional configuration of the emotion recognition device. FIG. 7 is a diagram illustrating a processing procedure of the emotion recognition method.

The point of the present invention is to recognize the emotion of the target speaker by using the emotion information of the other speaker, which is one of the context information included in the dialogue. Of the context information included in the dialogue, the emotion information of the other speaker is effective in recognizing the emotion of the target speaker. Emotion recognition is a process of classifying an utterance into a plurality of emotion classes. In the following explanation, the emotion class will be anger/joy/sadness/normal/other five types. However, the emotion class is not limited to these and can be set arbitrarily.

A specific example of emotion recognition using context information included in a dialogue will be described with reference to FIG. In the utterance of a target speaker, if the emotion immediately before the target speaker is “normal”, it is difficult to estimate the emotion of the utterance. However, if the emotion of the other speaker immediately before the utterance is "joy," it can be imagined that the emotion of the target speaker is likely to be "joy." This is because the nature of human empathy affects the emotion of the other speaker.

Table 1 shows the results of an investigation of the emotional relationship between the target speaker and the other speaker using a certain voice dialogue database. The unit of each value in the table is a ratio. For example, when the emotion of the target speaker's current utterance is "angry", the rate of the utterance of the other speaker's last utterance being "angry" is 0.38, that is, 38%, and "joy" Is 0.00, or 0%, and the rate of being "sad" is 0.02, or 2%.

The value on the diagonal line from the upper left to the lower right of Table 1 is the rate at which the emotion of the current utterance of the target speaker and the emotion of the utterance immediately before the other speaker match, that is, the rate of occurrence of empathy. According to Table 1, 45% (*1) when the target speaker's current utterance was "joy" and 42% (*2) when "sadness" was the other party's talk The utterance immediately before the person expresses the same feeling. That is, it is understood that the emotion of the target speaker is influenced by the emotion of the other speaker due to empathy. From this, it can be seen that in emotion recognition in dialogue, the emotion information of the other speaker is effective for emotion recognition of the target speaker.

In the example of FIG. 1, the target speaker and the other party speaker speak alternately, but the target speaker or the other party speaker may continuously make a plurality of utterances. For example, if the utterance of the other speaker continues multiple times before the utterance of the target speaker for emotion recognition, "the utterance immediately before the other speaker" is the last of the utterances of the other speaker that continues multiple times. Is the utterance. On the other hand, if the target speaker's utterance continues multiple times after the target speaker's utterance, the utterance immediately before the target speaker's utterance is the same for each target speaker's utterance that continues multiple times. Is used. In the following description, the utterance of the target speaker that is the target of emotion recognition is referred to as “target utterance”, and the utterance made by the other speaker immediately before the target utterance is referred to as “previous utterance”.

In the present invention, the emotion of the other speaker is used for re-estimating the emotion of the target speaker. That is, the emotions recognized from each utterance (hereinafter referred to as “emotions for each utterance”) are obtained for all utterances included in the dialogue, and each emotion of the target speaker and each utterance of the other speaker are obtained. Re-estimate the emotion of the target speaker based on the emotion. Hereinafter, the re-estimation model used in the present invention will be referred to as an “emotion interaction model”.

Hereinafter, embodiments of the present invention will be described in detail. In the drawings, components having the same function are denoted by the same reference numerals, and duplicate description will be omitted.

[Emotion interaction model learning device]
The emotion interaction model learning device according to the embodiment learns the emotion interaction model used for estimating the emotion of the target speaker as follows.

1. Dialogue voices that record dialogues that include multiple utterances of the target speaker and multiple utterances of the other speaker, and an emotion label that represents the correct answer value of the emotion of the target speaker assigned to each utterance of the target speaker. Prepare the learning data consisting of and. The emotion label is manually assigned in advance.

2. Recognize the emotion of each utterance of the target speaker and the other speaker from the dialogue voice of the learning data. For recognition of each utterance emotion, for example, the technique described in Non-Patent Document 2 or the like is used.

3. An emotion interaction model is learned using a series of three sets of emotion labels included in the learning data, an estimated value of emotion of each utterance of the target speaker, and an estimated value of emotion of each utterance of the other speaker.

FIG. 2 shows an example of the structure of the emotion interaction model. As shown in FIG. 2, the emotion interaction model configures one utterance emotion estimator for one target utterance. The utterance emotion estimator receives the estimated value of each utterance emotion of the target utterance and the estimated value of each utterance emotion of the immediately preceding utterance as input, and uses the past and/or future emotion information of the target speaker to determine the target utterance. The emotion is re-estimated and the estimated value is output. The utterance emotion estimator is specifically, for example, a recurrent neural network (RNN). By using the recurrent neural network, in addition to the estimated value of the emotion for each utterance of the target speaker and the estimated value of the emotion for each utterance of the other speaker, as well as the technique described in Non-Patent Document 1, It is possible to use information about past and/or future emotions. That is, emotion recognition based on the context information between the target speaker and the other speaker becomes possible.

As shown in FIG. 3, the emotion interaction model learning device 1 according to the exemplary embodiment includes a learning data storage unit 10, a speech detection unit 11, an emotion recognition unit for each utterance 12, a model learning unit 13, an emotion recognition model storage unit for each utterance 19, And an emotion interaction model storage unit 20. The emotion interaction model learning device 1 learns an emotion interaction model using the learning data stored in the learning data storage unit 10, and stores the learned emotion interaction model in the emotion interaction model storage unit 20. The emotion interaction model learning method of the embodiment is realized by the emotion interaction model learning device 1 performing the processing of each step shown in FIG.

The emotion interaction model learning device 1 is configured, for example, by loading a special program into a known or dedicated computer having a central processing unit (CPU), a main memory (RAM: Random Access Memory), and the like. It is a special device. The emotion interaction model learning device 1 executes each process under the control of the central processing unit, for example. The data input to the emotion interaction model learning device 1 and the data obtained by each process are stored in, for example, the main storage device, and the data stored in the main storage device is read to the central processing unit as necessary. And used for other processing. At least a part of each processing unit included in the emotion interaction model learning device 1 may be configured by hardware such as an integrated circuit. Each storage unit included in the emotion interaction model learning device 1 is, for example, a main storage device such as a RAM (Random Access Memory), an auxiliary storage device configured by a semiconductor memory element such as a hard disk, an optical disk, or a flash memory (Flash Memory). , Or middleware such as a relational database or key-value store. Each storage unit included in the emotion interaction model learning device 1 may be logically divided, and may be stored in one physical storage device.

The learning data storage unit 10 stores learning data used for learning the emotion interaction model. The learning data represents a dialogue voice containing a dialogue including a plurality of utterances of a target speaker and a plurality of utterances of a partner speaker, and a correct value of an emotion given to each utterance included in the dialogue voice. It consists of emotion labels. The emotion label may be manually assigned in advance.

The utterance emotion recognition model storage unit 19 stores the utterance emotion recognition model used by the utterance emotion recognition unit 12. The emotion recognition model for each utterance is assumed to be used in the method for emotion recognition for each utterance described in Non-Patent Document 2, for example. The emotion recognition model for each utterance is learned in advance by the method described in Non-Patent Document 2, for example. At this time, in the pre-learning of the emotion recognition model for each utterance, the dialogue voice stored in the learning data storage unit 10 may be used as the learning data, and another learning data (a utterance and a set of emotion labels corresponding to the utterance may be used). Set) may be used.

Hereinafter, the emotion interaction model learning method executed by the emotion interaction model learning device 1 according to the embodiment will be described with reference to FIG. 4.

In step S11, the utterance detection unit 11 detects the utterance section from the dialogue voice stored in the learning data storage unit 10, and obtains the series of the utterance of the target speaker and the series of the utterance of the partner speaker. As a method of detecting the utterance section, for example, a method based on power threshold processing can be used. Further, another utterance section detection method such as a method based on the likelihood ratio of the voice/non-voice model may be used. Hereinafter, the utterances of the speakers are arranged in chronological order of the dialogue and are referred to as “utterance series”. The utterance detection unit 11 outputs the acquired utterance sequence of the target speaker and the obtained utterance sequence of the other speaker to the emotion recognition unit 12 for each utterance.

In step S12, the utterance emotion recognition unit 12 receives the utterance sequence of the target speaker and the utterance sequence of the other speaker from the utterance detection unit 11, and utterance emotion recognition stored in the utterance emotion recognition model storage unit 19 Using the model, emotion recognition for each utterance is performed for each utterance included in each utterance sequence. Here, it is assumed that the method described in Non-Patent Document 2 is used for the recognition of emotion for each utterance. Further, for example, a utterance-by-utterance emotion recognition method such as classification based on the threshold of the utterance average of the fundamental frequency and power may be used. As a result of recognizing the emotion for each utterance for each utterance, an estimated value of the emotion for each utterance corresponding to each utterance can be obtained. This is a posterior probability vector that lists posterior probabilities for each emotion class. Hereinafter, a sequence of estimated emotion values for each utterance in chronological order of dialogue will be referred to as an “emotion series for each utterance”. The utterance emotion recognition unit 12 outputs the utterance emotion sequence of the target speaker and the utterance emotion sequence of the partner speaker to the model learning unit 13.

In step S<b>13, the model learning unit 13 receives the utterance-by-utterance emotional sequence of the target speaker and the utterance-by-utterance emotional sequence of the other speaker from the utterance-by-utterance emotion recognition unit 12, and stores the dialogue voice stored in the learning data storage unit 10. The emotion label corresponding to each utterance of the target utterance is read, the estimated value of the emotion of each utterance of the target utterance and the estimated value of the emotion of each utterance of the immediately preceding utterance are input, and the past and/or future emotion information of the target speaker is used. We re-estimate the emotion of the target utterance and learn the emotion interaction model that outputs the estimated value of the emotion of the target utterance. The model learning unit 13 stores the learned emotion interaction model in the emotion interaction model storage unit 20.

The emotion interaction model uses a recurrent neural network (RNN) as shown in FIG. Here, as the RNN, for example, a long short-term memory recurrent neural network (LSTM-RNN) is used. However, a recurrent neural network other than LSTM-RNN may be used, and for example, a gated recurrent unit (GRU) may be used. The LSTM-RNN is composed of an input gate and an output gate, or an input gate, an output gate, and a forget gate, and the GRU is composed of a reset gate and an update gate. The LSTM-RNN may be a bidirectional LSTM-RNN or a unidirectional LSTM-RNN. When one-way LSTM-RNN is used, emotion information can be recognized even in the middle of a dialogue because only past emotion information is used. When the interactive LSTM-RNN is used, information on future emotions can be used in addition to information on past emotions, so emotion recognition accuracy is improved, while all utterances from start to end of dialogue It is necessary to input the sequence based on the estimated values of emotions obtained from, at one time, and it is suitable when emotion recognition of the entire dialogue is performed after the dialogue ends. For learning the emotion interaction model, for example, the BPTT (Back Propagation Through Time) method, which is an existing LSTM-RNN learning method, is used.

[Emotion recognition device]
The emotion recognition apparatus of the embodiment recognizes the emotion of the utterance of the target speaker using the emotion interaction model as follows.

1. Recognize the emotions of each utterance of the target speaker and the other speaker from the target speech. As a method of recognizing emotions for each utterance, for example, the technique described in Non-Patent Document 2 or the like is used as in the case of learning the emotion interaction model.

2. Estimates of the emotions of each utterance of the target speaker and the other speaker are input to the emotion interaction model to re-estimate the emotions of the target speaker.

FIG. 5 shows an example of the operation of re-estimating the emotion of the target speaker. In FIG. 5, both the speaker A and the speaker B participating in the dialogue are target speakers. In this case, when the speaker A is the target speaker, the speaker B is regarded as the partner speaker, and when the speaker B is the target speaker, the speaker A is regarded as the partner speaker. Can perform emotion recognition. In the example of FIG. 5, the emotions for each utterance recognized from the utterances of the speaker A and the speaker B included in the dialogue voice are “normal”, “joy”, “normal”, and “normal” in order from the earliest time. , By re-estimating using the emotion interaction model, it is updated to "normal", "joy", "joy", and "joy" by being affected by the emotion of each utterance of the immediately preceding utterance.

As shown in FIG. 6, the emotion recognition device 2 according to the embodiment includes a utterance-by-utterance emotion recognition model storage unit 19, an emotion interaction model storage unit 20, a utterance detection unit 21, a utterance-by-utterance emotion recognition unit 22, and an emotion reestimation unit 23. including. The emotion recognition device 2 receives a conversation voice in which a conversation voice to be recognized as an emotion is recorded, and uses the emotion interaction model stored in the emotion interaction model storage unit 20 to obtain a target speaker included in the conversation voice. The emotion of each utterance is estimated and a sequence based on the estimated value of the emotion is output. The emotion recognition method of the embodiment is realized by the emotion recognition device 2 performing the processing of each step shown in FIG.

The emotion recognition device 2 is, for example, a special program configured by loading a special program into a known or dedicated computer having a central processing unit (CPU), a main memory (RAM: Random Access Memory), and the like. It is a device. The emotion recognition device 2 executes each process under the control of the central processing unit, for example. The data input to the emotion recognition device 2 and the data obtained by each process are stored in, for example, the main storage device, and the data stored in the main storage device is read to the central processing unit as necessary. It is used for other processing. At least a part of each processing unit of the emotion recognition device 2 may be configured by hardware such as an integrated circuit. Each storage unit included in the emotion recognition device 2 is, for example, a main storage device such as a RAM (Random Access Memory), an auxiliary storage device configured by a semiconductor memory element such as a hard disk, an optical disc, or a flash memory (Flash Memory), or It can be configured by middleware such as a relational database or key-value store. Each storage unit included in the emotion recognition device 2 may be logically divided, and may be stored in one physical storage device.

The utterance emotion recognition model storage unit 19 stores the utterance emotion recognition model used by the utterance emotion recognition unit 22. The emotion recognition model for each utterance is the same as the model used by the emotion interaction model learning device 1.

The emotion interaction model storage unit 20 stores the learned emotion interaction model generated by the emotion interaction model learning device 1.

Hereinafter, the emotion recognition method executed by the emotion recognition device 2 according to the embodiment will be described with reference to FIG. 7.

In step S21, the utterance detection unit 21 detects the utterance section from the dialogue voice input to the emotion recognition device 2, and obtains the utterance sequence of the target speaker and the utterance sequence of the other speaker. This dialogue voice includes a plurality of utterances of the target speaker and a plurality of utterances of the partner speaker, like the dialogue sound of the learning data. As a method of detecting the utterance section, the same method as the utterance detection unit 11 of the emotion interaction model learning device 1 may be used. The utterance detection unit 21 outputs the acquired utterance sequence of the target speaker and the obtained utterance sequence of the other speaker to the emotion recognition unit 22 for each utterance.

In step S22, the utterance-by-utterance emotion recognition unit 22 receives the utterance sequence of the target speaker and the utterance sequence of the partner speaker from the utterance detection unit 21, and utterance-by-utterance emotion recognition stored in the utterance-by-utterance emotion recognition model storage unit 19 Using the model, emotion recognition for each utterance is performed for each utterance included in each utterance sequence. As a method for recognizing the emotion for each utterance, a method similar to that for the emotion recognizing unit for each utterance 21 of the emotion interaction model learning device 1 may be used. The utterance emotion recognition unit 22 outputs the utterance emotion sequence of the target speaker and the utterance emotion sequence of the other speaker to the emotion reestimation unit 23.

In step S23, the emotion re-estimation unit 23 receives the utterance-by-utterance emotion sequence of the target speaker and the utterance-by-utterance emotion sequence of the partner speaker from the utterance-by-utterance emotion recognition unit 22, and estimates the utterance-by-utterance emotion of the target utterance and the immediately preceding value. The estimated value of each utterance emotion of the utterance is input to the emotion interaction model stored in the emotion interaction model storage unit 20 to re-estimate the emotion of the target speaker. This is equivalent to re-recognizing the emotion of the target speaker based on the emotional information of the partner speaker and the past and/or future emotional information of the target speaker. For example, based on the fact that it was difficult to classify “normal” or “joyful” by emotion recognition for each utterance, the feeling of the other speaker immediately before the utterance was “joyful” It can be re-estimated that was the feeling of "joy." This can be expected to improve emotion recognition accuracy. In the emotion re-estimation based on the emotion interaction model, the estimated value of the emotion of each utterance of the target utterance and the estimated value of the emotion of each utterance of the immediately preceding utterance are input to the emotion interaction model, and the emotion is re-estimated by forward propagation. The emotion re-estimation unit 23 re-estimates the emotion by using each utterance of the target speaker included in the dialogue voice as the target utterance, and outputs the sequence based on the estimated value of the emotion of the target speaker from the emotion recognition device 2.

[Modification]
In the above-described embodiment, an example in which the emotion interaction model learning device 1 and the emotion recognition device 2 are configured as separate devices has been described, but the emotion is recognized using the function of learning the emotion interaction model and the learned emotion interaction model. It is also possible to configure a single emotion recognition device that has the function to perform. That is, the emotion recognition device of the modified example includes a learning data storage unit 10, an utterance detection unit 11, an utterance emotion recognition unit 12, a model learning unit 13, an utterance emotion recognition model storage unit 19, an emotion interaction model storage unit 20, and The emotion re-estimation unit 23 is included.

As described above, the emotion interaction model learning device and the emotion recognition device of the present invention learn the emotion interaction model by using not only the emotional sequence for each utterance of the target speaker but also the emotional sequence for each utterance of the other speaker, and the emotion It is configured to re-estimate the emotion of the target speaker using an interaction model. Accordingly, not only the emotion information of the target speaker itself but also the context information included in the dialogue can be used, so that the estimation accuracy of the emotion of the target speaker can be improved.

Although the embodiments of the present invention have been described above, the specific configuration is not limited to these embodiments, and even if the design is appropriately changed without departing from the spirit of the present invention, Needless to say, it is included in the present invention. The various kinds of processing described in the embodiments may be executed not only in time series according to the order described, but also in parallel or individually according to the processing capability of the device that executes the processing or the need.

[Program, recording medium]
When various processing functions in each device described in the above embodiment are realized by a computer, processing contents of functions that each device should have are described by a program. Then, by executing this program on a computer, various processing functions of the above devices are realized on the computer.

The program describing the processing contents can be recorded in a computer-readable recording medium. The computer-readable recording medium may be, for example, a magnetic recording device, an optical disc, a magneto-optical recording medium, a semiconductor memory, or the like.

The distribution of this program is performed by, for example, selling, transferring, or lending a portable recording medium such as a DVD or a CD-ROM in which the program is recorded. Further, the program may be stored in a storage device of a server computer and transferred from the server computer to another computer via a network to distribute the program.

A computer that executes such a program first stores, for example, the program recorded on a portable recording medium or the program transferred from the server computer in its own storage device. Then, when executing the process, this computer reads the program stored in its own storage device and executes the process according to the read program. As another execution form of this program, a computer may directly read the program from a portable recording medium and execute processing according to the program, and the program is transferred from the server computer to this computer. Each time, the processing according to the received program may be sequentially executed. In addition, a configuration in which the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes a processing function only by the execution instruction and result acquisition without transferring the program from the server computer to this computer May be Note that the program in this embodiment includes information that is used for processing by an electronic computer and that conforms to the program (such as data that is not a direct command to a computer but has the property of defining computer processing).

Further, in this embodiment, the present apparatus is configured by executing a predetermined program on the computer, but at least a part of the processing content may be implemented by hardware.

1 Emotion Interaction Model Learning Device 10 Learning Data Storage Unit 11 Utterance Detection Unit 12 Emotion Recognition Unit 13 Model Learning Unit 19 Emotion Recognition Model Storage Unit 2 Emotion Recognition Model Memory Unit 20 Emotion Recognition Device 20 Emotion Interaction Model Storage Unit 21 Utterance Detection Unit 22 Utterance Detection Emotion recognition unit 23 Emotion re-estimation unit

Claims (8)

A learning data storage unit that stores learning data composed of a dialogue voice containing a dialogue consisting of a plurality of utterances of a target speaker and a plurality of utterances of a partner speaker, and learning data consisting of correct answer values of emotions for each utterance included in the dialogue. When,
An utterance-by-utterance recognition unit that recognizes the utterance-by-utterance for each utterance extracted from the dialogue voice, and generates the utterance-by-utterance emotion series of the target speaker and the utterance-by-utterance emotion series of the partner speaker,
Immediately before the target utterance and the utterance for each target utterance, which is the utterance of the target speaker, using the correct value of the emotion, the utterance sequence for each utterance of the target speaker, and the utterance sequence for each utterance of the other speaker A model learning unit that learns an emotion interaction model that re-estimates the emotion of the target utterance by inputting each utterance emotion of the immediately preceding utterance performed by the other speaker to
Emotional interaction model learning device including. The emotion interaction model learning device according to claim 1,
The emotion interaction model constitutes one utterance emotion estimator for one target utterance,
The utterance emotion estimator receives the utterance-by-utterance emotions of the target utterance and the utterance-by-utterance emotions of the immediately preceding utterance as input, and information about emotions related to utterances made by the target speaker before the target utterance or the target utterances. Using the emotion information about the utterance performed by the target speaker before and after, it is to re-estimate the emotion of the target utterance and output the estimated value of the emotion of the target utterance,
Emotional interaction model learning device. The emotion interaction model learning device according to claim 2, wherein
The speech emotion estimator is characterized by comprising any one of an input gate and an output gate, an input gate, an output gate and a forgetting gate, a reset gate and an update gate,
Emotional interaction model learning device. A model storage unit for storing an emotion interaction model learned by the emotion interaction model learning device according to claim 1.
Recognizing emotions for each utterance included in a dialogue consisting of a plurality of utterances of the target speaker and a plurality of utterances of the other speaker, and for each utterance sequence of the target speaker and each utterance of the other speaker An emotion recognition unit for each utterance that generates an emotion sequence,
The emotion of each target utterance, which is the utterance of the target speaker, and the emotion of each utterance of the immediately preceding utterance performed by the other speaker immediately before the target utterance are input to the emotion interaction model, and the emotion of the target utterance is input. An emotion re-estimation unit that re-estimates
Emotion recognition device including. In the learning data storage unit, learning data including a dialogue voice in which a dialogue composed of a plurality of utterances of a target speaker and a plurality of utterances of a partner speaker is recorded, and a correct emotion value for each utterance included in the dialogue. Remembered,
The utterance-by-utterance emotion recognition unit recognizes the utterance-by-utterance emotions for each utterance extracted from the dialogue voice, and generates the utterance-by-utterance emotional series of the target speaker and the utterance-by-utterance emotional series of the partner speaker,
The model learning unit uses the correct answer value of the emotion, the utterance-by-utterance emotional sequence of the target speaker, and the utterance-by-utterance emotional sequence of the partner speaker, and the utterance-by-utterance of the target utterance that is the utterance of the target speaker Learning an emotion interaction model that re-estimates the emotion of the target utterance by inputting the utterance for each utterance of the immediately preceding utterance performed by the other speaker immediately before the target utterance,
Emotional interaction model learning method. An emotion interaction model learned by the emotion interaction model learning method according to claim 5 is stored in the model storage unit,
The utterance-by-utterance emotion recognition unit recognizes the utterance-by-utterance emotions for each utterance included in the dialogue consisting of the plurality of utterances of the target speaker and the plurality of utterances of the partner speaker, and the utterance-by-utterance emotion series of the target speaker and Generate the emotional sequence for each utterance of the other speaker,
The emotion re-estimation unit inputs the utterance-by-utterance emotion of the target utterance, which is the utterance of the objective speaker, and the utterance-by-utterance emotion of the immediately preceding utterance performed by the opposite speaker immediately before the objective utterance, in the emotion interaction model. Re-estimate the emotion of the above target utterance,
Emotion recognition method. Program for causing a computer to function as the emotion interaction model learning equipment according to any one of claims 1 to 3. A program for causing a computer to function as the emotion recognition device according to claim 4.

Images