JP2023171101A - Learning device, estimation device, learning method, estimation method and program - Google Patents

Abstract

To provide a learning device, an estimation device, a learning method, an estimation method and a program, which improve estimation accuracy of an utterer's emotion.SOLUTION: In a learning device, a control unit includes a learning unit for learning a mathematical model which estimates emotion of an estimation object on the basis of an emotion time series which depends on emotion of the estimation object during utterance. The mathematical model performs common information acquisition processing which applies the same mapping to two or more types of integrated vectors to which semantic vectors indicating semantic intervals, which are obtained as a result in which the emotion time series are divided in a time direction, are applied under a division condition being a condition which is previously determined according to the type of the emotion time series on the division of the emotion time series for an emotion time series vector indicating the emotion time series. Mapping is updated by learning.SELECTED DRAWING: Figure 3

Description

Application for application of Article 30, Paragraph 2 of the Patent Act https://urldefense. com/v3/___https://doi. org/10.11517/pjsai. JSAI2021.0_1N4IS1a04＿＿;! ! OhYLZkit9p47d2A! tgXzqFzv9TekHSM3dtP_BsPwARY3xw8lrckskexffOipN2A1DVlYR2QxmCYlzbCvWpYvxDO60Jo8dPxFIZ07laQ$ Posting date June 8, 2021

The present invention relates to a learning device, an estimation device, a learning method, an estimation method, and a program.

There is growing interest in technology that uses machine learning technology to estimate the emotion of a speaker.

Kaicheng Yang, et.al., “CM-BERT: Cross-Modal BERT for Text-Audio Sentiment Analysis”, 2020 Association for Computing Machinery, ACM ISBN 978-1-4503-7988-5/20/10

However, conventional techniques have mainly used a single piece of information to estimate emotions. To estimate emotions, it is important to analyze information that has content corresponding to the emotion, but since emotions are complex, estimation accuracy using a single piece of information may be low. .

In view of the above circumstances, an object of the present invention is to provide a technique that improves the accuracy of estimating the emotion of a speaker.

One aspect of the present invention includes a learning unit that performs learning of a mathematical model for estimating the emotion of the estimation target based on an emotion time series that is a time series that depends on the emotion of the estimation target during utterance, and For the emotion time series vector indicating the time series, the emotion time series is divided in the time direction under the division conditions, which are conditions related to division of the emotion time series and are predetermined conditions according to the type of emotion time series. The mathematical model executes a common information acquisition process that applies the same mapping to two or more types of integrated vectors to which a semantic vector indicating a semantic interval that is an interval obtained as a result of the It is a learning device that is updated by the learning.

One aspect of the present invention includes an object acquisition unit that acquires an emotion time series that is a time series that depends on the emotion of the estimation target during utterance, and a mathematical model that estimates the emotion of the estimation target based on the emotion time series. a learning unit that performs learning, and for the emotion time series vector indicating the emotion time series, a categorization condition that is a condition related to categorization of the emotion time series and is a condition predetermined according to the type of the emotion time series. A common information acquisition process that applies the same mapping to two or more types of integrated vectors to which a semantic vector indicating a semantic interval is an interval obtained as a result of dividing an emotional time series in the temporal direction under The mathematical model executes, and the mapping is updated by the learning.The learned mathematical model obtained by the learning device allows the time series related to the utterances that depend on one's own emotions to be The estimation device includes an estimation unit that estimates the emotion of an estimation target that satisfies the condition that the emotion time series is the emotion time series acquired by the target acquisition unit.

One aspect of the present invention includes a learning step of learning a mathematical model for estimating the emotion of the estimation target based on an emotion time series that is a time series that depends on the emotion of the estimation target during utterance, For an emotion time series vector indicating an emotion time series, the emotion time series is plotted in the time direction under a segmentation condition that is a condition for segmenting the emotion time series and is a predetermined condition depending on the type of emotion time series. The mathematical model executes a common information acquisition process that applies the same mapping to two or more types of integrated vectors to which a semantic vector indicating a semantic interval, which is an interval obtained as a result of the division, is applied, and is a learning method that is updated by the learning.

One aspect of the present invention includes a target acquisition step of acquiring an emotion time series that is a time series that depends on the emotion of the estimation target during utterance, and a mathematical model that estimates the emotion of the estimation target based on the emotion time series. a learning unit that performs learning, and for the emotion time series vector indicating the emotion time series, a categorization condition that is a condition related to categorization of the emotion time series and is a condition predetermined according to the type of the emotion time series. A common information acquisition process that applies the same mapping to two or more types of integrated vectors to which a semantic vector indicating a semantic interval is an interval obtained as a result of dividing an emotional time series in the temporal direction under The mathematical model executes, and the mapping is updated by the learning.The learned mathematical model obtained by the learning device allows the time series related to the utterances that depend on one's own emotions to be This estimation method includes an estimating step of estimating the emotion of an estimation object that satisfies the condition that the emotion time series obtained in the object obtaining step is the emotion time series.

One aspect of the present invention is a program for causing a computer to function as the above learning device.

One aspect of the present invention is a program for causing a computer to function as the above estimation device.

According to the present invention, it is possible to improve the accuracy of estimating the emotion of the speaker.

FIG. 2 is an explanatory diagram illustrating an overview of a mathematical model in an embodiment. FIG. 1 is a diagram showing an example of a hardware configuration of a learning device according to an embodiment. The figure which shows an example of the structure of the control part with which the learning device of embodiment is provided. 1 is a flowchart illustrating an example of the flow of processing executed by the learning device according to the embodiment. FIG. 1 is a diagram showing an example of a hardware configuration of an estimation device according to an embodiment. The figure which shows an example of the structure of the control part with which the estimation device of embodiment is provided. 5 is a flowchart illustrating an example of the flow of processing executed by the estimation device according to the embodiment.

(Embodiment)
FIG. 1 is an explanatory diagram illustrating an overview of a mathematical model in an embodiment. More specifically, FIG. 1 is a diagram illustrating an overview of a mathematical model (hereinafter referred to as "emotion estimation model") for estimating the emotion of a speaker who is an estimation target. The emotion estimation model is updated through learning. Updated by learning means updated by a machine learning method.

More specifically, the emotion estimation model is a mathematical model that estimates an emotion to be estimated based on two or more types of emotion time series. The emotion time series is a time series related to utterances uttered by a speaker, and is a time series that depends on the speaker's emotion during the utterance.

The emotion time series is, for example, a time series of utterances of a speaker (hereinafter referred to as "utterance time series"). The emotional time series is, for example, a time series of sounds of speech indicated by the utterance time series (hereinafter referred to as "sound time series"). The emotion time series is, for example, a time series of video sounds of a speaker who is uttering the utterance indicated by the utterance time series (hereinafter referred to as "sound time series").

The emotion time series is, for example, a video (hereinafter referred to as "utterance video") that shows the speaker while uttering the utterance indicated by the utterance time series. In this way, the emotion time series consists of the time series of utterances indicated by the utterance time series, the time series related to the utterances indicated by the utterance time series, and the time series related to the speaker who is in the middle of uttering the utterances indicated by the utterance time series. Any time series may be used as long as it belongs to any one of .

For example, an utterance time series and a sound time series are different types of time series. Furthermore, the utterance time series and the utterance video are different types of time series. Both the sound time series and the speech video are different types of time series. Although any input time series may be used as long as two or more types of emotion time series are input to the emotion estimation model, it is preferable that at least an utterance time series is input.

FIG. 1 shows three emotional time series: an utterance time series, a sound time series, and a utterance video. In the emotion estimation model, vectorization acquisition processing is executed. The vectorization process is a process of acquiring a vector indicating an emotion time series (hereinafter referred to as an "emotion time series vector") for each emotion time series. Specifically, the emotion time series vector is a vector in which each element represents a sample of the time series. Therefore, in the example of FIG. 1, by executing the vectorization process, an emotion time series vector indicating the utterance time series, an emotion time series vector indicating the sound time series, and an emotion time series vector indicating the utterance video are obtained.

Next, the emotion estimation model performs meaning assignment processing. The meaning adding process is a process of adding a meaning vector to an emotion time series vector. The meaning vector is a vector indicating the meaning of each meaning interval. The meaning interval is a condition related to division of the emotion time series, and is an interval obtained as a result of dividing the emotion time series in the time direction under conditions predetermined according to the type of the emotion time series. Hereinafter, conditions related to classification of emotional time series and predetermined according to the type of emotional time series will be referred to as classification conditions.

Note that adding a vector to a vector means combining vectors. Combination of vectors is a process of generating a (N+M)-dimensional vector H3 from a vector H1, which is an N-dimensional vector, and a vector H2, which is an M-dimensional vector (N and M are integers of 1 or more). It is. The first to Nth elements of vector H3 are the first to Nth elements of vector H1, and the (N+1)th to (N+M)th elements of vector H3 are the first to Nth elements of vector H2, in order. These are elements up to the Mth element.

Hereinafter, a vector obtained by the meaning assignment process, which is a combination of an emotion time series vector and a meaning vector, will be referred to as an integrated vector.

In the example of FIG. 1, by executing the meaning adding process, an integrated vector corresponding to an emotion time series vector indicating an utterance time series, an integrated vector corresponding to an emotion time series vector indicating a sound time series, and an emotion indicating an utterance video are generated. Three types of integrated vectors are obtained: an integrated vector corresponding to a time series vector, and an integrated vector corresponding to a time series vector.

Note that the classification condition is, for example, in the case of an utterance time series, that only one word is included. If there is a sound time series, the division condition is, for example, that only one phrase from the start to the end of the sound is included. In the case of a speech video, the classification condition is, for example, that the video contains only one scene.

Therefore, the lengths of the semantic sections in the time direction differ from one semantic section to another, and are not necessarily the same. For example, in the case of an utterance time series, the length of the semantic interval is the length of the word, so if the utterance time series contains multiple words with different lengths, the utterance time series will There are multiple semantic intervals with different lengths.

In the meaning assignment process, a meaning vector is assigned based on information indicating meaning candidates and previously stored in a predetermined storage device (hereinafter referred to as a "semantic dictionary"). The meaning assignment process may be a technique using LMMS (Language Modeling Makes Sense) described in Reference 1 below, for example.

Reference 1: Daniel Loureiro, et al. Language Modeling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation, In Proc. ACL’19, 5682-5691

In addition, in the process of adding meaning to the utterance time series, for example, information indicating the meaning of each word in the utterance indicated by the utterance time series is attached to the emotion time series vector indicating the sound time series. In addition, in the process of adding meaning to the sound time series, for example, information on the pitch and magnitude of each sound indicated by the sound time series is assigned to the emotion time series vector indicating the sound time series. In addition, in the process of adding meaning to the utterance video, for example, information indicating the content of each scene of the video shown by the utterance video is added to the emotion time series vector representing the utterance video.

Next, in the emotion estimation model, common information acquisition processing is executed. The common information acquisition process is a process in which the same mapping is applied to each integrated vector regardless of the type of integrated vector. The mapping may be, for example, a mapping representing a binary operation with a predetermined vector. In this case, the predetermined vector and the definition of the binary operation are the same regardless of the type of integrated vector. The binary operation is, for example, a tensor product. The mapping may be a matrix, for example. When tensor product is used as a binary operation, each element is a tensor, such as a vector or a matrix. Therefore, in tensor product, all the elements of each element are multiplied together. Therefore, by using the tensor product, the relationship between two terms can be extracted with higher precision than when other binary operations are used.

The results of the execution of the common information acquisition process are input to the subsequent process. The result of execution of the common information acquisition process is the result of a binary operation. That is, the result of execution of the common information acquisition process is a mapping image. Specifically, the subsequent process is a process of estimating an emotion based on the result of execution of the common information acquisition process (hereinafter referred to as "emotion estimation post-processing"). Emotion estimation is performed, for example, by a sequence classification task based on the result of execution of common information acquisition processing.

After executing the common information acquisition process, the emotion estimation model executes emotion estimation post-processing. In this way, the emotion estimation model estimates the emotion of the estimation target.

<Effects of common information acquisition processing>
Here, the effects of the common information acquisition process will be explained. The common information acquisition process is a process in which the same mapping is applied to each integrated vector input as described above. Therefore, this is a process of acquiring information common to each integrated vector. Mathematically, the common information acquisition process is a process of obliquely projecting each integrated vector onto the same hyperplane in one vector space that includes each integrated vector. By the way, each integrated vector is a time series of the utterance indicated by the utterance time series, a time series related to the utterance indicated by the utterance time series, and a time series related to the speaker who is uttering the utterance indicated by the utterance time series. It is obtained from the time series belonging to either of the following.

Therefore, each integrated vector has the subject (topic) of the utterance indicated by the utterance time series as common information. Therefore, the information common to each integrated vector obtained by the common information acquisition process includes the theme. By updating the mapping by learning the emotion estimation model, the ratio of the information amount of the subject to the total information amount of the common information increases in the common information acquisition process. Even if it contains thematic information, if it is buried in a lot of other information, the influence of the topical information on the estimation results of the emotion estimation model will be small. That is, even if the subject information is included, if the subject information is buried in a lot of other information, the subject information will not have a significant effect on the estimation results of the emotion estimation model.

As mentioned above, the common information acquisition process is a process that increases the amount of information about the subject out of all the information used for estimation by the emotion estimation model. The influence of information on the estimation results of the emotion estimation model increases. As a result, the emotion estimation model can obtain estimation results under the significant effect of the information indicating the theme. Considering that the subject is important information that indicates the outline of the content of the utterance, an emotion estimation model that can also meaningfully use the subject information in estimation is a mathematical model that estimates emotions and requires common information acquisition processing. Estimation accuracy is higher than other mathematical models that do not run.

Note that the machine learning method may be, for example, a method using BERT (Bidirectional Encoder Representations from Transformers), a method using LSTM (Long short-term memory), or a method using CNN (Convolutional Neural Networks). ) may be used. In learning the emotion estimation model, pairs of correct data and two or more types of emotion time series are used as training data. Correct answer data is information indicating emotion. The loss function in learning is a function that indicates the difference from the correct data, and the emotion estimation model is updated so that the difference indicated by the loss function becomes smaller. Note that learning is executed until a predetermined end condition regarding learning (hereinafter referred to as "learning end condition") is satisfied. The learning end condition is, for example, a condition that learning has been completed a predetermined number of times. The learning end condition may be, for example, a condition that the estimation accuracy of the emotion estimation model is equal to or higher than a predetermined accuracy.

FIG. 2 is a diagram showing an example of the hardware configuration of the learning device 1 in the embodiment. The learning device 1 includes a control unit 11 including a processor 91 such as a CPU (Central Processing Unit) and a memory 92 connected via a bus, and executes a program. The learning device 1 functions as a device including a control section 11, an input section 12, a communication section 13, a storage section 14, and an output section 15 by executing a program.

More specifically, processor 91 reads a program stored in storage unit 14 and stores the read program in memory 92 . When the processor 91 executes the program stored in the memory 92, the learning device 1 functions as a device including a control section 11, an input section 12, a communication section 13, a storage section 14, and an output section 15.

The control unit 11 controls the operations of various functional units included in the learning device 1. The control unit 11 performs learning of an emotion estimation model, for example. The control unit 11 controls the operation of the output unit 15, for example. The control unit 11 records, for example, various information generated by learning the emotion estimation model in the storage unit 14.

The input unit 12 includes input devices such as a mouse, a keyboard, and a touch panel. The input unit 12 may be configured as an interface that connects these input devices to the learning device 1. The input unit 12 receives input of various information to the learning device 1. For example, an emotional time series is input to the input unit 12 .

The communication unit 13 includes a communication interface for connecting the learning device 1 to an external device. The communication unit 13 communicates with an external device via wire or wireless. The external device is, for example, a device that is a source of training data. The communication unit 13 acquires training data by communicating with a device that is a transmission source of the training data. The external device is, for example, the estimation device 2 described later.

The storage unit 14 is configured using a computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 14 stores various information regarding the learning device 1. The storage unit 14 stores information input via the input unit 12 or the communication unit 13, for example. The storage unit 14 stores various information generated by learning the emotion estimation model, for example. The storage unit 14 stores emotion estimation models in advance. Note that storing a mathematical model means storing a computer program that describes the mathematical model. The storage unit 14 may store the obtained learned emotion estimation model.

The output unit 15 outputs various information. The output unit 15 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, and an organic EL (Electro-Luminescence) display. The output unit 15 may be configured as an interface that connects these display devices to the learning device 1. The output unit 15 outputs, for example, information input to the input unit 12. The output unit 15 may display, for example, the result of executing the emotion estimation model.

FIG. 3 is a diagram showing an example of the configuration of the control unit 11 in the embodiment. The control unit 11 includes a data acquisition unit 110, a learning unit 120, a storage control unit 130, a communication control unit 140, and an output control unit 150.

The data acquisition unit 110 acquires training data. That is, the data acquisition unit 110 acquires correct answer data and emotion time series. The learning unit 120 uses the training data obtained by the data acquisition unit 110 to update the emotion estimation model until the learning end condition is satisfied. That is, the learning unit 120 obtains a learned emotion estimation model using the correct data and two or more types of emotion time series. The learned emotion estimation model is the emotion estimation model at the time when the learning end condition is satisfied.

The storage control unit 130 records various information in the storage unit 14. The communication control unit 140 controls the operation of the communication unit 13. The output control section 150 controls the operation of the output section 15.

FIG. 4 is a flowchart showing an example of the flow of processing executed by the learning device 1 in the embodiment. The data acquisition unit 110 acquires training data including correct data and two or more types of emotion time series (step S101). Next, the learning unit 120 estimates the emotion to be estimated by executing the emotion estimation model on the two or more types of emotion time series obtained in step S101 (step S102). Since the emotion estimation model is executed in step S102, vectorization processing, meaning assignment processing, common information acquisition processing, and emotion estimation post-processing are executed.

Next to step S102, the learning unit 120 uses the estimation result of step S102 to update the emotion estimation model based on the difference from the correct data obtained in step S101 (step S103). Next, the learning unit 120 determines whether the learning end condition is satisfied (step S104). If the learning end condition is satisfied (step S104: YES), the process ends. On the other hand, if the learning end condition is not satisfied (step S104: NO), the process returns to step S101.

The emotion estimation model at the time when the learning end condition is satisfied is the learned emotion estimation model, and the obtained learned emotion estimation model is used to estimate the emotion of the estimation target by the estimation device 2 shown in FIG. 7 below. used. The learned emotion estimation model may be placed under the control of the estimation device 2 by any method as long as the estimation device 2 becomes executable. The learned emotion estimation model can be executed by the estimation device 2 by being transmitted from the learning device 1 to the estimation device 2 via communication, for example, after the learning end condition is satisfied.

FIG. 7 is a diagram showing an example of the hardware configuration of the estimation device 2 in the embodiment. The estimation device 2 includes a control unit 21 including a processor 93 such as a CPU and a memory 94 connected via a bus, and executes a program. The estimation device 2 functions as a device including a control section 21, an input section 22, a communication section 23, a storage section 24, and an output section 25 by executing a program.

More specifically, the processor 93 reads a program stored in the storage unit 24 and stores the read program in the memory 94. When the processor 93 executes the program stored in the memory 94, the estimation device 2 functions as a device including the control section 21, the input section 22, the communication section 23, the storage section 24, and the output section 25.

The control unit 21 controls the operations of various functional units included in the estimation device 2. The control unit 21 executes, for example, a learned emotion estimation model. The control unit 21 controls the operation of the output unit 25, for example. The control unit 21 records, for example, various information generated by executing the learned emotion estimation model in the storage unit 24.

The input unit 22 includes input devices such as a mouse, a keyboard, and a touch panel. The input unit 22 may be configured as an interface that connects these input devices to the estimation device 2. The input unit 22 receives input of various information to the estimation device 2 .

The communication unit 23 includes a communication interface for connecting the estimation device 2 to an external device. The communication unit 23 communicates with an external device via wire or wireless. The external device is, for example, a device that sends the emotion time series. The external device is, for example, the learning device 1. The communication unit 23 acquires the learned emotion estimation model through communication with the learning device 1 . Note that the emotional time series does not necessarily need to be input to the communication unit 23 and may be input to the input unit 22.

The storage unit 24 is configured using a computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 24 stores various information regarding the estimation device 2. The storage unit 24 stores information input via the input unit 22 or the communication unit 23, for example. The storage unit 24 stores, for example, various information generated by executing a learned emotion estimation model. The storage unit 24 stores the learned emotion estimation model.

The output unit 25 outputs various information. The output section 25 is configured to include a display device such as a CRT display, a liquid crystal display, an organic EL display, or the like. The output unit 25 may be configured as an interface that connects these display devices to the estimation device 2. The output unit 25 outputs the information input to the input unit 22, for example. The output unit 25 may display, for example, the execution results of the learned emotion estimation model.

FIG. 6 is a diagram showing an example of the configuration of the control unit 21 in the embodiment. The control unit 21 includes a target acquisition unit 210, an estimation unit 220, a storage control unit 230, a communication control unit 240, and an output control unit 250. The target acquisition unit 210 acquires the emotion time series input to the input unit 22 or the communication unit 23.

The estimation unit 220 executes the learned emotion estimation model on the emotion time series acquired by the target acquisition unit 210. The estimation unit 220 estimates the speaker's emotion corresponding to the emotion time series acquired by the target acquisition unit 210 by executing the learned emotion estimation model.

As described above, the emotional time series is a time series related to the utterances uttered by the speaker, and is a time series that depends on the speaker's emotion during the utterance. Therefore, the speaker corresponding to the emotion time series acquired by the target acquisition unit 210 means that the time series related to the utterances that the speaker has uttered and that depends on his/her own emotion is the emotion time series acquired by the target acquisition unit 210. This is a speaker who satisfies the condition (hereinafter referred to as "corresponding speaker condition"). Therefore, the speaker who satisfies the corresponding speaker condition is the target of emotion estimation by the estimation unit 220.

The storage control unit 230 records various information in the storage unit 24. The communication control unit 240 controls the operation of the communication unit 23. The output control section 250 controls the operation of the output section 25.

FIG. 7 is a flowchart showing an example of the flow of processing executed by the estimation device 2 in the embodiment. The target acquisition unit 210 acquires two or more types of emotion time series input to the input unit 22 or the communication unit 23 (step S201). Next, the estimation unit 220 estimates the speaker's emotion corresponding to the emotion time series acquired by the target acquisition unit 210 by executing the learned emotion estimation model (step S202). Next, the output control unit 250 controls the operation of the output unit 25 to cause the output unit 25 to output the emotion estimated in step S202 (step S203).

The learning device 1 in the embodiment configured as described above updates the emotion estimation model that executes the common information acquisition process through learning. Therefore, as described in the explanation of the effect of the common information acquisition process above, the emotion estimation model obtains the estimation result by receiving a significant effect from the information indicating the theme. Therefore, the learning device 1 can improve the accuracy of estimating the speaker's emotion.

Furthermore, the estimation device 2 in the embodiment configured as described above uses the learned emotion estimation model obtained by the learning device 1 to estimate the emotion of the estimation target. Therefore, the estimation device 2 can improve the accuracy of estimating the emotion of the estimation target.

(Modified example)
The emotion estimation post-processing may be performed on the result of the dimension reduction process performed on the result of the common information acquisition process, instead of the result itself of the execution of the common information acquisition process. That is, dimension reduction processing may be performed between the common information acquisition processing and the emotional information estimation post-processing. Dimension reduction processing is processing that reduces the dimension of a tensor when the result of execution of the common information acquisition processing is a tensor such as a vector. The dimension of the tensor obtained by executing the dimension reduction process is a dimension that can be processed by the emotional information estimation post-processing.

The emotion estimation post-processing may be performed on the result of time-resolved embedding processing performed on the result of the common information acquisition process, instead of the result itself of the execution of the common information acquisition process. Time-resolved embedding processing is applicable when the result of execution of common information acquisition processing is a vector. In the meaning assignment process, a meaning is assigned to each meaning interval as described above. Therefore, the vector obtained by the meaning assignment process (ie, the integrated vector) has information on the length of time of the meaning interval. Since the common information acquisition process is also executed for the integrated vector, information indicating the length of time of each semantic interval (hereinafter referred to as "semantic interval length information") is used in the execution of the common information acquisition process. It also has results.

However, the emotion estimation post-processing may be a process of performing calculations based on information that the lengths of the semantic intervals are the same. Therefore, in the time-resolved embedding process, the length of each semantic interval indicated by the semantic interval length information is the same regardless of the semantic interval, so that the vector obtained as a result of the common information acquisition process is Convert.

Specifically, the following processing is executed.

The left side of equation (1) means an embedding tensor divided into words in the utterance of each modality. The right-hand side of equation (1) means that the left-hand tensor consists of modality embedding vectors. At this time, m means {l, a, v}. That is, m means a set of l, a, and v. m is placed as a general term for subscripts for modalities, l means language modality, a means audio modality, and v means video modality. Each element on the right side of equation (1) means an embedding vector for each word. n _u means the word length of the utterance. The left side of equation (2) means the feature quantity tensor for the entire utterance of the time series modality generated by the time-resolved embedding process. “conv1D” on the right side of equation (2) means one-dimensional convolution processing, and indicates processing of convolving all data points in the time series to the number of data points of the word length in the utterance. M' on the right side of equation (2) means a time-series feature tensor of each modality in the utterance. M is a pair of A and V. A means audio modality and V means video modality, respectively. The left side of equation (3) means the feature tensor of the entire utterance. “concat” on the right side of equation (3) means matrix combination processing. A ^u in equation (3) means the audio modality component of the M ^u tensor in equation (2). W ^a in equation (3) means a weight for A ^u . V ^u in equation (3) means the video modality component of the M ^u tensor in equation (2). W ^v in equation (3) means a weight for V ^u . means. m in Equation (3) also means a set of language, audio, and video modalities; that is, this equation indicates that the operation on the right side is performed on the word segmentation embedding tensor of each modality. In this case, _E'm on the right side is the same as the left side of equation (1), that is, the word division embedding tensor described above.

In this way, the time-resolved embedding process is a process of changing the content of the semantic interval length information of an input vector indicating a time series and having the semantic interval length information, This process converts the length of each semantic section indicated by the information into the same length regardless of the semantic section. The length of each semantic interval after conversion is the length obtained by dividing the length in the time direction of the time series indicated by the input vector by the average value of the length of each utterance included in the utterance time series.

Note that the time-resolved embedding process only needs to be executed before the emotion estimation post-processing, and may be executed, for example, on the result of the dimensionality reduction process.

Note that the learning device 1 may be implemented using a plurality of information processing devices communicatively connected via a network. In this case, each functional unit included in the learning device 1 may be distributed and implemented in a plurality of information processing devices.

Note that the estimation device 2 may be implemented using a plurality of information processing devices communicably connected via a network. In this case, each functional unit included in the estimation device 2 may be distributed and implemented in a plurality of information processing devices.

All or part of each function of the learning device 1 and the estimation device 2 may be implemented using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). It may also be realized by The program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, magneto-optical disk, ROM, or CD-ROM, or a storage device such as a hard disk built into a computer system. The program may be transmitted via a telecommunications line.

Note that the emotion time series acquired by the target acquisition unit 210 is an example of target information. The sentence indicated by the emotion time series acquired by the target acquisition unit 210 is an example of a processing target.

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configuration is not limited to these embodiments, and includes designs within the scope of the gist of the present invention.

(Additional note 1)
a learning unit that performs learning of a mathematical model that estimates the emotion of the estimation target based on an emotion time series that is a time series that depends on the emotion of the estimation target during utterance;
Equipped with
With respect to the emotion time series vector indicating the emotion time series, the emotion time series is determined in the time direction under a classification condition that is a condition regarding the division of the emotion time series and is a predetermined condition depending on the type of emotion time series. The mathematical model executes a common information acquisition process that applies the same mapping to two or more types of integrated vectors to which a semantic vector indicating a semantic interval that is an interval obtained as a result of segmentation is applied,
the mapping is updated by the learning;
learning device.

(Additional note 2)
one of the emotion time series is a time series of the utterances of the estimation target;
The learning device described in Appendix 1.

(Additional note 3)
Another one of the emotional time series is a time series of the sounds of the utterance,
The learning device described in Appendix 2.

(Additional note 4)
Another one of the emotion time series is a video showing the state of the estimation target while uttering the utterance.
The learning device according to appendix 2 or 3.

(Appendix 5)
After the common information acquisition process, the mathematical model calculates the semantic interval length of the input vector indicating a time series and having semantic interval length information that is information indicating the time length of the semantic interval. a time-resolved embedding process that converts the length of each semantic interval indicated by the semantic interval length information to the same length regardless of the semantic interval;
The length of each semantic interval after conversion by the time-resolved embedding process is the time series indicated by the vector input by the average value of the length of each utterance included in the utterance time series that is the time series of the utterances to be estimated. is the length divided by the length in the time direction,
The learning device according to any one of Supplementary Notes 1 to 4.

(Appendix 6)
an object acquisition unit that acquires an emotion time series that is a time series that depends on the emotion of the estimation target during utterance;
a learning unit that learns a mathematical model for estimating the emotion of the estimation target based on the emotion time series, and for the emotion time series vector indicative of the emotion time series, conditions regarding classification of the emotion time series are determined. Two or more types of semantic vectors that indicate semantic intervals, which are intervals obtained as a result of dividing an emotion time series in the temporal direction under division conditions that are predetermined according to the type of time series. The mathematical model executes a common information acquisition process that applies the same mapping to the integrated vector, and the mapping is updated by the learning. an estimation unit that estimates the emotion of an estimation target that satisfies the condition that the time series that is a time series related to the emotion and that depends on one's own emotion is the emotion time series acquired by the target acquisition unit;
An estimation device comprising:

(Appendix 7)
a learning step of learning a mathematical model for estimating the emotion of the estimation target based on an emotion time series that is a time series that depends on the emotion of the estimation target during utterance;
has
With respect to the emotion time series vector indicating the emotion time series, the emotion time series is determined in the time direction under a classification condition that is a condition regarding the division of the emotion time series and is a predetermined condition depending on the type of emotion time series. The mathematical model executes a common information acquisition process that applies the same mapping to two or more types of integrated vectors to which a semantic vector indicating a semantic interval that is an interval obtained as a result of segmentation is applied,
the mapping is updated by the learning;
How to learn.

(Appendix 8)
a target acquisition step of acquiring an emotion time series that is a time series that depends on the emotion of the estimation target during the utterance;
a learning unit that learns a mathematical model for estimating the emotion of the estimation target based on the emotion time series, and for the emotion time series vector indicative of the emotion time series, conditions regarding classification of the emotion time series are determined. Two or more types of semantic vectors that indicate semantic intervals, which are intervals obtained as a result of dividing an emotion time series in the temporal direction under division conditions that are predetermined according to the type of time series. The mathematical model executes a common information acquisition process that applies the same mapping to the integrated vector, and the mapping is updated by the learning. an estimation step of estimating the emotion of an estimation target that satisfies the condition that the time series that is a time series related to the emotion and that depends on one's own emotion is the emotion time series acquired in the target acquisition step;
An estimation method with

(Appendix 9)
A program for causing a computer to function as a learning device according to any one of Supplementary Notes 1 to 5.

(Appendix 10)
A program for causing a computer to function as the estimation device according to appendix 6.

DESCRIPTION OF SYMBOLS 1...Learning device, 2...Estimation device, 11...Control unit, 12...Input unit, 13...Communication unit, 14...Storage unit, 15...Output unit, 110...Data acquisition unit, 120...Learning unit, 130...Storage control 140... Communication control section, 150... Output control section, 21... Control section, 22... Input section, 23... Communication section, 24... Storage section, 25... Output section, 210... Target acquisition section, 220... Estimation section, 230...Storage control unit, 240...Communication control unit, 250...Output control unit, 91...Processor, 92...Memory, 93...Processor, 94...Memory

Claims (10)

a learning unit that performs learning of a mathematical model that estimates the emotion of the estimation target based on an emotion time series that is a time series that depends on the emotion of the estimation target during utterance;
Equipped with
With respect to the emotion time series vector indicating the emotion time series, the emotion time series is determined in the time direction under a classification condition that is a condition regarding the division of the emotion time series and is a predetermined condition depending on the type of emotion time series. The mathematical model executes a common information acquisition process that applies the same mapping to two or more types of integrated vectors to which a semantic vector indicating a semantic interval that is an interval obtained as a result of segmentation is applied,
the mapping is updated by the learning;
learning device. one of the emotion time series is a time series of the utterances of the estimation target;
The learning device according to claim 1. Another one of the emotional time series is a time series of the sounds of the utterance,
The learning device according to claim 2. Another one of the emotion time series is a video showing the state of the estimation target while uttering the utterance.
The learning device according to claim 2. After the common information acquisition process, the mathematical model calculates the semantic interval length of the input vector indicating a time series and having semantic interval length information that is information indicating the time length of the semantic interval. a time-resolved embedding process that converts the length of each semantic interval indicated by the semantic interval length information to the same length regardless of the semantic interval;
The length of each semantic interval after conversion by the time-resolved embedding process is the time series indicated by the vector input by the average value of the length of each utterance included in the utterance time series that is the time series of the utterances to be estimated. is the length divided by the length in the time direction,
The learning device according to any one of claims 1 to 4. an object acquisition unit that acquires an emotion time series that is a time series that depends on the emotion of the estimation target during utterance;
a learning unit that learns a mathematical model for estimating the emotion of the estimation target based on the emotion time series, and for the emotion time series vector indicative of the emotion time series, conditions regarding classification of the emotion time series are determined. Two or more types of semantic vectors that indicate semantic intervals, which are intervals obtained as a result of dividing an emotion time series in the temporal direction under division conditions that are predetermined according to the type of time series. The mathematical model executes a common information acquisition process that applies the same mapping to the integrated vector, and the mapping is updated by the learning. an estimation unit that estimates the emotion of an estimation target that satisfies the condition that the time series that is a time series related to the emotion and that depends on one's own emotion is the emotion time series acquired by the target acquisition unit;
An estimation device comprising: a learning step of learning a mathematical model for estimating the emotion of the estimation target based on an emotion time series that is a time series that depends on the emotion of the estimation target during utterance;
has
With respect to the emotion time series vector indicating the emotion time series, the emotion time series is determined in the time direction under a classification condition that is a condition regarding the division of the emotion time series and is a predetermined condition depending on the type of emotion time series. The mathematical model executes a common information acquisition process that applies the same mapping to two or more types of integrated vectors to which a semantic vector indicating a semantic interval that is an interval obtained as a result of segmentation is applied,
the mapping is updated by the learning;
How to learn. a target acquisition step of acquiring an emotion time series that is a time series that depends on the emotion of the estimation target during the utterance;
a learning unit that learns a mathematical model for estimating the emotion of the estimation target based on the emotion time series, and for the emotion time series vector indicative of the emotion time series, conditions regarding classification of the emotion time series are determined. Two or more types of semantic vectors that indicate semantic intervals, which are intervals obtained as a result of dividing an emotion time series in the temporal direction under division conditions that are predetermined according to the type of time series. The mathematical model executes a common information acquisition process that applies the same mapping to the integrated vector, and the mapping is updated by the learning. an estimation step of estimating the emotion of an estimation target that satisfies the condition that the time series that is a time series related to the emotion and that depends on one's own emotion is the emotion time series acquired in the target acquisition step;
An estimation method with A program for causing a computer to function as the learning device according to claim 1. A program for causing a computer to function as the estimation device according to claim 6.

Images