JP7279287B2 - Emotion estimation device and emotion estimation system - Google Patents

Description

The present invention relates to an emotion estimation device and an emotion estimation system.

In recent years, services for estimating emotions such as joy, anger and sadness have become popular. For example, Patent Literature 1 discloses an emotion estimating device that estimates an emotion of a user based on voice information indicating the voice of the user. This emotion estimating device uses voice information input multiple times by a certain user to perform voice recognition. Values and standard deviations are pre-calculated. Then, this emotion estimation apparatus normalizes the feature amount of speech information input when estimating the emotion of the user by using a pre-calculated average value and standard deviation, and converts the feature amount into a plurality of normalized feature amounts. Based on this, the user's feelings are estimated.

JP-A-2006-259641

However, when the above-described conventional technology is applied to a device that estimates the emotion of a user using a learning model that has already learned the relationship between a plurality of feature values and emotions based on voice information, the learning model is applied to each user. I had to prepare for When using a general-purpose learning model that has been trained using speech information of a large number of users as training data, the difference between the average speech characteristics of a large number of users and the characteristics of the user's speech is not absorbed. could not be estimated accurately.

An emotion estimating apparatus according to a preferred aspect of the present invention is a device that has learned the relationship between a plurality of feature amounts corresponding to human speech and the intensity of each of a plurality of emotions held by a person who has uttered the speech. inputting a plurality of feature amounts based on speech information indicating a user's speech to a learning model, and obtaining speech emotion information including a speech evaluation value indicating the intensity of each of the plurality of emotions held by the user from the learning model; a correction unit for generating corrected emotional information obtained by correcting the voice emotional information using correction information based on the characteristics of the user's voice; an estimation unit for estimating one or more emotions that the user has from among them.

An emotion estimating system according to a preferred aspect of the present invention is an emotion estimating system comprising a server device and a terminal device capable of communicating with the server device, wherein the server device includes a sound indicating a sound including a user's voice. a first communication device that receives information; a noise removal unit that removes noise from the sound indicated by the sound information to generate voice information indicating the user's voice; and a plurality of feature quantities corresponding to the human voice. inputting a plurality of features based on the voice information to a learning model that has already learned the relationship between the intensity of each of the plurality of emotions felt by the person who uttered the voice and the intensity of each of the plurality of emotions held by the person who emitted the voice A speech evaluation unit that acquires speech emotion information including a speech evaluation value indicating the strength of each of the plurality of emotions from the learning model, and a speech recognition process that recognizes the content of speech uttered by a person are performed on the sound information. a character evaluation unit for generating character emotion information including a character evaluation value indicating the intensity of each of the plurality of emotions held by the user, based on the recognized character string indicating the recognition result of the speech recognition processing; wherein the first communication device transmits the text emotion information and the voice emotion information to the terminal device, and the terminal device comprises a sound collection device that collects sound including the user's voice; a second communication device that transmits the sound information output by the sound collector to the server device and receives the character emotion information and the voice emotion information from the server device; and correction information based on the features of the user's voice. and an estimation unit for estimating one or more emotions of the user based on the corrected emotion information and the text emotion information. .

ADVANTAGE OF THE INVENTION According to this invention, even if it is a case where the learning model which has been trained by using the audio|speech information of several people as teacher data is used, the emotion which a user has can be estimated with high accuracy.

3 is a diagram showing an overview of functions of the user device 1; FIG. 2 is a block diagram showing the configuration of the user device 1 according to the first embodiment; FIG. 4 is a diagram showing an example of contents stored in analysis dictionary information 31. FIG. 4 is a diagram showing an example of contents stored in emotion classification information 33. FIG. 3 is a diagram showing an overview of functions of the user device 1; FIG. FIG. 4 is a diagram showing a flowchart showing the operation of the user device 1; FIG. 2 is a block diagram showing a user device 1a according to a second embodiment; FIG. The figure which shows the outline|summary of the function of the user apparatus 1a concerning 2nd Embodiment. The figure which shows the flowchart which shows the operation|movement of the user apparatus 1a at the time of calibration mode. The figure which shows the whole structure of emotion estimation system SYS. FIG. 3 is a block diagram showing the configuration of the user device 1b; 2 is a block diagram showing the configuration of the server device 10; FIG. The figure which shows the outline|summary of the function of emotion estimation system SYS. FIG. 11 is a diagram showing an overview of a function of adjusting correction information CI for a non-calibration user; The figure which shows the flowchart which shows the operation|movement of the emotion estimation system SYS in emotion estimation mode (part 1). The figure which shows the flowchart which shows the operation|movement of the emotion estimation system SYS in emotion estimation mode (part 2). The figure which shows the whole structure of emotion estimation system SYSc. FIG. 2 is a block diagram showing the configuration of a server device 10C; The figure which shows the outline|summary of the function of emotion estimation system SYSc. FIG. 10 is a diagram showing an overview of a function of adjusting parameter information TI for a non-calibration user; The figure which shows the flowchart which shows the operation|movement of the emotion estimation system SYSc in emotion estimation mode. The figure which shows the whole structure of emotion estimation system SYSd. FIG. 3 is a block diagram showing the configuration of the user device 1d; FIG. 2 is a block diagram showing the configuration of a server device 10D; The figure which shows the outline|summary of the function of emotion estimation system SYSd. The figure which shows the whole structure of emotion estimation system SYSe. FIG. 2 is a block diagram showing the configuration of a user device 1e; The figure which shows the outline|summary of the function of emotion estimation system SYSe. The figure which shows the whole structure of emotion estimation system SYSf. FIG. 2 is a block diagram showing the configuration of the user device 1f; FIG. 2 is a block diagram showing the configuration of a server device 10F; The figure which shows the outline|summary of the function of the 1st emotion estimation part 25f and the 2nd emotion estimation part 25F. The figure which shows the outline|summary of the function of the user apparatus 1g in a 1st modification.

1. 1. First Embodiment FIG. 1 is a diagram showing an overview of functions of a user device 1. As shown in FIG. The user device 1 is assumed to be a smart phone. The user device 1 is an example of an "emotion estimation device". However, any information processing device can be adopted as the user device 1. For example, it may be a terminal-type information device such as a personal computer, or a portable device such as a notebook computer, a wearable terminal, and a tablet terminal. information terminal.

The user device 1 has a function of transmitting a recognized character string obtained by performing voice recognition processing on sound information indicating a sound including the voice of the user U possessing the user device 1 to a device used by another person. Alternatively, it has a function of emitting a sound indicating the recognized character string so that others located near the user U can hear it. Further, the user device 1 estimates the emotion that the user U has based on the voice of the user U, and adds a figure corresponding to the estimated emotion to the recognized character string, or adds the figure corresponding to the estimated emotion to the recognized character string. Emotional expressions necessary for communication can be added by emitting a sound indicating a recognition character string with an intonation corresponding to the character string.
In the example of FIG. 1, the user U utters "Hello" and the user device 1 adds a graphic PI corresponding to the estimated emotion.

FIG. 2 is a block diagram showing the configuration of the user device 1 according to the first embodiment. The user device 1 is implemented by a computer system including a processing device 2 , a storage device 3 , a display device 4 , an operation device 5 , a communication device 6 , a sound emitting device 7 and a sound collecting device 8 . Each element of the user device 1 is interconnected by a bus or buses 9 for communicating information. Note that the term "apparatus" in this specification may be replaced with another term such as a circuit, a device, or a unit. Also, each element of the user device 1 may be composed of one or more devices, and some elements of the user device 1 may be omitted.

The processing device 2 is a processor that controls the entire user device 1, and is composed of, for example, a single chip or multiple chips. The processing device 2 is composed of, for example, a central processing unit (CPU) including interfaces with peripheral devices, arithmetic devices, registers, and the like. Some or all of the functions of the processing device 2 are realized by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array). may The processing device 2 executes various processes in parallel or sequentially.

The storage device 3 is a recording medium readable by the processing device 2, and stores a plurality of programs including a control program PR executed by the processing device 2, analysis dictionary information 31, emotion classification information 33, and a learning model LM. do. The storage device 3 is composed of, for example, one or more types of storage circuits such as ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), and RAM (Random Access Memory).

FIG. 3 is a diagram showing an example of contents stored in the analysis dictionary information 31. As shown in FIG. The analysis dictionary information 31 is information in which the part of speech, the sub-classification of the part of speech, and the root form information are associated with each other for each morpheme. A morpheme is a string of the smallest units of meaningful expressions. A part of speech is a type of word classified by grammatical properties, such as nouns, verbs, and adjectives. The part-of-speech fine classification is an item obtained by further finely classifying the part of speech. The original form information is a character string indicating the original form of the word if the morpheme is a word that is used, and is the same character string as the morpheme if the morpheme is a word that is not used.

FIG. 4 is a diagram showing an example of contents stored in the emotion classification information 33. As shown in FIG. The emotion classification information 33 is information in which character strings are classified into joy, anger, sadness, and normal. A character string registered in the emotion classification information 33 represents any emotion of joy, anger, sadness, or normal. In the example of FIG. 4, the character string group 331 classified as joy includes "happy", "pass", "win", and "win". Similarly, the character string group 332 classified as anger includes "frustration", "furious", and the like. Similarly, the character string group 333 classified as sadness includes “sad”, “defeat”, and the like. Similarly, the character string group 334 classified as normal includes "safety" and the like.

Returning to FIG. The learning model LM has already learned the relationship between a plurality of feature quantities corresponding to human speech and the intensity of each of a plurality of emotions.

The display device 4 displays various images under the control of the processing device 2 . For example, various display panels such as a liquid crystal display panel or an organic EL (Electro Luminescence) display panel are preferably used as the display device 4 .

The operation device 5 is a device for inputting information used by the user device 1 . The operation device 5 receives an operation by the user U. Specifically, the operation device 5 receives an operation for inputting codes such as numbers and letters, and an operation for selecting an icon displayed by the display device 4 . For example, a touch panel that detects contact with the display surface of the display device 4 is suitable as the operation device 5 . Note that the operating device 5 may include an operator that can be operated by the user. The manipulator is, for example, a touch pen.

The communication device 6 is hardware (transmitting/receiving device) for communicating with other devices via a network. The communication device 6 is also called a network device, a network controller, a network card, a communication module, etc., for example.

The sound emitting device 7 is composed of, for example, a speaker, and emits sound under the control of the processing device 2 . The sound collecting device 8 is composed of, for example, a microphone and an AD converter, and collects sounds including the voice of the user U under the control of the processing device 2 . A microphone converts the collected sound into an electrical signal. The AD converter AD-converts the electric signal converted by the microphone into sound information SI shown in FIG. The sound indicated by the sound information SI may include noise emitted from the speaker's surroundings in addition to the speaker's voice.

1.1. Functions of First Embodiment The processing device 2 functions as an acquisition unit 21 , an emotion estimation unit 25 and an output unit 26 by reading and executing the control program PR from the storage device 3 .
Functions realized by the processing device 2 will be described with reference to FIG.

FIG. 5 is a diagram showing an overview of the functions of the user device 1. As shown in FIG. The acquisition unit 21 acquires sound information SI output by the sound collector 8 that collects sounds including the user's U voice. The emotion estimation unit 25 estimates one or more emotions that the user U has from among the plurality of emotions that the user U has. In the first embodiment, it is assumed that the user U has four emotions: joy, anger, sadness, and normality. Below, joy, anger, sadness, and normal are examples of multiple emotions.

The emotion estimation unit 25 includes a noise removal unit 251 , a voice evaluation unit 252 , a correction unit 253 , a character evaluation unit 256 and an estimation unit 258 .

The noise removal unit 251 removes noise from the sound indicated by the sound information SI to generate voice information VI. For example, a first parameter P1 and a second parameter P2 are provided to the noise removal unit 251 . The first parameter P1 designates a frequency band that is regarded as noise. The second parameter P2 specifies the magnitude of the amplitude component regarded as noise. The noise removal unit 251 executes the first to fourth processes. In the first process, an amplitude component is calculated for each of a plurality of frequency bands by performing a fast Fourier transform process on the sound information SI. In the second process, the amplitude component of the frequency band specified by the first parameter P1 is reduced. The frequency of human speech is generally between 100 Hz and 2000 Hz. The first parameter P1 specifies the lower frequency limit and the upper frequency limit. Therefore, the noise removal unit 251 uses the first parameter P1 to reduce the amplitude component in the frequency band below the lower limit frequency and reduce the amplitude component in the frequency band above the lower limit frequency. The third process reduces amplitude components that are equal to or less than the magnitude specified by the second parameter P2. In the fourth process, an inverse Fourier transform process is applied to the result of the third process to generate audio information VI. The voice information VI indicates the voice of the user U from which environmental noise and the like are removed from the sound information SI.

The speech evaluation unit 252 inputs a plurality of feature amounts based on the speech information VI to the learning model LM, and outputs speech emotion information VE including a speech evaluation value x indicating the intensity of each of a plurality of emotions from the learning model LM. get.
The learning model LM has already learned the relationship between a plurality of feature quantities corresponding to human speech and the intensity of each of a plurality of emotions held by the person who uttered the speech for a plurality of people. The learning model LM learns a lot of teacher data in the process of learning. The teacher data is given as a set of a plurality of feature values, which are input data, and the intensity for each of a plurality of emotions, which are label data. Also, teacher data is generated based on the voice information VI of a large number of users. In other words, the learning model LM is a generic model that is not tailored to a specific individual.
The plurality of feature quantities are sound feature quantities, such as 12-dimensional MFCC (Mel-Frequency Cepstrum Coefficients), loudness, fundamental frequency (F0), speech probability, zero-crossing rate, HNR (Harmonics-to-Noise-Ratio ), and their first derivative, MFCC, and second derivative of loudness, for a total of 47 pieces. Loudness is the loudness of sound and indicates the intensity of sound sensed by human hearing. The speech probability indicates the probability that speech is included in the sound indicated by the speech information VI. The zero-crossing rate is the number of times the sound pressure is zero.
The speech evaluation unit 252 generates a plurality of feature amounts by performing processing for extracting sound features from the speech information VI.

The voice emotion information VE includes a joyful voice evaluation value x1, an anger voice evaluation value x2, a sadness voice evaluation value x3, and a normal voice evaluation value x4. The audio evaluation value x is a real number of 0 or more. In the following description, when distinguishing elements of the same type, reference numerals are used such as a voice evaluation value of joy x1 and a voice evaluation value of anger x2. On the other hand, when the elements of the same type are not distinguished, only the common number among the reference codes is used, like the speech evaluation value x.

The correction unit 253 generates corrected emotion information CVE by correcting the voice emotion information VE using correction information CI based on the features of the user's U voice. The correction information CI includes, for example, a coefficient k1 for correcting the joyful voice evaluation value x1, a coefficient k2 for correcting the anger voice evaluation value x2, a coefficient k3 for correcting the sadness voice evaluation value x3, and a normal voice evaluation value. It contains a coefficient k4 that corrects x4. k1 to k4 are real numbers of 0 or more. The corrected emotion information CVE includes a joyful voice evaluation value X1, an anger voice evaluation value X2, a sadness voice evaluation value X3, and a normal voice evaluation value X4. The correction unit 253 generates corrected emotion information CVE, for example, according to the following formula.

X1=x1×k1
X2=x2×k2
X3=x3×k3
X4=x4×k4

There are, for example, the following two modes of generating the correction information CI. In the first mode, the user U speaks toward the sound collector 8 in a normal state. The processing device 2 generates coefficients k1 to k4 by extracting a plurality of feature amounts from the speech information VI corresponding to the utterance and comparing the extracted plurality of feature amounts with a predetermined threshold value. For example, when the extracted fundamental frequency is higher than a predetermined threshold, the user U has a high fundamental frequency even in normal times, and it is easy to erroneously determine that the user U is feeling joy or anger. Therefore, the processing device 2 sets the coefficient k1 corresponding to joy and the coefficient k2 corresponding to anger to values greater than 0 and less than 1 for the purpose of lowering the voice evaluation value X1 of joy and the voice evaluation value X2 of anger. do.

In the second aspect, the processing device 2 allows the user U to input information regarding the characteristics of his/her own voice. For example, the processing device 2 allows the user U to input gender and age as information regarding the characteristics of his/her own voice. When the gender is female, the fundamental frequency is generally higher than that of male. The corresponding coefficient k1 and the coefficient k2 corresponding to anger are set to values greater than zero and less than one. Similarly, in general, the lower the age, the higher the voice. Therefore, if the input age is equal to or less than a predetermined threshold, the processing device 2 converts the voice evaluation value X1 of joy and the voice evaluation value X2 of anger into For the purpose of lowering, the coefficient k1 corresponding to joy and the coefficient k2 corresponding to anger are set to values greater than zero and less than one.

The character evaluation unit 256 performs speech recognition processing on the sound information SI to recognize the content of speech uttered by a person, and evaluates a plurality of emotions based on the recognition character string RT indicating the recognition result of the speech recognition processing. Textual emotion information TE is generated that includes text evaluation values Y that indicate strength for each. The text emotion information TE includes a text evaluation value Y1 of joy, a text evaluation value Y2 of anger, a text evaluation value Y3 of sadness, and a normal text evaluation value Y4. The character evaluation value Y is a real number of 0 or more.

More specifically, the character evaluation section 256 includes a speech recognition processing section 2561 , a morphological analysis processing section 2563 and an evaluation value calculation section 2565 .
The speech recognition processing unit 2561 performs speech recognition processing on the sound information SI and outputs a recognized character string RT. The speech recognition processing unit 2561 outputs a recognized character string RT by various methods including a method of recognizing a character string from speech using, for example, an acoustic model and a language model prepared in advance.

The morphological analysis processing unit 2563 refers to the analysis dictionary information 31, executes morphological analysis processing on the recognized character string RT, and outputs a corrected recognized character string CRT. The morphological analysis process is a process of decomposing the recognized character string RT into morphemes. In the morphological analysis processing, the part of speech and the sub-classification of the part of speech of the analysis dictionary information 31 are used. The post-correction recognition character string CRT is a character string excluding character strings that are not necessary for estimating the feelings of the user U, such as fillers. Fillers are words that are inserted between utterances, such as "um," "that," and "well."

The evaluation value calculation unit 2565 calculates the character evaluation value Y of each emotion by comparing the character string included in the emotion classification information 33 and the corrected recognized character string CRT, and includes the character evaluation value Y of each emotion. Generate text emotion information TE. Regarding the calculation of the character evaluation value Y of each emotion, the evaluation value calculation unit 2565 calculates the character string included in the corrected recognized character string CRT when the corrected recognized character string CRT includes the character string included in the emotion classification information 33. Increase the character evaluation value Y of the emotion corresponding to the character string.
For example, if the recognized character string after correction CRT is "I won the game today", the evaluation value calculation unit 2565 outputs the following character evaluation values Y for each emotion.

Character evaluation value of joy Y1 1
Character evaluation value of anger Y2 0
Character evaluation value of sadness Y3 0
Normal character evaluation value Y4 0

In the above example, since the corrected recognized character string CRT includes "win" included in the emotion classification information 33, the evaluation value calculation unit 2565 calculates the joy character evaluation value Y1 corresponding to "win". Increase by 1. The amount by which the character evaluation value Y is increased is not limited to 1, and may differ for each character string included in the emotion classification information 33 . For example, the amount of increase in the character evaluation value Y for a character string that strongly indicates joy may be set to 2. Furthermore, when the corrected recognized character string CRT includes a character string included in the emotion classification information 33 and a character string that emphasizes the content, the evaluation value calculation unit 2565 increases the amount of increase in the character evaluation value Y. may For example, if the post-correction recognition character string CRT is "I am very happy that I won the match today", the post-correction recognition character string CRT includes "happy" included in the emotion classification information 33 and is "very". Since the character string emphasizing the content is included, the evaluation value calculation unit 2565 increases the character evaluation value Y1 of joy by 2, for example. It is possible to determine which character string among the corrected recognized character strings CRT is the character string for emphasizing the content, based on the morphemes obtained by the morphological analysis process. In the following example, the increment of the character evaluation value Y is assumed to be 1 for ease of explanation.
Furthermore, if the corrected recognized character string CRT includes a character string included in the emotion classification information 33 and a character string that negates the contents, the evaluation value calculation unit 2565 A process different from the process of increasing the character evaluation value Y corresponding to the character string may be executed. For example, if the post-correction recognition character string CRT is "I couldn't win the game today", the post-correction recognition character string CRT includes "win" included in the emotion classification information 33, but "not". Since a character string that negates the content is included, the evaluation value calculation unit 2565 increases the sadness character evaluation value Y3 by one, for example. It is possible to determine which character string among the corrected recognized character strings CRT is a character string that negates the content, based on the morphemes obtained by the morphological analysis process. In this way, it is possible to estimate whether the post-correction recognized character string CRT has positive or negative content by the morphological analysis processing. In the following example, for ease of explanation, if the post-correction recognition character string CRT includes a character string included in the emotion classification information 33, the character string corresponding to the character string included in the post-correction recognition character string CRT is A description will be given assuming that the evaluation value Y is increased.

The estimation unit 258 estimates one or more emotions that the user U has based on the corrected emotion information CVE and the text emotion information TE. For example, the estimating unit 258 adds the voice evaluation values X1 to X4 of the corrected emotion information CVE and the text evaluation values Y1 to Y4 of the text emotion information TE for each of the plurality of emotions. Calculate the value. The estimating unit 258 compares the added value for each emotion with the threshold and identifies the added value exceeding the threshold. Estimation unit 258 estimates one or more emotions corresponding to the identified added value as one or more emotions that user U has. In the following description, the addition of the voice evaluation value X and the character evaluation value Y means adding for each emotion to generate four added values.
The estimating unit 258 is not limited to simply adding the voice evaluation value X and the character evaluation value Y; and the evaluation value of the other may be added. The predetermined value α is, for example, a value set by the developer of the user device 1, the user U, or the like.

The estimation unit 258 outputs estimated emotion information EI indicating one or more emotions that the estimated user U has. The estimated emotion information EI has, for example, the following two forms. A first aspect of the estimated emotion information EI is an identifier indicating one or more emotions that the estimated user U has. Emotion identifiers include joy identifiers, anger identifiers, sadness identifiers, and normality identifiers. A second aspect of the estimated emotion information EI is an identifier indicating one or more emotions that the estimated user U has, and an evaluation value of the estimated emotion that the user U has. The evaluation value of the estimated emotion of the user U is, for example, a value obtained by adding the voice evaluation value X and the character evaluation value Y of the corrected emotion information CVE for each emotion of one or more emotions that the estimated user U has. .

The output unit 26 outputs data obtained by executing processing corresponding to one or more emotions indicated by the estimated emotion information EI. For example, the output unit 26 has the following two aspects. The output unit 26 in the first mode outputs data obtained by executing processing according to one or more emotions indicated by the estimated emotion information EI on the recognized character string RT obtained by the speech recognition processing unit 2561. Output. There are, for example, the following two modes of processing according to emotion.
A first mode of processing according to emotion is processing of adding a figure embodying emotion to the recognized character string RT. The figure embodying emotion is, for example, a pictogram embodying emotion and an emoticon embodying emotion. A pictogram is an image associated with a character code. The character code is Unicode, for example. An emoticon is a character string that expresses a face by combining symbols and characters. In the following description, it is assumed that the figure embodying emotion is a pictogram embodying emotion. A pictogram embodying joy is, for example, a smiley pictogram. A pictogram embodying anger is, for example, a pictogram showing an angry face. A pictogram embodying sadness is, for example, a pictogram showing a crying face. Furthermore, when the estimated emotion information EI is in the second mode, the output unit 26 embody the emotion indicated by the estimated emotion information EI and having an intensity corresponding to the evaluation value included in the estimated emotion information EI. The resulting pictogram may be determined as a pictogram to be added to the recognized character string RT. For example, when the emotion indicated by the estimated emotion information EI is sadness and the evaluation value included in the estimated emotion information EI is equal to or less than a predetermined threshold value, the output unit 26 recognizes a pictogram representing a tearful face as a recognized character string. It is determined as a pictogram to be added to RT. On the other hand, when the emotion indicated by the estimated emotion information EI is sadness and the evaluation value included in the estimated emotion information EI is greater than the predetermined threshold value, the output unit 26 outputs a pictogram representing a crying face as the recognition character string RT. It is determined as a pictogram to be added. An emoji showing a crying face embodies a higher intensity of sadness compared to an emoji showing a tearful face.
The output unit 26 outputs a character string with pictograms obtained by adding pictograms to the recognized character string RT. There are, for example, the following two positions for adding pictograms. The first position is the end of the recognition string RT. The second position is next to the character string included in the emotion classification information 33 within the recognized character string RT. The display device 4 displays an image based on the character string with pictograms output by the output unit 26 .

A second aspect of the process according to emotion is a process of generating synthesized speech that is read aloud by adding inflections based on emotion. Inflection is, for example, speeding up or slowing down reading speed, or increasing or decreasing volume. A pleasure-based intonation is, for example, increasing the reading speed. Anger-based intonation is, for example, increasing the volume. An inflection based on grief is, for example, lowering the volume. The output unit 26 outputs information indicating synthesized speech read aloud with an intonation based on emotion. Then, the output unit 26 adds an emotion-based intonation to the synthesized speech indicated by the generated data, and outputs information indicating the synthesized speech read aloud with the emotion-based intonation added. The sound emitting device 7 emits synthesized speech indicated by the data output by the output unit 26 .

The output unit 26 in the second mode outputs pictographs embodying one or more emotions indicated by the estimated emotion information EI. The output unit 26 in the second mode does not need to use the recognized character string RT. In the description below, the output unit 26 is described as being in the first mode.

1.2. Operation of First Embodiment Next, the operation of the user device 1 will be described with reference to FIG.

FIG. 6 is a flow chart showing the operation of the user device 1. As shown in FIG. The processing device 2 generates the correction information CI according to one of the two methods of generating the correction information CI described above (step S1). Next, the obtaining unit 21 obtains the sound information SI (step S2). Then, the speech recognition processing unit 2561 performs speech recognition processing on the sound information SI, and outputs a recognized character string RT (step S3). Next, the morphological analysis processing unit 2563 refers to the analysis dictionary information 31, executes morphological analysis processing on the recognized character string RT, and outputs the corrected recognized character string CRT (step S4). Then, the evaluation value calculation unit 2565 calculates the character evaluation value Y of each emotion by comparing the character string included in the emotion classification information 33 and the post-correction recognized character string CRT, and calculates the character evaluation value Y of each emotion. is generated (step S5).

Further, the noise removal unit 251 removes noise from the sound indicated by the sound information SI according to the first parameter P1 and the second parameter P2 to generate the voice information VI (step S6). Then, the speech evaluation unit 252 extracts the sound feature amount from the speech information VI from which noise has been removed (step S7). Next, the voice evaluation unit 252 inputs the sound feature amount to the learning model LM, and acquires the voice emotion information VE including the voice evaluation value x of each emotion from the learning model LM (step S8). The correction unit 253 generates corrected emotion information CVE by correcting the voice evaluation value x of each emotion included in the voice emotion information VE using the correction information CI (step S9).

The estimation unit 258 estimates the emotion that the user U has based on the corrected emotion information CVE and the text emotion information TE, and outputs estimated emotion information EI (step S10). The output unit 26 outputs information obtained by performing processing according to the emotion indicated by the estimated emotion information EI on the recognized character string RT (step S11). After completing the process of step S11, the user device 1 ends the series of processes shown in FIG.

1.3. Effect of the First Embodiment According to the above description, the user device 1 estimates the emotions of the user U using the general-purpose learning model LM. By comparison, the time required to generate the learning model LM can be shortened.
By inputting a plurality of feature amounts relating to an average human voice to the general-purpose learning model LM, it is possible to estimate the emotions of an average human being. However, the user U's voice is affected by the user's U gender, age, characteristics of the user's U's speaking style, and the like, and thus differs from an average human voice. Therefore, simply using the general-purpose learning model LM lowers the accuracy of determination of the emotion that the user U has.
In the user device 1 described above, since the speech emotion information VE output from the learning model LM is corrected using the correction information CI based on the features of the voice of the user U, while using the general-purpose learning model LM, The emotion that the user U has can be estimated with high accuracy.

In addition, the user device 1 removes noise from the sound indicated by the sound information SI to generate voice information VI, and inputs the feature amount of the sound based on the voice information VI to the learning model LM. By inputting the sound feature amount based on the voice information VI into the learning model LM, the speech emotion information VE with higher accuracy can be generated than when the sound feature amount based on the sound information SI is input into the learning model LM. Obtainable.

Moreover, since the user device 1 estimates one or more emotions that the user U has based on the corrected emotion information CVE and the text emotion information TE, the user device 1 estimates the emotions that the user U has based only on the corrected emotion information CVE. Compared to the case, the emotion that the user U has can be estimated with high accuracy.

2. Second Embodiment A user device 1a according to a second embodiment prompts a user U to utter a voice that expresses an explicit emotion, and acquires the user U's explicit voice emotion information VEa from a learning model LM. However, the user device 1 differs from the user device 1 according to the first embodiment in that the correction information CI is adjusted for the purpose of increasing the possibility that the estimation unit 258 estimates that the emotion that the user U has is an explicit emotion. . In the second embodiment, the user device 1a can take an emotion estimation mode for estimating the emotion of the user U and a calibration mode for adjusting the correction information CI. Since the emotion estimation mode corresponds to the first embodiment, the description is omitted. The user device 1a according to the second embodiment will be described below. In the second embodiment exemplified below, elements having the same actions or functions as those of the first embodiment are denoted by reference numerals in the above description, and detailed descriptions thereof are appropriately omitted.

2.1. Functions of Second Embodiment FIG. 7 is a block diagram showing a user device 1a according to the second embodiment. The user device 1a is implemented by a computer system including a processing device 2a, a storage device 3a, a display device 4, an operation device 5, a communication device 6, a sound emitting device 7, and a sound collecting device 8. FIG. The storage device 3a is a recording medium readable by the processing device 2a, and stores a plurality of programs including the control program PRa executed by the processing device 2a, analysis dictionary information 31, and emotion classification information 33. FIG.

The processing device 2a functions as an acquisition unit 21a, an emotion estimation unit 25a, and an output unit 26 by reading and executing the control program PRa from the storage device 3a.

FIG. 8 is a diagram showing an overview of functions of the user device 1a according to the second embodiment. The emotion estimation unit 25 a includes a noise removal unit 251 , a voice evaluation unit 252 , a correction unit 253 , an adjustment unit 254 , a character evaluation unit 256 and an estimation unit 258 .

Acquisition unit 21a acquires sound information SIa indicating sound including voice in which user U expresses one emotion out of a plurality of emotions. Specifically, when the user U operates the operation device 5 to set the user device 1a to the calibration mode, the processing device 2 expresses one emotion out of the plurality of emotions and pronounces it. A screen prompting to do so is displayed on the display device 4 . "One emotion" is hereinafter referred to as "explicit emotion". The obtaining unit 21a obtains the sound information SI obtained after the screen is displayed as the sound information SIa indicating the sound including the voice that the user U expresses the explicit emotion. Which of the plurality of emotions to set as the explicit emotion may be set in advance by, for example, the developer of the user device 1a, or the user U may select the explicit emotion from among the plurality of emotions. good.

The noise removal unit 251 removes noise from the sound indicated by the sound information SIa to generate voice information VIa.

The voice evaluation unit 252 inputs the sound feature amount based on the voice information VIa to the learning model LM, and acquires the user U's explicit voice emotion information VEa from the learning model LM.

The adjustment unit 254 adjusts the correction information CI based on the explicit voice emotion information VEa for the purpose of increasing the possibility of the estimation unit 258 estimating that the emotion that the user U has is the explicit emotion. For example, the adjustment unit 254 performs one or both of a process of increasing the coefficient k corresponding to the explicit emotion and a process of decreasing the coefficient k corresponding to emotions other than the explicit emotion. For example, the adjuster 254 generates coefficients k1 to k4 according to the following equations. However, regarding the ideal voice evaluation value X obtained when the user U expresses emotion and utters a predetermined voice, Xa1 is the voice evaluation value of joy, Xa2 is the voice evaluation value of anger, and Xa2 is the voice of sadness. The evaluation value is represented by Xa3, and the normal voice evaluation value by Xa4.
k1=Xa1/x1
k2=Xa2/x2
k3=Xa3/x3
k4=Xa4/x4
However, the coefficient k does not necessarily have to match Xa/x.

2.2. Operation of Second Embodiment Next, the operation of the user device 1a in the calibration mode will be described with reference to FIG.

FIG. 9 is a flow chart showing the operation of the user device 1a in the calibration mode. The acquisition unit 21a acquires the sound information SIa indicating the sound including the voice in which the user U expresses the explicit emotion (step S21). Next, the noise removing unit 251 removes noise according to the first parameter P1 and the second parameter P2 to generate the voice information VIa (step S22). Then, the speech evaluation unit 252 extracts the sound feature amount from the noise-removed speech information VIa (step S23). Next, the voice evaluation unit 252 inputs the sound feature amount to the learning model LM, and acquires voice emotion information VEa including the voice evaluation value x of each emotion from the learning model LM (step S24).

The adjuster 254 corrects the plurality of voice evaluation values x included in the explicit voice emotion information VEa by the same method as the corrector 253 (step S25). Next, the adjustment unit 254 adjusts the correction information CI for the purpose of increasing the possibility of the estimation unit 258 estimating that the emotion that the user U has is the explicit emotion (step S26). After completing the process of step S26, the user device 1a ends the series of processes shown in FIG.

2.3. Effect of the Second Embodiment According to the above description, when the user U utters a voice that expresses an explicit emotion, the estimation unit 258 estimates that the emotion that the user U has is an explicit emotion. For the purpose of increasing the possibility of correcting the correction information CI, the correction information CI is adjusted based on the explicit voice emotion information VEa. In this aspect, the correct answer for the emotion estimated by the voice emotion information VEa is known, and the corrected emotion information CVE for the user U whose correction information CI is adjusted is the corrected emotion information CVE for the user U whose correction information CI is not adjusted. The accuracy of estimating the emotion that the user U has can be improved compared to the emotion information CVE.
Further, even if the user U utters a voice that expresses an explicit emotion, the intensity of the voice that expresses the emotion is different among the users U. For example, there is a case where a certain user U expresses emotions with voice with high intensity, while another user U expresses emotions with voice with low intensity. The correction information CI in the second embodiment also reflects the difference in the intensity of expressing emotion in voice. For example, a user U who expresses emotion in voice with a high intensity has a voice evaluation value x close to the ideal voice evaluation value X described above, and a coefficient k close to one. On the other hand, the voice evaluation value x of the user U, whose intensity of expressing emotion in voice is low, is smaller than the above-described ideal voice evaluation value X, and the coefficient k is a value away from 1. As described above, the coefficient k becomes a value farther from 1 for the user U who expresses the emotion with a lower intensity, and the difference in the intensity of expressing the emotion in the voice is reflected in the correction information CI. can be improved.

3. Third Embodiment The emotion estimation system SYS according to the third embodiment sets the user device 1b to the calibration mode by the function shown in the second embodiment, and estimates the emotion estimation result of the user U who expresses the explicit emotion. The user device 1a differs from the user device 1a according to the second embodiment in that it has a configuration for adjusting the correction information CI of the user U who has not set the user device 1b to the calibration mode and is not expressing an explicit emotion. do. In the following description, a user U who has set the user device 1b to the calibration mode and has expressed an explicit emotion is referred to as a "calibrated user", who has not set the calibration mode and has expressed an explicit emotion. A user U who does not have a calibration is referred to as a "non-calibrated user".
The emotion estimation system SYS according to the third embodiment will be described below. In addition, in the third embodiment illustrated below, the elements whose actions or functions are the same as those of the second embodiment are referred to by the same reference numerals in the above description, and the detailed description thereof will be omitted as appropriate.

3.1. Overview of Third Embodiment FIG. 10 is a diagram showing the overall configuration of an emotion estimation system SYS. The emotion estimation system SYS includes a user device 1b possessed by a user U, a network NW, and a server device . User devices 1 included in emotion estimation system SYS are from user device 1b1 to user device 1bm. m is an integer of 2 or more. The user U possessing the user device 1b1 is the user U1, and the user U possessing the user device 1bm is the user Um.

In the following, for the sake of simplification of explanation, it is assumed that user U1 is a calibrated user and user U2 is a non-calibrated user.

FIG. 11 is a block diagram showing the configuration of the user device 1b. The user device 1b is implemented by a computer system including a processing device 2b, a storage device 3b, a display device 4, an operation device 5, a communication device 6, a sound emitting device 7, and a sound collecting device 8. FIG. The storage device 3b is a recording medium readable by the processing device 2b, and stores a plurality of programs including a control program PRb executed by the processing device 2b.

The processing device 2b functions as an acquisition unit 21 and an output unit 26 by reading and executing the control program PRb from the storage device 3b.

FIG. 12 is a block diagram showing the configuration of the server device 10. As shown in FIG. The server device 10 is implemented by a computer system including a processing device 2B, a storage device 3B, and a communication device 6B. Each element of the server device 10 is interconnected by one or more buses 9B for communicating information. The storage device 3B is a recording medium readable by the processing device 2B, and stores a plurality of programs including the control program PRB executed by the processing device 2B, analysis dictionary information 31, emotion classification information 33, and a learning model LM. do.

The processing device 2B functions as an emotion estimation section 25B by reading and executing the control program PRB from the storage device 3B. Functions of the emotion estimation system SYS will be described with reference to FIG.

FIG. 13 is a diagram showing an overview of the functions of emotion estimation system SYS. The emotion estimation unit 25B includes a noise removal unit 251, a speech evaluation unit 252B, a correction unit 253, an adjustment unit 254, a character evaluation unit 256, an estimation unit 258, and a specification unit 259.

The acquiring unit 21 of the user device 1b1 acquires sound information SI1 output by the sound collecting device 8 that collects sound including the voice of the user U1. FIG. 14 shows an overview of the adjustment function of the non-calibration user's correction information CI, which is a function realized by the processing device 2B.

FIG. 14 is a diagram showing an overview of the function of adjusting the correction information CI for the non-calibration user. FIG. 14 shows a state in which the user U1 who has been calibrated utters "thank you" and the acquisition unit 21 of the user device 1b1 acquires the sound information SI1.

Return the description to FIG. Regarding the user U1, the noise removal unit 251 removes noise from the sound indicated by the sound information SI1 to generate voice information VI1. The speech evaluation unit 252B inputs the sound feature amount extracted from the speech information VI1 to the learning model LM, and acquires the speech emotional information VE1 from the learning model LM. The correction unit 253 generates corrected emotion information CVE1 by correcting the voice emotion information VE1 using the correction information CI1 based on the voice features of the user U1. Also, the speech recognition processing unit 2561 executes speech recognition processing on the sound information SI1 and acquires a recognized character string RT1 indicating the recognition result of the speech recognition processing. Subsequently, the morphological analysis processing unit 2563 refers to the analysis dictionary information 31, executes morphological analysis processing on the recognized character string RT1, and outputs the corrected recognized character string CRT1. The evaluation value calculation unit 2565 compares the character string included in the emotion classification information 33 with the post-correction recognition character string CRT1 to generate character emotion information TE1.
FIG. 14 shows a state in which the server device 10 has generated the corrected emotion information CVE1 and the character emotion information TE1 based on the sound information SI1.

If the value indicating the degree of difference between the plurality of voice evaluation values X included in the corrected emotion information CVE1 and the character evaluation value Y included in the text emotion information TE1 is equal to or less than a predetermined value, the specifying unit 259 Identify RT1 as a specific character string ST. A character string that is likely to be identified as the specific character string ST is a character string that is often uttered in its original meaning, such as "Thank you" and "Don't be silly".
However, the word "thank you" is sometimes uttered as a social greeting or sarcasm, and may not be uttered with the original meaning of "thank you", which is the meaning of "thank you." In this case, since joy is not expressed in the voice information VI1, the voice evaluation value X and the text evaluation value Y are greatly different. Therefore, the server device 10 associates and stores the recognized character string RT of the calibrated user, the voice evaluation value X, the character evaluation value Y, and the date and time when the voice information VI1 was generated as a log. may refer to this log and specify the specific character string ST based on the tendency of the degree of difference between the speech evaluation value X and the character evaluation value Y with respect to the recognized character string RT. For example, the specifying unit 259 recognizes that the ratio of values indicating the degree of difference between the speech evaluation value X and the character evaluation value Y is equal to or less than a predetermined value from the current time to a certain time in the past. Character string RT is identified as specific character string ST.
There are, for example, the following two aspects of the value indicating the degree of difference. The value indicating the degree of difference in the first mode is the sum of the squares of the difference between the speech evaluation value X and the text evaluation value Y for each of the plurality of emotions, Sum _XY . The sum Sum _XY is obtained, for example, by the following equation (1).
Sum _XY = (X1-Y1) ² + (X2-Y2) ² + (X3-Y3) ² + (X4-Y4) ² (1)
The value indicating the degree of difference in the second mode is the angle θ between the corrected emotion information CVE1 and the text emotion information TE1 when the corrected emotion information CVE1 and the text emotion information TE1 are regarded as a four-dimensional vector. It can be said that the greater the angle θ, the greater the difference between the corrected emotion information CVE1 and the character emotion information TE1. For example, the angle θ is obtained by the following formula (2).
θ=cos ⁻¹ ((CVE1 TE1)/(|CVE1|×|TE1|)) (2)
However, CVE1·TE1 indicates the inner product of the corrected emotion information CVE1 and the character emotion information TE1. |CVE1| indicates the magnitude of the corrected emotion information CVE1. |TE1| indicates the size of the text emotion information TE1.
In the following description, it is assumed that the value indicating the degree of difference is the sum Sum _XY .
FIG. 14 shows an example in which the sum Sum _XY is less than or equal to a predetermined value. Therefore, the identifying unit 259 identifies the recognized character string RT1, “Thank you”, as the specific character string ST.

Regarding the user U2, as shown in FIG. 14, it is assumed that the user U2 utters the specific character string ST "thank you". Acquisition unit 21 of user device 1b2 acquires sound information SI2. The noise removing unit 251 removes noise from the sound indicated by the sound information SI2 to generate the voice information VI2. The speech evaluation unit 252B inputs the sound feature amount extracted from the speech information VI2 to the learning model LM, and acquires the speech emotion information VE2 from the learning model LM. The correction unit 253 generates corrected emotion information CVE2 by correcting the voice emotion information VE2 using the correction information CI2 based on the features of the voice of the user U2. Also, the speech recognition processing unit 2561 executes speech recognition processing on the sound information SI2 and acquires a recognized character string RT2 indicating the recognition result of the speech recognition processing. Subsequently, the morphological analysis processing unit 2563 refers to the analysis dictionary information 31, performs morphological analysis processing on the recognized character string RT2, and outputs the corrected recognized character string CRT2. The evaluation value calculator 2565 compares the character string included in the emotion classification information 33 with the post-correction recognition character string CRT2 to generate text emotion information TE2.
FIG. 14 shows a state in which the server device 10 has generated the corrected emotion information CVE2 and the character emotion information TE2 based on the sound information SI2.

When user U2, who is a non-calibrated user, utters the specific character string ST, the adjustment unit 254 converts a plurality of voice evaluation values X included in the corrected emotion information CVE2 of user U2 to , the correction information CI2 for the user U2 is adjusted for the purpose of bringing it closer to the plurality of character evaluation values Y included in the character emotion information TE2 of the user U2. For example, the adjuster 254 generates coefficients k1 to k4 according to the following equations.
k1=Y1/X1
k2=Y2/X2
k3=Y3/X3
k4=Y4/X4
However, the coefficient k does not necessarily have to match Y/X.

3.2. Operation of the Third Embodiment As in the second embodiment, in the third embodiment, the user device 1b can take an emotion estimation mode for estimating the emotion of the user U and a calibration mode for adjusting the correction information CI. . When the user device 1b is set to the calibration mode, the server device 10 may acquire the sound information SI shown in step S21 from the user device 1b, and execute the steps after step S21. After the series of processes shown in FIG. 9 is completed, the server device 10 stores the identification information of the user device 1b set in the calibration mode in the storage device 3B as the user device 1b possessed by the calibrated user. The identification information of the user device 1b is, for example, a UID (User Identifier), a MAC (Media Access Control) address, an IMSI (International Mobile Subscriber Identity) recorded in a subscriber authentication module (SIM), or a user ID etc. The UID is an ID assigned to each user by a service provider. The operation of emotion estimation system SYS in the emotion estimation mode will be described with reference to FIGS. 15 and 16. FIG.

15 and 16 are flow charts showing the operation of the emotion estimation system SYS in the emotion estimation mode. The server device 10 acquires the correction information CI from the user device 1b (step S31). Specifically, the user device 1b generates the correction information CI according to one of the two generation methods of the correction information CI described above, and transmits the correction information CI to the server device 10 . Next, the server device 10 acquires the sound information SI from the user device 1b (step S32). Then, the speech recognition processing unit 2561 of the emotion estimation unit 25B performs speech recognition processing on the sound information SI, and outputs a recognized character string RT (step S33). Next, the morphological analysis processing unit 2563 of the emotion estimation unit 25B refers to the analysis dictionary information 31, executes morphological analysis processing on the recognized character string RT, and outputs the corrected recognized character string CRT ( step S34). Then, the evaluation value calculation unit 2565 of the emotion estimation unit 25B calculates the character evaluation value Y of each emotion by comparing the character string included in the emotion classification information 33 and the post-correction recognition character string CRT. character evaluation value Y is generated (step S35).

Further, the noise removal unit 251 of the emotion estimation unit 25B removes noise from the sound indicated by the sound information SI according to the first parameter P1 and the second parameter P2 to generate voice information VI (step S41). Then, the voice evaluation unit 252B of the emotion estimation unit 25B extracts the sound feature amount from the noise-removed voice information VI (step S42). Next, the voice evaluation unit 252B of the emotion estimation unit 25B inputs the sound feature amount to the learning model LM, and obtains voice emotion information VE including the voice evaluation value x of each emotion from the learning model LM (step S43). .

Next, the server device 10 determines whether or not the user who owns the user device 1b that is the transmission source of the correction information CI and the sound information SI is a calibrated user (step S44). As a method of determining whether the user is a calibrated user or a non-calibrated user, the user device 1b identifies the user device 1b at one or both of the time of transmitting the correction information CI and the time of transmitting the sound information SI. Send information. When the received identification information of the user device 1b matches the identification information stored as the user device 1b possessed by the calibrated user, the server device 10 outputs a positive determination result and stores it in the storage device 3B. If it does not match the identification information, a negative judgment result is output.

If the determination result in step S44 is affirmative, the correction unit 253 of the emotion estimation unit 25B uses the correction information CI to generate corrected emotion information CVE by correcting the voice evaluation value x of each emotion included in the voice emotion information VE. (step S45). Then, the specifying unit 259 of the emotion estimating unit 25B determines whether the sum Sum _XY of the squares of the difference between the voice evaluation value X included in the corrected emotion information CVE and the text evaluation value Y included in the text emotion information TE is equal to or less than a predetermined value. It is determined whether or not (step S46).
When the determination result of step S44 is affirmative and the determination result of step S46 is affirmative, the identifying unit 259 of the emotion estimating unit 25B identifies the recognized character string RT as the specific character string ST (step S47). Then, the estimation unit 258 of the emotion estimation unit 25B estimates the emotion that the user U has based on the corrected emotion information CVE and the text emotion information TE (step S61). On the other hand, even when the determination result of step S44 is affirmative and the determination result of step S46 is negative, the estimation unit 258 of the emotion estimation unit 25B executes the process of step S61.

If the determination result in step S44 is negative, that is, if the user U who owns the user device 1b that is the transmission source of the correction information CI and the sound information SI is a non-calibration user, the server device 10 determines that the specific character string ST and It is determined whether or not the recognized character string RT matches (step S50).
If the determination result in step S44 is negative and the determination result in step S50 is positive, the specifying unit 259 of the emotion estimation unit 25B includes the speech evaluation value X included in the corrected emotion information CVE in the text emotion information TE. The correction information CI for the non-calibrated user is adjusted for the purpose of approximating the plurality of character evaluation values Y that are used (step S51). Then, the correction unit 253 of the emotion estimation unit 25B uses the correction information CI to generate corrected emotion information CVE by correcting the voice evaluation value x of each emotion included in the voice emotion information VE (step S52).
Even when the determination result of step S44 is negative and the determination result of step S45 is negative, the correction unit 253 of the emotion estimation unit 25B executes the process of step S52.
After completing the process of step S52, the estimation unit 258 of the emotion estimation unit 25B executes the process of step S61.

After executing the process of step S61, the server device 10 transmits the recognized character string RT and the estimated emotion information EI, which is the result of the process of step S61, to the user device 1b. The output unit 26 outputs information obtained by performing processing according to the emotion indicated by the estimated emotion information EI on the recognized character string RT (step S62). After completing the process of step S62, the emotion estimation system SYS ends the series of processes shown in FIGS.

3.3. Effects of the Third Embodiment According to the above description, when the user U2 who is a non-calibration user utters the specific character string ST, the server device 10 converts the plurality of voice evaluation values X included in the corrected emotion information CVE2 into , the correction information CI for the user U2 is adjusted for the purpose of bringing each of the plurality of emotions closer to the plurality of character evaluation values Y included in the character emotion information TE2. The specific character string ST is the recognized character string RT when the value indicating the degree of difference between the voice evaluation value X and the character evaluation value Y for the user U1 who is the calibration user is equal to or less than a predetermined value.
The reason why the correction information CI for the user U2 is adjusted only when the user U2 utters the specific character string ST will be described. Even for a calibrated user, the corrected emotion information CVE and text emotion information TE may not be close to each other. For example, when a calibrated user utters a character string with a meaning different from the original meaning of the character string, the corrected emotion information CVE and the character emotion information TE may not be close to each other. Examples of the user U uttering a meaning different from the original meaning of the character string include a case where the user U utters an ironic content and a case where the user U utters a joke. If the user U speaks sarcastically or jokingly, the accuracy of the textual emotion information TE is lowered. Therefore, if the user U's emotions are estimated based only on the textual emotion information TE, the accuracy is lowered. In addition, when the user U utters an office communication such as "I just arrived", the accuracy of the text emotion information TE is lowered. . The reason why the accuracy of the character emotion information TE is lowered when the utterance content is business communication is that the utterance content indicating business communication includes a character string representing an emotion registered in the emotion classification information 33. This is because the character evaluation values Y1 to Y4 tend to be low compared to general utterance contents, and the character evaluation values Y1 to Y4 are small values. When the user U utters a sarcastic content, when the user U utters a joke, and when the user U utters an office contact, the emotions of the user U can be accurately estimated based only on the character emotion information TE. This is an example of a case where estimation is not possible. When the original meaning of the character string is uttered, the values of the corrected emotional information CVE and the text emotional information TE tend to be close to each other.
Therefore, since the specific character string ST has a value indicating the degree of difference between the voice evaluation value X and the character evaluation value Y being equal to or less than a predetermined value, the specific character string ST is highly likely to be uttered in its original meaning. I can say. When the user U2 utters the specific character string ST, there is a high possibility that the user U2 is uttering the original meaning of the specific character string ST. The value should be close to the emotion information TE.
Here, for non-calibrated users, the accuracy of the corrected emotion information CVE is generally lower than the accuracy of the character emotion information TE. The reason for this is that the text emotion information TE is less influenced by the voice features of the user U, while the voice emotion information VE is greatly influenced by the voice features of the user U, and the non-calibrated user's correction information CI is not properly adjusted.
Therefore, in the third embodiment, when the user U2 utters the specific character string ST, there is a high possibility that the character emotion information TE indicates the correct emotion. The correction information CI for the user U2 is adjusted for the purpose of approximating the character evaluation value Y. FIG. As described above, it is possible to improve the accuracy of estimating the emotion of the user U without using the calibration mode for the non-calibration user. Since the non-calibration user can improve the estimation accuracy of emotion without setting the user device 1b to the calibration mode, the user device 1b can reduce the trouble of the non-calibration user.

4. Fourth Embodiment The emotion estimation system SYSc according to the fourth embodiment uses the emotion estimation result of the calibrated user to adjust the first parameter P1 and the second parameter P2 for the non-calibrated user. It differs from the emotion estimation system SYS according to the third embodiment.
The emotion estimation system SYSc according to the fourth embodiment will be described below. In addition, in the fourth embodiment illustrated below, the elements whose actions or functions are the same as those of the third embodiment are referred to by reference numerals in the above description, and their detailed descriptions are appropriately omitted.

FIG. 17 is a diagram showing the overall configuration of emotion estimation system SYSc. The emotion estimation system SYSc includes a user device 1b possessed by a user U, a network NW, and a server device 10C.

FIG. 18 is a block diagram showing the configuration of the server device 10C. The server device 10C is implemented by a computer system including a processing device 2C, a storage device 3C, and a communication device 6B. Storage device 3C is a recording medium readable by processing device 2C, and stores a plurality of programs including control program PRC executed by processing device 2C, analysis dictionary information 31, emotion classification information 33, and learning model LM. do.

The processing device 2C functions as an emotion estimation section 25C by reading and executing the control program PR from the storage device 3C. Functions of the emotion estimation system SYSc will be described with reference to FIG.

4.1. Functions of Fourth Embodiment FIG. 19 is a diagram showing an overview of the functions of the emotion estimation system SYSc. The emotion estimation unit 25C includes a noise removal unit 251C, a speech evaluation unit 252B, a correction unit 253, an adjustment unit 254C, a character evaluation unit 256, an estimation unit 258, and a specification unit 259.

In the fourth embodiment, the first parameter P1 and the second parameter P2 used in the noise removal section 251C are prepared for each user U. In the following description, information including the first parameter P1 and the second parameter P2 for the user U1 is referred to as parameter information TI1, and information including the first parameter P1 and the second parameter P2 for the user U2 is referred to as parameter information TI2. .

FIG. 20 is a diagram showing an overview of the adjustment function of parameter information TI for non-calibration users. FIG. 20 shows a state in which user U1, who has been calibrated, utters "thank you" and acquisition unit 21 of user device 1b1 acquires sound information SI1.

Returning the description to FIG. For the user U1, the noise removal unit 251C removes noise from the sound indicated by the sound information SI1 based on the first parameter P1 and the second parameter P2 included in the parameter information TI1 to generate voice information VI1. For subsequent processing, the emotion estimation unit 25C performs the same processing as in the third embodiment, generates corrected emotion information CVE1 and text emotion information TE1, and converts the recognition character string RT1 "thank you" to the specific character string ST. Identify as

Regarding the user U2, as shown in FIG. 20, it is assumed that the user U2 utters the specific character string ST "thank you". The adjustment unit 254C adjusts the plurality of voice evaluation values X included in the corrected emotion information CVE2 of the user U2 closer to the plurality of character evaluation values Y included in the text emotion information TE2 of the user U2 for each of the plurality of emotions. , adjust the parameter information TI2 for user U2. Specifically, the adjuster 254C causes the noise remover 251C to generate the voice information VI2 based on the first parameter P1 and the second parameter P2 of the current parameter information TI2. Then, the adjustment unit 254C causes the voice evaluation unit 252B and the correction unit 253 to generate corrected emotion information CVE2, and the character evaluation unit 256 to generate text emotion information TE2. Then, the adjustment unit 254C compares the voice evaluation value X included in the corrected emotion information CVE2 with the text evaluation value Y included in the text emotion information TE2. For example, the adjuster 254C slightly changes the first parameter P1 and the second parameter P2 of the parameter information TI2. The adjustment unit 254C regenerates the corrected emotion information CVE based on the first parameter P1 and the second parameter P2 that are slightly changed, and determines the degree of difference between the regenerated corrected emotion information CVE and the text emotion information TE2. If the indicated value is smaller than the value indicating the degree of difference between the corrected emotion information CVE before re-creation and the character emotion information TE2, the plurality of voice evaluation values X of the user U2 are replaced with the plurality of character evaluation values Y of the user U2. It is determined that the purpose of bringing the

In FIG. 20, the adjustment unit 254C adjusts the first parameter P1 and the second parameter P2 included in the parameter information TI2 for the purpose of bringing the plurality of voice evaluation values X of the user U2 closer to the plurality of character evaluation values Y of the user U2. indicates to adjust.

4.2. Operation of Fourth Embodiment Next, the operation of the emotion estimation system SYSc in the emotion estimation mode will be described with reference to FIG.

FIG. 21 is a flow chart showing the operation of emotion estimation system SYSc in emotion estimation mode. In the operation of the emotion estimation system SYSc in the emotion estimation mode shown in the third embodiment and the operation of the emotion estimation system SYSc in the emotion estimation mode of the fourth embodiment, the processing from step S31 to step S35 shown in FIG. are common. Therefore, illustration and description of the processing from step S31 to step S35 are omitted.

After the process of step S35 ends, the server device 10C determines whether or not the user U who owns the user device 1b, which is the transmission source of the correction information CI and the sound information SI, is a calibrated user (step S71). If the determination result in step S71 is affirmative, the noise removal unit 251C of the emotion estimation unit 25C removes noise from the sound indicated by the sound information SI according to the first parameter P1 and the second parameter P2 of the parameter information T1, and removes the noise from the voice information. A VI is generated (step S72). The voice evaluation unit 252B of the emotion estimation unit 25C extracts the sound feature amount from the noise-removed voice information VI (step S73). Next, the voice evaluation unit 252B of the emotion estimation unit 25C inputs the sound feature amount to the learning model LM, and acquires voice emotion information VE including the voice evaluation value x of each emotion from the learning model LM (step S74). . Then, the correction unit 253 of the emotion estimation unit 25C uses the correction information CI to generate corrected emotion information CVE by correcting the voice evaluation value x of each emotion included in the voice emotion information VE (step S75).

Then, the specifying unit 259 of the emotion estimating unit 25C determines whether the sum Sum _XY of the squares of the difference between the voice evaluation value X included in the corrected emotion information CVE and the text evaluation value Y included in the text emotion information TE is equal to or less than a predetermined value. It is determined whether or not (step S76).
When the determination result of step S71 is affirmative and the determination result of step S76 is affirmative, specifying unit 259 of emotion estimating unit 25C specifies recognized character string RT as specific character string ST (step S77). Then, the estimation unit 258 of the emotion estimation unit 25C estimates the emotion that the user U has based on the corrected emotion information CVE and the text emotion information TE (step S91). On the other hand, even when the determination result of step S71 is affirmative and the determination result of step S76 is negative, the estimation unit 258 of the emotion estimation unit 25C executes the process of step S91.

If the determination result in step S71 is negative, that is, if the user U who owns the user device 1b that is the transmission source of the correction information CI and the sound information SI is a non-calibration user, the server device 10C determines that the specific character string ST and It is determined whether or not the recognized character string RT matches (step S81). If the determination result in step S71 is negative and the determination result in step S81 is positive, the adjustment unit 254C of the emotion estimation unit 25B includes the voice evaluation value X included in the corrected emotion information CVE in the text emotion information TE. The parameter information TI for non-calibrated users is adjusted for the purpose of approximating a plurality of character evaluation values Y (step S82). Then, the noise removal unit 251C of the emotion estimation unit 25C removes noise from the sound indicated by the sound information SI according to the first parameter P1 and the second parameter P2 of the parameter information TI to generate voice information VI (step S83). .
Even when the determination result of step S71 is negative and the determination result of step S81 is negative, the noise removing section 251C of the emotion estimating section 25C executes the process of step S83.

After the process of step S83 is completed, the voice evaluation unit 252B of the emotion estimation unit 25C extracts the sound feature amount from the noise-removed voice information VI (step S84). Next, the voice evaluation unit 252B of the emotion estimation unit 25C inputs the sound feature amount to the learning model LM, and acquires voice emotion information VE including the voice evaluation value x of each emotion from the learning model LM (step S85). . Then, the correction unit 253 of the emotion estimation unit 25C uses the correction information CI to generate corrected emotion information CVE by correcting the voice evaluation value x included in the voice emotion information VE (step S86). After completing the process of step S86, the estimation unit 258 of the emotion estimation unit 25C executes the process of step S91.

After completing the process of step S91, the server device 10C transmits the recognized character string RT and the estimated emotion information EI, which is the result of the process of step S61, to the user device 1b. The output unit 26 outputs information obtained by performing processing according to the emotion indicated by the estimated emotion information EI on the recognized character string RT (step S92). After completing the process of step S92, the emotion estimation system SYSc ends the series of processes shown in FIG.

4.3. Effects of the Fourth Embodiment In the fourth embodiment, similarly to the third embodiment, when the user U2 utters the specific character string ST, there is a high possibility that the character emotion information TE indicates the correct emotion. Therefore, the server device 10C adjusts the parameter information TI for the user U2 for the purpose of bringing the speech evaluation value X closer to the character evaluation value Y. FIG. As described above, it is possible to improve the accuracy of estimating the emotion of the user U without using the calibration mode for the non-calibration user. The non-calibration user can improve the accuracy of emotion estimation without setting the user device 1b to the calibration mode, and the user device 1b can reduce the labor of the non-calibration user.
The performance of the sound collector 8 differs between the user devices 1b. For example, different manufacturers of sound collectors 8 generally have different performance of the sound collectors 8 . In addition, since the performance of the sound collector 8 tends to deteriorate due to aging, even if the sound collector 8 is manufactured by the same manufacturer, the performance of the sound collector 8 will be different if the number of days from the time of manufacture is different. There is a tendency. Since the performance of the sound collector 8 differs among the user devices 1b and the amount of noise included in the sound information SI is also different, the emotion of the user U can be accurately estimated by adjusting the parameter information TI.
For example, when noise processing is performed using learned parameter information TI, information necessary for emotion estimation may be missing from voice information VI due to differences in the performance of the sound collector 8 . Therefore, by adjusting the parameter information TI according to the performance of the sound collector 8, the emotion of the user U can be accurately estimated.

5. Fifth Embodiment The emotion estimation system SYSd according to the fifth embodiment differs from the first embodiment in that the processing of the emotion estimation unit 25 shown in the first embodiment is distributed between the server device 10D and the user device 1d. This user device 1 is different. The emotion estimation system SYSd according to the fifth embodiment will be described below. In addition, in the fifth embodiment illustrated below, elements having the same action or function as those of the first embodiment are denoted by reference numerals in the above description, and detailed description thereof will be omitted as appropriate.

5.1. Overview of Fifth Embodiment FIG. 22 is a diagram showing the overall configuration of an emotion estimation system SYSd. The emotion estimation system SYSd includes a user device 1d owned by a user U, a network NW, and a server device 10D.

FIG. 23 is a block diagram showing the configuration of the user device 1d. The user device 1d is implemented by a computer system including a processing device 2d, a storage device 3d, a display device 4, an operation device 5, a communication device 6, a sound emitting device 7, and a sound collecting device 8. FIG. The storage device 3d is a recording medium readable by the processing device 2d, and stores a plurality of programs including the control program PRd executed by the processing device 2d. The communication device 6 is an example of a "second communication device".

The processing device 2d functions as an acquisition unit 21, a first emotion estimation unit 25d, and an output unit 26 by reading and executing the control program PRd from the storage device 3d.

FIG. 24 is a block diagram showing the configuration of the server device 10D. The server device 10D is implemented by a computer system including a processing device 2D, a storage device 3D, and a communication device 6B. The storage device 3D is a recording medium readable by the processing device 2D, and stores a plurality of programs including a control program PRD executed by the processing device 2D, analysis dictionary information 31, emotion classification information 33, and a learning model LM. do. The communication device 6B is an example of the "first communication device".

The processing device 2D functions as a second emotion estimation section 25D by reading and executing the control program PRD from the storage device 3D. Functions of the emotion estimation system SYSd will be described with reference to FIG.

FIG. 25 is a diagram showing an outline of functions of the emotion estimation system SYSd. First emotion estimation section 25 d includes correction section 253 and estimation section 258 . Second emotion estimation unit 25D includes noise removal unit 251, voice evaluation unit 252, and character evaluation unit 256. FIG.

The acquisition unit 21 acquires sound information SI1 output by the sound collector 8 that collects sound including the voice of the user U1. The communication device 6 transmits the sound information SI to the server device 10D. Second emotion estimation section 25D generates voice emotion information VE, text emotion information TE, and recognized character string RT based on sound information SI. The communication device 6B transmits the voice emotion information VE, the text emotion information TE, and the recognized character string RT to the user device 1d.

The correction unit 253 uses the correction information CI to generate the corrected emotion information CVE. The estimation unit 258 estimates the emotion that the user U has based on the corrected emotion information CVE and the text emotion information TE. The output unit 26 outputs data obtained by performing processing corresponding to the emotion indicated by the estimated emotion information EI on the recognized character string RT.

5.2. Effect of Fifth Embodiment According to the above description, in the emotion estimation system SYSd, the user device 1d can reduce the load compared to the user device 1 in the first embodiment.

6. Sixth Embodiment The emotion estimation system SYSe according to the sixth embodiment differs from the second embodiment in that the processing of the emotion estimation unit 25 shown in the second embodiment is distributed between the server device 10D and the user device 1e. It is different from the user device 1a. The emotion estimation system SYSe according to the sixth embodiment will be described below. In the sixth embodiment exemplified below, the elements whose actions or functions are equivalent to those of the second embodiment or the fifth embodiment will be referred to in the above description by using the reference numerals, and their detailed description will be appropriately described. abbreviated to

6.1. Overview of Sixth Embodiment FIG. 26 is a diagram showing the overall configuration of an emotion estimation system SYSe. The emotion estimation system SYSe includes a user device 1e possessed by a user U, a network NW, and a server device 10D.

FIG. 27 is a block diagram showing the configuration of the user device 1e. The user device 1e is implemented by a computer system including a processing device 2e, a storage device 3e, a display device 4, an operation device 5, a communication device 6, a sound emitting device 7, and a sound collecting device 8. FIG. The storage device 3e is a recording medium readable by the processing device 2e, and stores a plurality of programs including the control program PRe executed by the processing device 2e.

The processing device 2e functions as an acquisition unit 21a, a first emotion estimation unit 25e, and an output unit 26 by reading and executing the control program PRe from the storage device 3e. Functions of the emotion estimation system SYSe will be described with reference to FIG.

FIG. 28 is a diagram showing an outline of functions of the emotion estimation system SYSe. First emotion estimating portion 25 e includes correcting portion 253 , adjusting portion 254 , and estimating portion 258 .

Acquisition unit 21a acquires sound information SIa indicating a sound including a voice in which user U expresses an explicit emotion. The server device 10D generates voice emotion information VEa based on the sound information SIa. The communication device 6B then transmits the voice emotion information VEa to the user device 1 .

The adjustment unit 254 adjusts the correction information CI based on the explicit voice emotion information VEa for the purpose of increasing the possibility of the estimation unit 258 estimating that the emotion that the user U has is the explicit emotion.

6.2. Effect of Sixth Embodiment According to the above description, in the emotion estimation system SYS, the user device 1d can reduce the load compared to the user device 1 in the second embodiment.

7. Seventh Embodiment The emotion estimation system SYSf according to the seventh embodiment differs from the third embodiment in that the processing of the emotion estimation unit 25 shown in the third embodiment is distributed between the server device 10F and the user device 1f. It differs from the emotion estimation system SYS. The emotion estimation system SYSf according to the seventh embodiment will be described below. It should be noted that, in the seventh embodiment illustrated below, the elements whose actions or functions are equivalent to those of the third embodiment or the fifth embodiment will be referred to in the above description by using the reference numerals, and the detailed description of each will be made as appropriate. abbreviated to

7.1. Overview of Seventh Embodiment FIG. 29 is a diagram showing the overall configuration of an emotion estimation system SYSf. The emotion estimation system SYSf includes a user device 1f owned by a user U, a network NW, and a server device 10F. User U1 is an example of a "first user". User U2 is an example of a "second user".

FIG. 30 is a block diagram showing the user device 1f. The user device 1f is implemented by a computer system including a processing device 2f, a storage device 3f, a display device 4, an operation device 5, a communication device 6, a sound emitting device 7, and a sound collecting device 8. FIG. The storage device 3f is a recording medium readable by the processing device 2f, and stores a plurality of programs including a control program PRf executed by the processing device 2f.

The processing device 2f functions as an acquisition unit 21, a first emotion estimation unit 25f, and an output unit 26 by reading and executing the control program PRf from the storage device 3f.

FIG. 31 is a block diagram showing the configuration of the server device 10F. The server device 10F is implemented by a computer system including a processing device 2F, a storage device 3F, and a communication device 6B. The storage device 3F is a recording medium readable by the processing device 2F, and stores a plurality of programs including the control program PRF executed by the processing device 2F, analysis dictionary information 31, emotion classification information 33, and a learning model LM. do.

The processing device 2F functions as a second emotion estimation section 25F by reading and executing the control program PRD from the storage device 3F. Functions of the emotion estimation system SYSf will be described with reference to FIG.

FIG. 32 is a diagram showing an overview of the functions of the emotion estimation system SYSf. First emotion estimator 25 f includes corrector 253 , adjuster 254 , and estimator 258 . Second emotion estimation unit 25</b>F includes noise removal unit 251 , voice evaluation unit 252 , character evaluation unit 256 and identification unit 259 . The user device 1f1 is an example of a "first terminal device". The user device 1f2 is an example of a "second terminal device". In order to avoid complication of the drawing, illustration of the functions realized by the processing device 2f of the user device 1f1 is omitted.

The acquisition unit 21 of the user device 1f1 acquires sound information SI1 output by the sound collector 8 of the user device 1f1. The sound collector 8 of the user device 1f1 is an example of the "first sound collector". The communication device 6 of the user device 1f1 transmits the sound information SI1 to the server device 10F. The noise removal unit 251 removes noise from the sound indicated by the sound information SI1 to generate the voice information VI1. Regarding the user U1, the subsequent processing is the same as the speech evaluation unit 252B, the correction unit 253, the speech recognition processing unit 2561, the morphological analysis processing unit 2563, the evaluation value calculation unit 2565, and the identification unit 259 shown in FIG. omitted. Further, although not shown, the second emotion estimation unit 25F transmits the recognized character string RT1, the voice emotion information VE1 of the user U1, and the text emotion information TE1 of the user U1 to the user device 1f1. Then, in order for the specifying unit 259 to specify the specific character string ST, the communication device 6 of the user device 1f1 transmits the corrected emotion information CVE1 of the user U1 to the server device 10F.
Regarding the user U2, the acquisition unit 21 of the user device 1f2 acquires sound information SI2 output by the sound collector 8 of the user device 1f2. The sound collector 8 of the user device 1f2 is an example of the "second sound collector". The communication device 6 of the user device 1f2 transmits the sound information SI2 to the server device 10F. The communication device 6 of the user device 1f2 is an example of a "third communication device". Regarding the user U2, the subsequent processing is the same as the speech evaluation unit 252B, the correction unit 253, the speech recognition processing unit 2561, the morphological analysis processing unit 2563, and the evaluation value calculation unit 2565 shown in FIG. 13, so the description is omitted. The communication device 6B transmits the specific character string ST, the voice emotion information VE2 of the user U2, and the character emotion information TE2 of the user U2 to the user device 1f2.

When user U2, who is a non-calibrated user, utters the specific character string ST, the adjustment unit 254 converts a plurality of voice evaluation values X included in the corrected emotion information CVE2 of user U2 to , the correction information CI2 for the user U2 is adjusted for the purpose of bringing it closer to the plurality of character evaluation values Y included in the character emotion information TE2 of the user U2.

7.2. Effect of Seventh Embodiment According to the above description, in the emotion estimation system SYSf, the server device 10F can reduce the load compared to the server device 10 in the third embodiment.

8. Modifications The present invention is not limited to the embodiments illustrated above. Specific modification modes are exemplified below. Two or more aspects arbitrarily selected from the following examples may be combined.

(1) As a first modified example, the estimating unit 258 estimates the emotion of the user U based on the corrected emotion information CVE and the text emotion information TE, but the present invention is not limited to this. An example in which the estimation unit 258 estimates one or more emotions that the user U has based on the corrected emotion information CVE will be described with reference to FIG.

FIG. 33 is a diagram showing an overview of functions of the user device 1g in the first modified example. The processing device 2 of the user device 1g functions as an acquisition unit 21, an emotion estimation unit 25g, and an output unit 26g by reading and executing a control program from the storage device 3 of the user device 1g. The emotion estimation unit 25g includes a noise removal unit 251, a voice evaluation unit 252, a correction unit 253, and an estimation unit 258g. The estimation unit 258g estimates one or more emotions that the user U has based on the corrected emotion information CVE. For example, the estimating unit 258g compares the audio evaluation values X1 to X4 of the corrected emotion information CVE with a threshold and identifies the audio evaluation value X exceeding the threshold. The estimation unit 258g estimates one or more emotions corresponding to the identified voice evaluation value X as one or more emotions that the user U has. The emotion estimation unit 25g outputs estimated emotion information EI indicating one or more emotions that the estimated user U has.
The output unit 26g outputs the estimated emotion information EI. For example, the output unit 26g outputs to the display device 4 a character string indicating the emotion indicated by the estimated emotion information EI.

(2) The processing of the emotion estimation unit 25C shown in the fourth embodiment may be distributed between the server device 10 and the user device 1. FIG. For example, the second emotion estimation unit 25 in the server device 10 has a noise removal unit 251C, a voice evaluation unit 252B, a character evaluation unit 256, and a specification unit 259. First emotion estimating section 25 in user device 1 has correcting section 253 , adjusting section 254 , and estimating section 258 .

(3) The emotion estimator 25 estimates one or more emotions out of joy, anger, sadness, and normality, but it may estimate one emotion. For example, the estimation unit 258 adds the voice evaluation values X1 to X4 of the corrected emotion information CVE and the text evaluation values Y1 to Y4 of the text emotion information TE for each emotion to calculate the added value for each emotion. The estimation unit 258 may estimate the emotion having the largest value among the added values for each emotion as the emotion that the user U has.

(4) In the third embodiment, the emotion estimation unit 25B is implemented by the server device 10, but it may be applied to one user device 1 as well. For example, this is the case where the user device 1 is possessed by a plurality of users U. In a certain period, the user U1 possesses the user device 1 and sets the user device 1 to the calibration mode, and in a period after the certain period, the user U2 possesses the user device 1, the third embodiment may apply.

(5) In the fifth, sixth, and seventh embodiments, the communication device 6B converts the recognized character string RT, the voice emotion information VE2 of the user U2, and the character emotion information TE2 of the user U2 to , to the user device 1f2, but the recognition character string RT may not be transmitted. For example, the user device 1f outputs to the display device 4 a character string indicating the emotion indicated by the estimated emotion information EI.

(6) The user device 1 may not have the sound collector 8 . If the user device 1 does not have the sound collecting device 8 , the user device 1 may acquire the sound information SI via the communication device 6 or may acquire the sound information SI stored in the storage device 3 .

(7) The user device 1 does not have to have the sound emitting device 7 .

(8) The user device 1 may be a smart speaker. If the user device 1 is a smart speaker, the user device 1 may not have the display device 4 and the operation device 5 .

(9) The emotion classification information 33, as shown in FIG. 4, includes each of a plurality of morphemes used by a word, such as "win" and "win", as joy, anger, sadness, and normal. Although classified as one, it is not limited to this. For example, the emotion classification information 33 may classify the character strings registered in the original data of the analysis dictionary information 31 into either joy, anger, sadness, or normal. For example, the emotion classification information 33 classifies the character strings "happy", "pass", and "win" registered in the original data of the analysis dictionary information 31 as joy. The evaluation value calculation unit 2565 decomposes the corrected recognized character string CRT into morphemes, and converts the decomposed morphemes into character strings registered in the original form data of the analysis dictionary information 31 . Then, when the character string obtained by the conversion matches the character string included in the emotion classification information 33, the evaluation value calculation unit 2565 calculates the character string included in the post-correction recognized character string CRT. Increase character evaluation value Y of emotion.

(10) The evaluation value calculation unit 2565 calculates the character evaluation value Y for each emotion for the corrected recognized character string CRT, but does not calculate the character evaluation value Y for each emotion for the recognized character string RT. good too. However, the recognized character string RT includes character strings that are unnecessary for estimating emotions. Therefore, by calculating the character evaluation value Y for each emotion with respect to the post-correction recognition character string CRT, it is possible to estimate the emotion in comparison with the case where the character evaluation value Y for each emotion is calculated for the recognition character string RT. Can improve accuracy.

(11) Although the value indicating the degree of difference in the first mode was the sum of the squares of the difference between the voice evaluation value X and the character evaluation value Y, the difference between the voice evaluation value X and the character evaluation value Y It may be a value determined by an arbitrary evaluation function that defines the distance between evaluation values, such as the sum of the absolute values of .

(12) Although the example in which the user U speaks Japanese has been used, the above aspects can be applied to any language the user speaks. For example, even if the user U speaks English, French, Chinese, or the like other than Japanese, each of the above aspects can be applied. For example, when the user U speaks English, the analysis dictionary information 31 is information about English morphemes, and the emotion classification information 33 is data in which English words are classified into either joy, anger, sadness, or normal. If it is

(13) The block diagrams used to describe each of the above aspects show blocks in units of functions. These functional blocks (components) are implemented by any combination of hardware and/or software. Further, means for realizing each functional block is not particularly limited. That is, each functional block may be implemented by one device physically and/or logically coupled, or may be implemented by two or more physically and/or logically separated devices directly and/or indirectly. These multiple devices may be connected together (eg, wired and/or wirelessly).

(14) As long as there is no contradiction, the order of the processing procedures, sequences, flowcharts, and the like in each aspect described above may be changed. For example, the methods described herein present elements of the various steps in a sample order and are not limited to the specific order presented.

(15) In each aspect described above, input/output information and the like may be stored in a specific location (for example, memory) or managed in a management table. Input/output information and the like can be overwritten, updated, or appended. The output information and the like may be deleted. The entered information and the like may be transmitted to another device.

(16) In each aspect described above, the determination may be made by a value represented by 1 bit (0 or 1), or by a true/false value (Boolean: true or false). , may be performed by numerical comparison (eg, comparison with a predetermined value).

(17) In each aspect described above, a portable information processing device such as a smart phone was exemplified as the user device 1, but the specific form of the user device 1 is arbitrary, and is not limited to the examples of the above forms. . For example, a portable or stationary personal computer may be used as the user device 1 .

(18) In each aspect described above, the storage device 3 is a recording medium that can be read by the processing device 2. Although ROM and RAM are examples, flexible discs, magneto-optical discs (e.g., compact discs, digital versatile discs, Blu-ray discs), smart cards, flash memory devices (e.g. cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable discs, hard disks, floppies ) disk, magnetic strip, database, server or other suitable storage medium. Also, the program may be transmitted from a network. Also, the program may be transmitted from a communication network via an electric communication line.

(19) Each aspect described above includes LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G, 5G, FRA (Future Radio Access), W-CDMA (registered trademark) , GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark) ), systems utilizing other suitable systems, and/or future generation systems enhanced based on these.

(20) In each of the above aspects, the information, signals, etc. described may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may refer to voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these. may be represented by a combination of
The terms explained in this specification and/or terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

(21) FIGS. 2, 5, 7, 8, 11, 12, 13, 14, 18, 19, 20, 23, 24, 25, 27, and 28 , 30, 31, 32, and 33 are implemented by any combination of hardware and software. Also, each function may be implemented by a single device, or may be implemented by two or more devices configured separately from each other.

(22) The programs exemplified in each of the above embodiments, whether referred to as software, firmware, middleware, microcode, hardware description language or by any other name, may include instructions, instruction sets, code, code segments. , program code, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures or functions, or the like.
Software, instructions, etc. may also be sent and received over a transmission medium. For example, the software can be used to access websites, servers, or other When transmitted from a remote source, these wired and/or wireless technologies are included within the definition of transmission media.

(23) In each of the above-described embodiments, information, parameters, etc. may be represented by absolute values, may be represented by relative values from a predetermined value, or may be represented by corresponding other information. good too.

(24) The names used for the parameters described above are not limiting in any way. Further, the formulas, etc. using these parameters may differ from those explicitly disclosed herein.

(25) In each of the above embodiments, the user equipment 1 may be a mobile station. A mobile station is defined by those skilled in the art as subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless It may also be called a terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable term.

(26) In each of the embodiments described above, the phrase "based on" does not mean "based only on," unless expressly specified otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

(27) Any reference to elements using the "first," "second," etc. designations used herein does not generally limit the quantity or order of those elements. These designations may be used herein as a convenient method of distinguishing between two or more elements. Thus, references to first and second elements do not imply that only two elements may be employed therein, or that the first element must precede the second element in any way.

(28) To the extent that "including," "comprising," and variations thereof are used in each of the above-described embodiments in the specification or claims, these terms include: Like the term "comprising," it is intended to be inclusive. Furthermore, the term "or" as used in this specification or the claims is not intended to be an exclusive OR.

(29) Throughout this application, where articles have been added by translation, e.g., a, an and the in English, these articles shall be used unless the context clearly indicates otherwise. Including multiple.

(30) It will be clear to those skilled in the art that the present invention is not limited to the embodiments described herein. The present invention can be implemented as modifications and changes without departing from the spirit and scope of the present invention determined based on the description of the claims. Accordingly, the description herein is for illustrative purposes only and is not meant to be limiting in any way. Also, a plurality of aspects selected from the aspects exemplified in this specification may be combined.

1, 1a, 1b, 1d, 1e, 1f, 1g... user device, 10, 10C, 10D, 10F... server device, 21, 21a... acquisition unit, 26... output unit, 31... dictionary information for analysis, 33... emotion Classification information 251, 251C Noise removal unit 252, 252B Speech evaluation unit 253 Correction unit 254, 254C Adjustment unit 256 Character evaluation unit 258 Estimation unit 259 Identification unit CI Correction information, CVE... corrected emotional information, LM... learning model, P1... first parameter, P2... second parameter, TE... text emotional information, U... user, VE... voice emotional information, VI... voice information, X... voice evaluation Value, Y... Character evaluation value.

Claims (9)

Speech information indicating a user's speech for a learning model that has already learned the relationship between a plurality of feature values corresponding to a human speech and the intensity of each of a plurality of emotions held by a person who uttered the speech. a speech evaluation unit for inputting a plurality of feature amounts based on the learning model, and obtaining speech emotion information including a speech evaluation value indicating the strength of each of the plurality of emotions held by the user, from the learning model;
a correcting unit that generates corrected emotional information obtained by correcting the voice emotional information using correction information based on the features of the voice of the user;
an estimation unit for estimating one or more emotions of the user from among the plurality of emotions based on the corrected emotion information;
Emotion estimation device. an acquisition unit that acquires sound information output by a sound collecting device that collects sound including the user's voice;
a noise removal unit that removes noise from the sound indicated by the sound information to generate the audio information;
The emotion estimation device according to claim 1, comprising: The speech evaluation unit inputs, to the learning model, a plurality of feature amounts based on speech information indicating a speech in which the user explicitly expresses one of the plurality of emotions, and obtaining explicit speech affective information from the learning model;
an adjustment unit that adjusts the correction information based on the explicit voice emotion information for the purpose of increasing the possibility that the estimation unit estimates that the emotion held by the user is the one emotion;
The emotion estimation device according to claim 1 or 2, comprising: A speech recognition process for recognizing the utterance content of the voice uttered by a person is performed on sound information output by a sound collecting device that collects sound including the user's voice , and the recognition result of the speech recognition process is performed. a character evaluation unit that generates character emotion information including a character evaluation value indicating the strength of each of the plurality of emotions held by the user, based on the recognized character string shown;
The estimation unit estimates one or more emotions of the user based on the corrected emotion information and the character emotion information.
The emotion estimation device according to claim 3. If a value indicating the degree of difference between the plurality of voice evaluation values included in the corrected emotion information of the user and the plurality of character evaluation values included in the text emotion information is equal to or less than a predetermined value, the recognized character string is specified. a specific part for specifying as a character string,
When another user who does not speak a voice that expresses an explicit emotion speaks the specific character string,
The voice evaluation unit learns the voice emotion information of the other user by inputting a plurality of feature amounts according to the voice of the specific character string uttered by the other user to the learning model. obtained from the model,
The character evaluation unit
generating character emotion information of the other user based on the voice of the specific character string uttered by the other user;
The purpose of the adjustment unit is to bring the plurality of voice evaluation values included in the corrected emotion information of the other user closer to the plurality of character evaluation values included in the text emotion information of the other user for each of the plurality of emotions. adjusting the correction information for the other user;
The emotion estimation device according to claim 4. a noise removal unit that removes noise based on a predetermined threshold value from the sound indicated by the sound information to generate the audio information;
If a value indicating the degree of difference between the plurality of voice evaluation values included in the corrected emotion information of the user and the plurality of character evaluation values included in the text emotion information is equal to or less than a predetermined value, the recognized character string is specified. a specific part for specifying as a character string,
When another user who does not speak a voice that expresses an explicit emotion speaks the specific character string,
The voice evaluation unit learns the voice emotion information of the other user by inputting a plurality of feature amounts according to the voice of the specific character string uttered by the other user to the learning model. obtained from the model,
The character evaluation unit generates character emotion information of the other user based on the voice of the specific character string uttered by the other user,
The purpose of the adjustment unit is to bring the plurality of voice evaluation values included in the corrected emotion information of the other user closer to the plurality of character evaluation values included in the text emotion information of the other user for each of the plurality of emotions. adjusting the predetermined threshold for the other user at
The emotion estimation device according to claim 4. An emotion estimation system comprising a server device and a terminal device capable of communicating with the server device,
The server device
a first communication device that receives sound information indicating sound including the user's voice;
a noise removing unit that removes noise from the sound indicated by the sound information to generate voice information indicating the voice of the user;
For a learning model that has been trained on a plurality of humans , a relationship between a plurality of feature quantities corresponding to human speech and the intensity of each of a plurality of emotions held by the person who uttered the speech is evaluated based on the speech information. a voice evaluation unit for inputting the feature quantity of and acquiring from the learning model voice emotion information including a voice evaluation value indicating the strength of each of the plurality of emotions held by the user;
speech recognition processing is performed on the sound information to recognize the utterance content of the voice uttered by a human, and the plurality of emotions of the user are identified based on the recognition character string indicating the recognition result of the speech recognition processing. a character evaluator for generating character emotion information including character evaluation values indicative of strength for each;
The first communication device is
transmitting the text emotion information and the voice emotion information to the terminal device;
The terminal device
a sound collecting device that collects sound including the user's voice;
a second communication device that transmits the sound information output by the sound collecting device to the server device and receives the text emotion information and the voice emotion information from the server device;
a correcting unit that generates corrected emotional information obtained by correcting the voice emotional information using correction information based on the features of the voice of the user;
an estimation unit for estimating one or more emotions of the user based on the corrected emotion information and the character emotion information;
emotion estimation system. The speech evaluation unit inputs, to the learning model, a plurality of feature amounts based on speech information indicating a speech in which the user explicitly expresses one of the plurality of emotions, and obtaining explicit speech affective information from the learning model;
The terminal device
an adjustment unit that adjusts the correction information based on the explicit voice emotion information for the purpose of increasing the possibility that the estimation unit estimates that the emotion held by the user is the one emotion;
The emotion estimation system of claim 7, comprising: The terminal device is a first terminal device,
The user is a first user,
The sound collector is a first sound collector,
The server device is capable of communicating with a second terminal device different from the first terminal device,
The second communication device is
transmitting the corrected emotion information of the first user to the server device;
The server device
when a value indicating the degree of difference between the plurality of voice evaluation values included in the corrected emotion information of the first user and the plurality of character evaluation values included in the text emotion information is equal to or less than a predetermined value, the recognized character string as a specific character string,
When the second user who possesses the second terminal device does not utter a voice that expresses an explicit emotion and utters the specific character string,
The speech evaluation unit learns the speech emotional information of the second user by inputting a plurality of feature quantities corresponding to speech in which the second user utters the specific character string to the learning model. obtained from the model,
The character evaluation unit generates character emotion information of the second user based on the voice of the specific character string uttered by the second user,
The first communication device is
transmitting the specific character string, the second user's voice emotion information, and the second user's text emotion information to the second terminal device;
The second terminal device
a second sound collecting device that collects sound including the voice of the second user;
a third communication device that transmits sound information output by the second sound collecting device to the server device, and receives text emotional information of the second user and voice emotional information of the second user from the server device;
a correction unit that generates corrected emotional information of the second user by correcting the voice emotional information of the second user using correction information based on the characteristics of the voice of the user;
For the purpose of bringing the plurality of voice evaluation values included in the corrected emotion information of the second user closer to the plurality of text evaluation values included in the text emotion information of the second user for each of the plurality of emotions, the second an adjustment unit that adjusts the correction information for a user,
The emotion estimation system according to claim 8.

Images