JP2020201334A - Emotion estimation device and emotion estimation system - Google Patents

Abstract

To highly precisely estimate emotion which a user has even if a general-purpose learning model which is learnt is used with voice information on a plurality of persons as teacher data.SOLUTION: An emotion estimation device comprises: a voice evaluation section for inputting a plurality of feature amounts on the basis of voice information indicating user's voice to learning models which have learnt a relation between the plurality of feature amounts corresponding to human voice and intensity to a plurality of emotions which the person emitting the voice has on the plurality of persons and acquiring voice emotion information including a voice evaluation value indicating the intensity of the plurality of emotions which the user has from the learning model; a correction section for generating corrected emotion information obtained by correcting the voice emotion information by using the correction information on the basis of the feature of the user's voice; and an estimation section for estimating one or more emotions which the user has from the plurality of emotions on the basis of the corrected emotion information.SELECTED DRAWING: Figure 5

Description

The present invention relates to an emotion estimation device and an emotion estimation system.

In recent years, services for estimating emotions such as joy, anger, and sadness have become widespread. For example, Patent Document 1 discloses an emotion estimation device that estimates an emotion held by a user based on voice information indicating a user's voice. This emotion estimator is an average of each of a plurality of features such as frequency, volume, and speed, which are unique data of this user who performs voice recognition from a plurality of voice information input by one user. Calculate the value and standard deviation in advance. Then, this emotion estimation device normalizes the feature amount of the voice information input when estimating the emotion held by the user using the mean value and the standard deviation calculated in advance, and converts it into a plurality of normalized feature amounts. Estimate the emotions that this user has based on.

Japanese Unexamined Patent Publication No. 2006-259641

However, when the above-mentioned conventional technique is applied to a device for estimating emotions held by a user by using a learning model in which the relationship between a plurality of features based on voice information and emotions has been learned, the learning model is applied to each user. I had to prepare for it. When a general-purpose learning model that has been trained using the voice information of many users as teacher data is used, the difference between the average voice characteristics of many users and the voice characteristics of users is not absorbed, so that the emotions that the users have. Could not be estimated accurately.

In the emotion estimation device according to the preferred embodiment of the present invention, the relationship between a plurality of feature quantities corresponding to a human voice and the intensity of each of the plurality of emotions held by the human who emits the voice has been learned for a plurality of humans. A plurality of feature quantities based on the voice information indicating the user's voice are input to the learning model, and voice emotion information including a voice evaluation value indicating the intensity of each of the plurality of emotions held by the user is obtained from the learning model. A voice evaluation unit to be acquired, a correction unit that generates corrected emotion information obtained by correcting the voice emotion information using correction information based on the characteristics of the user's voice, and a correction unit of the plurality of emotions based on the corrected emotion information. It includes an estimation unit that estimates one or more emotions held by the user from the inside.

The emotion estimation system according to a preferred embodiment of the present invention is an emotion estimation system including a server device and a terminal device capable of communicating with the server device, and the server device is a sound indicating a sound including a user's voice. A first communication device that receives information, a noise removing unit that removes noise from the sound indicated by the sound information to generate audio information indicating the user's voice, and a plurality of feature quantities corresponding to the human voice. And the relationship between the intensity of each of the plurality of emotions held by the person who emitted the sound and the intensity of each of the plurality of emotions held by the user. The sound information includes a voice evaluation unit that acquires voice emotion information including a voice evaluation value indicating the intensity of each of the plurality of emotions from the learning model, and a voice recognition process that recognizes the utterance content of a human-generated sound. A character evaluation unit that executes the sound recognition process and generates character emotion information including a character evaluation value indicating the intensity of each of the plurality of emotions held by the user based on the recognition character string indicating the recognition result of the voice recognition process. The first communication device transmits the character emotion information and the voice emotion information to the terminal device, and the terminal device collects sounds including the user's voice, and the sound collecting device. A second communication device that transmits the sound information output by the sound collecting device to the server device and receives the character emotion information and the voice emotion information from the server device, and correction information based on the characteristics of the user's voice. It is provided with a correction unit that generates corrected emotion information that corrects the voice emotion information using the above, and an estimation unit that estimates one or more emotions held by the user based on the corrected emotion information and the character emotion information. ..

According to the present invention, even when a learned learning model using a plurality of human voice information as teacher data is used, the emotions held by the user can be estimated with high accuracy.

The figure which shows the outline of the function of the user apparatus 1. The block diagram which shows the structure of the user apparatus 1 which concerns on 1st Embodiment. The figure which shows an example of the storage contents of the dictionary information 31 for analysis. The figure which shows an example of the memory content of the emotion classification information 33. The figure which shows the outline of the function of the user apparatus 1. The figure which shows the flowchart which shows the operation of the user apparatus 1. The block diagram which shows the user apparatus 1a which concerns on 2nd Embodiment. The figure which shows the outline of the function of the user apparatus 1a which concerns on 2nd Embodiment. The figure which shows the flowchart which shows the operation of the user apparatus 1a in the calibration mode. The figure which shows the whole structure of the emotion estimation system SYS. The block diagram which shows the structure of the user apparatus 1b. The block diagram which shows the structure of the server apparatus 10. The figure which shows the outline of the function of the emotion estimation system SYS. The figure which shows the outline of the adjustment function of the correction information CI of a non-calibration user. The figure which shows the flowchart which shows the operation of the emotion estimation system SYS in the emotion estimation mode (the 1). The figure which shows the flowchart which shows the operation of the emotion estimation system SYS in the emotion estimation mode (the 2). The figure which shows the whole structure of the emotion estimation system SYSc. The block diagram which shows the structure of the server apparatus 10C. The figure which shows the outline of the function of the emotion estimation system SYSc. The figure which shows the outline of the adjustment function of the parameter information TI of a non-calibration user. The figure which shows the flowchart which shows the operation of the emotion estimation system SYSc in the emotion estimation mode. The figure which shows the whole structure of the emotion estimation system SYSd. The block diagram which shows the structure of the user apparatus 1d. The block diagram which shows the structure of the server apparatus 10D. The figure which shows the outline of the function of the emotion estimation system SYSd. The figure which shows the whole structure of the emotion estimation system SYSTEM. The block diagram which shows the structure of the user apparatus 1e. The figure which shows the outline of the function of the emotion estimation system SYSTEM. The figure which shows the whole structure of the emotion estimation system SYSf. The block diagram which shows the structure of the user apparatus 1f. The block diagram which shows the structure of the server apparatus 10F. The figure which shows the outline of the function of the 1st emotion estimation part 25f and the 2nd emotion estimation part 25F. The figure which shows the outline of the function of the user apparatus 1g in the 1st modification.

1. 1. 1st Embodiment FIG. 1 is a diagram showing an outline of a function of a user device 1. The user device 1 is assumed to be a smartphone. The user device 1 is an example of an “emotion estimation device”. However, as the user device 1, any information processing device can be adopted, and for example, it may be a terminal-type information device such as a personal computer, or a portable type such as a notebook computer, a wearable terminal, or a tablet terminal. It may be an information terminal of.

The user device 1 has a function of transmitting a recognition character string obtained by executing a voice recognition process on sound information indicating a sound including a sound of a user U who owns the user device 1 to a device used by another person. Or, it has a function of emitting a sound indicating a recognition character string in order to let another person located near the user U hear it. Further, the user device 1 estimates the emotion held by the user U based on the voice of the user U, adds a figure corresponding to the estimated emotion to the recognition character string, or adds the estimated emotion to the recognition character string. Emotional expressions necessary for communication can be added by emitting a sound indicating a recognition character string with intonation according to.
In the example of FIG. 1, a user U is say "Hello", the user device 1, appended figures PI according to the estimated emotions.

FIG. 2 is a block diagram showing a configuration of the user device 1 according to the first embodiment. The user device 1 is realized by a computer system including a processing device 2, a storage device 3, a display device 4, an operating device 5, a communication device 6, a sound emitting device 7, and a sound collecting device 8. Each element of the user device 1 is connected to each other by a single unit or a plurality of buses 9 for communicating information. The term "device" in the present specification may be read as another term such as a circuit, a device, or a unit. Further, each element of the user device 1 may be composed of a single device or a plurality of devices, and some elements of the user device 1 may be omitted.

The processing device 2 is a processor that controls the entire user device 1, and is composed of, for example, a single chip or a plurality of chips. The processing device 2 is composed of, for example, a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, registers, and the like. Part or all of the functions of the processing device 2 are realized by hardware such as DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), FPGA (Field Programmable Gate Array), etc. You may. The processing device 2 executes various processes in parallel or sequentially.

The storage device 3 is a recording medium that can be read by the processing device 2, and stores a plurality of programs including the control program PR executed by the processing device 2, analysis dictionary information 31, emotion classification information 33, and learning model LM. To do. The storage device 3 is composed of, for example, one or more types of storage circuits such as ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), and RAM (Random Access Memory).

FIG. 3 is a diagram showing an example of the stored contents of the analysis dictionary information 31. The analysis dictionary information 31 is information in which part of speech, part of speech subclassification, and original form information are associated with each other for each morpheme. A morpheme is a character string that is the smallest unit of a meaningful expression element. Part of speech is a type of word classified according to its grammatical nature, such as nouns, verbs, and adjectives. Part of speech subclassification is an item in which part of speech is further subdivided. The original form information is a character string indicating the original form of the word when the word is utilized by the corresponding morpheme, and is the same character string as the corresponding morpheme when the word is not utilized by the relevant morpheme.

FIG. 4 is a diagram showing an example of the stored contents of the emotion classification information 33. The emotion classification information 33 is information in which the character string is classified into any of joy, anger, sadness, and normal. The character string registered in the emotion classification information 33 represents an emotion of joy, anger, sadness, or normality. In the example of FIG. 4, the character string group 331 classified as joy includes "happy", "pass", "win", "win", and the like. Similarly, the character string group 332 classified as anger includes "irritated", "mucked up" and the like. Similarly, the character string group 333 classified as sad includes "sad", "losing", and the like. Similarly, the character string group 334 normally classified includes "safety" and the like.

The explanation is returned to FIG. The learning model LM has learned the relationship between a plurality of features corresponding to human voice and the intensity of each of the plurality of emotions.

The display device 4 displays various images under the control of the processing device 2. For example, various display panels such as a liquid crystal display panel or an organic EL (Electro Luminescence) display panel are preferably used as the display device 4.

The operation device 5 is a device for inputting information used by the user device 1. The operation device 5 accepts an operation by the user U. Specifically, the operating device 5 accepts an operation for inputting a code such as a number and a character and an operation for selecting an icon displayed by the display device 4. For example, a touch panel that detects contact with the display surface of the display device 4 is suitable as the operation device 5. The operation device 5 may include an operator that can be operated by the user. The operator is, for example, a stylus.

The communication device 6 is hardware (transmission / reception device) for communicating with another device via a network. The communication device 6 is also called, for example, a network device, a network controller, a network card, a communication module, or the like.

The sound emitting device 7 is composed of, for example, a speaker, and emits sound under the control of the processing device 2. The sound collecting device 8 is composed of, for example, a microphone and an AD converter, and collects sounds including the voice of the user U under the control of the processing device 2. The microphone converts the collected voice into an electric signal. The AD converter AD-converts the electric signal converted by the microphone and converts it into the sound information SI shown in FIG. The sound indicated by the sound information SI may include noise emitted from the surroundings of the speaker in addition to the voice of the speaker.

1.1. The function processing device 2 of the first embodiment functions as an acquisition unit 21, an emotion estimation unit 25, and an output unit 26 by reading and executing the control program PR from the storage device 3.
The function realized by the processing apparatus 2 will be described with reference to FIG.

FIG. 5 is a diagram showing an outline of the functions of the user device 1. The acquisition unit 21 acquires the sound information SI output by the sound collecting device 8 that collects sounds including the voice of the user U. The emotion estimation unit 25 estimates one or more emotions held by the user U from a plurality of emotions held by the user U. In the first embodiment, the plurality of emotions held by the user U will be described as four emotions: joy, anger, sadness, and normality. Below, joy, anger, sadness, and normality are examples of multiple emotions.

The emotion estimation unit 25 includes a noise removal unit 251, a voice evaluation unit 252, a correction unit 253, a character evaluation unit 256, and an estimation unit 258.

The noise removing unit 251 removes noise from the sound indicated by the sound information SI to generate the voice information VI. For example, a first parameter P1 and a second parameter P2 are given to the noise removing unit 251. The first parameter P1 specifies a frequency band to be regarded as noise. The second parameter P2 specifies the magnitude of the amplitude component regarded as noise. The noise removing unit 251 executes the first to fourth processes. In the first process, the amplitude component is calculated for each of the plurality of frequency bands by performing the fast Fourier transform process on the sound information SI. In the second process, the amplitude component of the frequency band specified by the first parameter P1 is reduced. The frequency of human voice is generally 100 Hz or more and 2000 Hz or less. The first parameter P1 specifies the lower limit frequency and the upper limit frequency. Therefore, when the noise removing unit 251 uses the first parameter P1, the amplitude component is reduced in the frequency band below the lower limit frequency, and the amplitude component is reduced in the frequency band above the lower limit frequency. The third process reduces the amplitude component of the magnitude or less specified by the second parameter P2. In the fourth process, the processing result of the third process is subjected to an inverse Fourier transform process to generate voice information VI. The voice information VI indicates the voice of the user U in which environmental noise and the like are removed from the sound information SI.

The voice evaluation unit 252 inputs a plurality of feature quantities based on the voice information VI into the learning model LM, and inputs the voice emotion information VE including the voice evaluation value x indicating the intensity for each of the plurality of emotions from the learning model LM. get.
In the learning model LM, the relationship between a plurality of features corresponding to a human voice and the intensity of each of the plurality of emotions held by the person who emits the voice has been learned for the plurality of humans. The learning model LM learns a large amount of teacher data in the learning process. The teacher data is given as a set of a plurality of features which are input data and an intensity for each of a plurality of emotions which are label data. Further, the teacher data is generated based on the voice information VI of a large number of users. In other words, the learning model LM is a general purpose model that is not tailored to a particular individual.
The plurality of features are sound features, for example, MFCC (Mel-Frequency Cepstrum Coefficients) 12-dimensional, loudness, fundamental frequency (F0), voice probability, zero crossover rate, HNR (Harmonics-to-Noise-Ratio). ), And these first-order differentials, MFCC and loudness second-order differentials, for a total of 47. Loudness is the loudness of a sound, which indicates the intensity of the sound felt by human hearing. The voice probability indicates the probability that the sound indicated by the voice information VI includes voice. The zero crossing rate is the number of times the sound pressure becomes zero.
The voice evaluation unit 252 performs a process of extracting sound features on the voice information VI to generate a plurality of feature quantities.

The voice emotion information VE includes a voice evaluation value of joy x1, a voice evaluation value of anger x2, a voice evaluation value of sadness x3, and a normal voice evaluation value x4. The voice evaluation value x is a real number of 0 or more. In the following description, when distinguishing the same kind of elements, reference codes are used such as joy voice evaluation value x1 and anger voice evaluation value x2. On the other hand, when the same type of elements are not distinguished, only the common number among the reference codes is used, such as the voice evaluation value x.

The correction unit 253 generates the corrected emotion information CVE in which the voice emotion information VE is corrected by using the correction information CI based on the voice characteristics of the user U. The correction information CI includes, for example, a coefficient k1 for correcting the voice evaluation value x1 of joy, a coefficient k2 for correcting the voice evaluation value x2 for anger, a coefficient k3 for correcting the voice evaluation value x3 for sadness, and a normal voice evaluation value. Includes a coefficient k4 that corrects x4. k1 to k4 are real numbers of 0 or more. The corrected emotion information CVE includes a joy voice evaluation value X1, an anger voice evaluation value X2, a sadness voice evaluation value X3, and a normal voice evaluation value X4. The correction unit 253 generates the correction emotion information CVE according to, for example, the following equation.

X1 = x1 × k1
X2 = x2 x k2
X3 = x3 × k3
X4 = x4 x k4

The method for generating the correction information CI has, for example, the following two aspects. In the first aspect, the user U speaks to the sound collecting device 8 in a normal state. The processing device 2 extracts a plurality of feature quantities from the voice information VI corresponding to the utterance, and compares the extracted plurality of feature quantities with a predetermined threshold value to generate coefficients k1 to k4. For example, when the extracted fundamental frequency is higher than a predetermined threshold value, the user U has a higher fundamental frequency even in normal times, and it is easy to erroneously determine that the emotion held by the user U is joy or anger. Therefore, the processing device 2 sets the coefficient k1 corresponding to joy and the coefficient k2 corresponding to anger to values larger than 0 and smaller than 1 for the purpose of lowering the voice evaluation value X1 of joy and the voice evaluation value X2 of anger. To do.

In the second aspect, the processing device 2 causes the user U to input information regarding the characteristics of his / her voice. For example, the processing device 2 causes the user U to input the gender and age as information regarding the characteristics of his / her voice. When the gender is female, the fundamental frequency is generally higher than that of male, so that the processing device 2 is delighted for the purpose of lowering the joy voice evaluation value X1 and the anger voice evaluation value X2. The corresponding coefficient k1 and the corresponding coefficient k2 for anger are set to values greater than 0 and less than 1. Similarly, in general, the younger the age, the louder the voice. Therefore, when the input age is equal to or less than a predetermined threshold value, the processing device 2 sets the joy voice evaluation value X1 and the anger voice evaluation value X2. For the purpose of lowering, the coefficient k1 corresponding to joy and the coefficient k2 corresponding to anger are set to values larger than 0 and smaller than 1.

The character evaluation unit 256 executes a voice recognition process for recognizing the utterance content of the voice uttered by a human to the sound information SI, and based on the recognition character string RT indicating the recognition result of the voice recognition process, of a plurality of emotions. The character emotion information TE including the character evaluation value Y indicating the strength for each is generated. The character emotion information TE includes a joy character evaluation value Y1, an anger character evaluation value Y2, a sad character evaluation value Y3, and a normal character evaluation value Y4. The character evaluation value Y is a real number of 0 or more.

More specifically, the character evaluation unit 256 includes a voice recognition processing unit 2561, a morphological analysis processing unit 2563, and an evaluation value calculation unit 2565.
The voice recognition processing unit 2561 performs voice recognition processing on the sound information SI and outputs the recognition character string RT. The voice recognition processing unit 2561 outputs the recognition character string RT by various methods including a method of recognizing a character string from the voice by using, for example, an acoustic model and a language model prepared in advance.

The morphological analysis processing unit 2563 executes the morphological analysis process on the recognition character string RT with reference to the analysis dictionary information 31, and outputs the corrected recognition character string CRT. The morphological analysis process is a process of decomposing the recognition character string RT for each morpheme. In the morphological analysis process, the part of speech and the part of speech subclassification of the analysis dictionary information 31 are used. The corrected recognition character string CRT is a character string excluding a character string such as a filler, which is unnecessary for estimating the emotion held by the user U. Filler is a word that is inserted between utterances, such as "um", "that", and "well".

The evaluation value calculation unit 2565 calculates the character evaluation value Y of each emotion by comparing the character string included in the emotion classification information 33 with the corrected recognition character string CRT, and includes the character evaluation value Y of each emotion. Generates character emotion information TE. Regarding the calculation of the character evaluation value Y of each emotion, the evaluation value calculation unit 2565 includes the corrected recognition character string CRT when the corrected recognition character string CRT includes the character string included in the emotion classification information 33. Increases the character evaluation value Y of the emotion corresponding to the character string.
For example, if the corrected recognition character string CRT is "winning the game today", the evaluation value calculation unit 2565 outputs the character evaluation value Y of each emotion as follows.

Character evaluation value of joy Y1 1
Anger character evaluation value Y2 0
Character evaluation value of sadness Y30
Normal character evaluation value Y40

In the above example, since the corrected recognition character string CRT includes the "win" included in the emotion classification information 33, the evaluation value calculation unit 2565 sets the joy character evaluation value Y1 corresponding to the "win". Increase by 1. The amount of increase in the character evaluation value Y to be increased is not limited to 1, and may differ for each character string included in the emotion classification information 33. For example, the amount of increase in the character evaluation value Y with respect to the character string indicating more joy may be set to 2. Further, when the corrected recognition character string CRT includes a character string included in the emotion classification information 33 and a character string that emphasizes the content, the evaluation value calculation unit 2565 increases the amount of increase in the character evaluation value Y. You may. For example, if the corrected recognition character string CRT is "very happy to win the game today", the corrected recognition character string CRT contains "happy" included in the emotion classification information 33 and is "very". Since the character string emphasizing the content is included, the evaluation value calculation unit 2565 increases, for example, the character evaluation value Y1 of joy by 2. Which character string of the corrected recognition character string CRT is a character string that emphasizes the content can be determined by the morpheme obtained by the morphological analysis process. In the following example, for ease of explanation, it is assumed that the amount of increase in the character evaluation value Y to be increased is 1.
Further, when the corrected recognition character string CRT includes a character string included in the emotion classification information 33 and a character string denying the content, the evaluation value calculation unit 2565 is included in the corrected recognition character string CRT. A process different from the process of increasing the character evaluation value Y corresponding to the character string may be executed. For example, if the corrected recognition character string CRT is "could not win the game today", the corrected recognition character string CRT includes "win" included in the emotion classification information 33, but is said to be "not". Since the character string denying the content is included, the evaluation value calculation unit 2565 increases, for example, the character evaluation value Y3 of sadness by 1. Which character string of the corrected recognition character string CRT is a character string whose content is negated can be determined by the morpheme obtained by the morphological analysis process. In this way, it is possible to estimate whether the corrected recognition character string CRT has positive content or negative content by the morphological analysis process. In the following example, if the corrected recognition character string CRT includes the character string included in the emotion classification information 33, the character corresponding to the character string included in the corrected recognition character string CRT is included in the following example. The explanation will be given as increasing the evaluation value Y.

The estimation unit 258 estimates one or more emotions held by the user U based on the corrected emotion information CVE and the character emotion information TE. For example, the estimation unit 258 adds the voice evaluation values X1 to X4 of the corrected emotion information CVE and the character evaluation values Y1 to Y4 of the character emotion information TE for each emotion for each of the plurality of emotions, and adds each emotion. Calculate the value. The estimation unit 258 compares the added value for each emotion with the threshold value, and identifies the added value exceeding the threshold value. The estimation unit 258 estimates one or more emotions corresponding to the specified addition value as one or more emotions held by the user U. In the following description, the addition of the voice evaluation value X and the character evaluation value Y means that each emotion is added to generate four addition values.
The estimation unit 258 is not limited to simply adding the voice evaluation value X and the character evaluation value Y, but is a value obtained by multiplying one of the voice evaluation value X and the character evaluation value Y by a predetermined value α. And the other evaluation value may be added. The predetermined value α is, for example, a value set by the developer of the user device 1, the user U, or the like.

The estimation unit 258 outputs an estimated emotion information EI indicating one or more emotions held by the estimated user U. The estimated emotional information EI has, for example, the following two aspects. The first aspect of the estimated emotion information EI is an identifier indicating one or more emotions held by the estimated user U. The identifier indicating emotion includes an identifier indicating joy, an identifier indicating anger, an identifier indicating sadness, and an identifier indicating normality. The second aspect of the estimated emotion information EI is an identifier indicating one or more emotions held by the estimated user U and an evaluation value of the emotions held by the estimated user U. The evaluation value of the emotions held by the estimated user U is, for example, a value obtained by adding the voice evaluation value X of the corrected emotion information CVE and the character evaluation value Y for each emotion for one or more emotions held by the estimated user U. ..

The output unit 26 outputs the data obtained by executing the processing corresponding to one or more emotions indicated by the estimated emotion information EI. For example, the output unit 26 has the following two modes. The output unit 26 in the first aspect executes data obtained by executing processing according to one or more emotions indicated by the estimated emotion information EI on the recognition character string RT obtained by the voice recognition processing unit 2561. Output. There are two modes of processing according to emotions, for example, as shown below.
The first aspect of the process according to the emotion is a process of adding a figure embodying the emotion to the recognition character string RT. The figures that embody emotions are, for example, pictograms that embody emotions and emoticons that embody emotions. A pictogram is an image associated with a character code. The character code is, for example, Unicode. An emoticon is a character string that expresses a face by combining symbols and characters. In the following description, a figure that embodies emotions will be described as a pictogram that embodies emotions. The pictogram that embodies joy is, for example, a pictogram that shows a smile. The pictogram that embodies anger is, for example, a pictogram that shows the face of anger. The pictogram that embodies sadness is, for example, a pictogram that shows a crying face. Further, when the estimated emotion information EI is the second aspect, the output unit 26 embodies the emotion indicated by the estimated emotion information EI and has an intensity corresponding to the evaluation value included in the estimated emotion information EI. The resulting pictogram may be determined as a pictogram to be added to the recognition character string RT. For example, when the emotion indicated by the estimated emotion information EI is sadness and the evaluation value included in the estimated emotion information EI is equal to or less than a predetermined threshold value, the output unit 26 recognizes a pictogram indicating a face spilling tears. Determined as a pictogram to be added to RT. On the other hand, when the emotion indicated by the estimated emotion information EI is sadness and the evaluation value included in the estimated emotion information EI is larger than a predetermined threshold value, the output unit 26 converts the pictogram indicating the crying face into the recognition character string RT. Determine as the pictogram to be added. The emoji showing a crying face embodies a higher degree of sadness than the emoji showing a tearful face.
The output unit 26 outputs a character string with a pictogram obtained by adding a pictogram to the recognition character string RT. For example, there are two positions for adding pictograms as shown below. The first position is the end of the recognition string RT. The second position is next to the character string included in the emotion classification information 33 in the recognition character string RT. The display device 4 displays an image based on the character string with pictograms output by the output unit 26.

The second aspect of the emotion-based process is a process of generating a synthetic voice read aloud by adding emotion-based intonation. Inflection is, for example, increasing or decreasing the reading speed, or increasing or decreasing the volume. Pleasure-based intonation is, for example, speeding up reading. Anger-based intonation is, for example, increasing the volume. Sadness-based intonation is, for example, reducing the volume. The output unit 26 outputs information indicating a synthetic voice read aloud with an emotion-based intonation added. Then, the output unit 26 adds emotion-based intonation to the synthetic voice indicated by the generated data, and outputs information indicating the synthetic voice read by adding emotion-based intonation. The sound emitting device 7 emits a synthetic voice indicated by the data output by the output unit 26.

The output unit 26 in the second aspect outputs a pictogram that embodies one or more emotions indicated by the estimated emotion information EI. In the output unit 26 in the second aspect, it is not necessary to use the recognition character string RT. In the following description, the output unit 26 will be described as the first aspect.

1.2. Operation of First Embodiment Next, the operation of the user device 1 will be described with reference to FIG.

FIG. 6 is a flowchart showing the operation of the user device 1. The processing device 2 generates the correction information CI according to one of the two methods for generating the correction information CI described above (step S1). Next, the acquisition unit 21 acquires the sound information SI (step S2). Then, the voice recognition processing unit 2561 executes voice recognition processing on the sound information SI and outputs the recognition character string RT (step S3). Next, the morphological analysis processing unit 2563 performs morphological analysis processing on the recognition character string RT with reference to the analysis dictionary information 31, and outputs the corrected recognition character string CRT (step S4). Then, the evaluation value calculation unit 2565 calculates the character evaluation value Y of each emotion by comparing the character string included in the emotion classification information 33 with the corrected recognition character string CRT, and the character evaluation value Y of each emotion. The character emotion information TE including the above is generated (step S5).

Further, the noise removing unit 251 removes noise from the sound indicated by the sound information SI according to the first parameter P1 and the second parameter P2 to generate the voice information VI (step S6). Then, the voice evaluation unit 252 extracts the feature amount of the sound from the voice information VI from which the noise has been removed (step S7). Next, the voice evaluation unit 252 inputs the feature amount of the sound into the learning model LM, and acquires the voice emotion information VE including the voice evaluation value x of each emotion from the learning model LM (step S8). The correction unit 253 uses the correction information CI to generate the correction emotion information CVE that corrects the voice evaluation value x of each emotion included in the voice emotion information VE (step S9).

The estimation unit 258 estimates the emotion held by the user U based on the corrected emotion information CVE and the character emotion information TE, and outputs the estimated emotion information EI (step S10). The output unit 26 outputs information obtained by executing processing according to the emotion indicated by the estimated emotion information EI with respect to the recognition character string RT (step S11). After the processing in step S11 is completed, the user device 1 ends a series of processing shown in FIG.

1.3. Effect of First Embodiment According to the above description, in order to execute the emotion estimation of the user U by using the general-purpose learning model LM, the user device 1 may generate a learning model adjusted for each individual. In comparison, the time required to generate the learning model LM can be shortened.
By inputting a plurality of features related to the average human voice into the general-purpose learning model LM, the emotions of the average human can be estimated. However, the voice of the user U is different from the average human voice because it is affected by the gender, age, characteristics of the user U's speaking style, and the like. Therefore, simply using the general-purpose learning model LM reduces the accuracy of determining the emotions held by the user U.
In the user device 1 described above, the voice emotion information VE output from the learning model LM is corrected by using the correction information CI based on the voice characteristics of the user U, so that while using the general-purpose learning model LM, The emotions held by the user U can be estimated with high accuracy.

Further, the user device 1 removes noise from the sound indicated by the sound information SI to generate the voice information VI, and inputs the feature amount of the sound based on the voice information VI into the learning model LM. By inputting the sound feature amount based on the voice information VI into the learning model LM, the voice emotion information VE with higher accuracy can be obtained as compared with the case where the sound feature amount based on the sound information SI is input to the learning model LM. Obtainable.

Further, since the user device 1 estimates one or more emotions held by the user U based on the corrected emotion information CVE and the character emotion information TE, the emotions held by the user U are estimated based only on the corrected emotion information CVE. Compared with the case, the emotion held by the user U can be estimated with high accuracy.

2. 2. Second Embodiment The user device 1a according to the second embodiment urges the user U to utter a voice that explicitly expresses emotions, and acquires the explicit voice emotion information VEa of the user U from the learning model LM. However, it differs from the user device 1 according to the first embodiment in that the correction information CI is adjusted for the purpose of increasing the possibility that the estimation unit 258 estimates that the emotion held by the user U is an explicit emotion. .. In the second embodiment, the user device 1a can take an emotion estimation mode for estimating the emotion held by the user U and a calibration mode for adjusting the correction information CI. Since the emotion estimation mode corresponds to the first embodiment, the description thereof will be omitted. Hereinafter, the user device 1a according to the second embodiment will be described. Regarding the elements whose actions or functions are equivalent to those of the first embodiment in the second embodiment illustrated below, the reference numerals are used in the above description, and detailed description of each is appropriately omitted.

2.1. Function of the second embodiment FIG. 7 is a block diagram showing a user device 1a according to the second embodiment. The user device 1a is realized by a computer system including a processing device 2a, a storage device 3a, a display device 4, an operating device 5, a communication device 6, a sound emitting device 7, and a sound collecting device 8. The storage device 3a is a recording medium that can be read by the processing device 2a, and stores a plurality of programs including the control program PRa executed by the processing device 2a, the analysis dictionary information 31, and the emotion classification information 33.

The processing device 2a functions as an acquisition unit 21a, an emotion estimation unit 25a, and an output unit 26 by reading and executing the control program PRa from the storage device 3a.

FIG. 8 is a diagram showing an outline of the function of the user device 1a according to the second embodiment. The emotion estimation unit 25a includes a noise removal unit 251, a voice evaluation unit 252, a correction unit 253, an adjustment unit 254, a character evaluation unit 256, and an estimation unit 258.

The acquisition unit 21a acquires sound information SIa indicating a sound including a voice in which the user U explicitly expresses one of the plurality of emotions. Specifically, when the user U sets the user device 1a to the calibration mode by operating the operation device 5, the processing device 2 explicitly expresses and pronounces one of the plurality of emotions. The screen prompting the user to do so is displayed on the display device 4. "One emotion" is hereinafter referred to as "explicit emotion". The acquisition unit 21a acquires the sound information SI acquired after displaying the above-mentioned screen as the sound information SIa indicating the sound including the voice in which the user U expresses the explicit emotion. Of the plurality of emotions, which emotion is set as the explicit emotion may be set in advance by the developer of the user device 1a, or may be selected by the user U from the plurality of emotions. Good.

The noise removing unit 251 removes noise from the sound indicated by the sound information SIa to generate the voice information VIa.

The voice evaluation unit 252 inputs the feature amount of the sound based on the voice information VIa into the learning model LM, and acquires the explicit voice emotion information VEa of the user U from the learning model LM.

The adjustment unit 254 adjusts the correction information CI based on the explicit voice emotion information VEa for the purpose of increasing the possibility that the estimation unit 258 estimates that the emotion held by the user U is an explicit emotion. For example, the adjustment unit 254 executes one or both of a process of increasing the coefficient k corresponding to the explicit emotion and a process of decreasing the coefficient k corresponding to the emotion other than the explicit emotion. For example, the adjusting unit 254 generates the coefficients k1 to k4 according to the following equation. However, regarding the ideal voice evaluation value X obtained when the user U utters a predetermined voice by expressing emotions, the voice evaluation value of joy is Xa1, the voice evaluation value of anger is Xa2, and the voice of sadness. The evaluation value is represented by Xa3, and the normal voice evaluation value is represented by Xa4.
k1 = Xa1 / x1
k2 = Xa2 / x2
k3 = Xa3 / x3
k4 = Xa4 / x4
However, the coefficient k does not necessarily have to match Xa / x.

2.2. Operation of the Second Embodiment Next, the operation of the user apparatus 1a in the calibration mode will be described with reference to FIG.

FIG. 9 is a flowchart showing the operation of the user device 1a in the calibration mode. The acquisition unit 21a acquires sound information SIa indicating a sound including a voice in which the user U expresses an explicit emotion (step S21). Next, the noise removing unit 251 removes noise according to the first parameter P1 and the second parameter P2 to generate voice information VIa (step S22). Then, the voice evaluation unit 252 extracts the feature amount of the sound from the voice information VIa from which the noise has been removed (step S23). Next, the voice evaluation unit 252 inputs the feature amount of the sound into the learning model LM, and acquires the voice emotion information VEa including the voice evaluation value x of each emotion from the learning model LM (step S24).

The adjusting unit 254 corrects a plurality of voice evaluation values x included in the explicit voice emotion information VEa by the same method as the correction unit 253 (step S25). Next, the adjustment unit 254 adjusts the correction information CI for the purpose of increasing the possibility that the estimation unit 258 estimates that the emotion held by the user U is an explicit emotion (step S26). After the processing in step S26 is completed, the user device 1a ends a series of processing shown in FIG.

2.3. Effect of the Second Embodiment According to the above description, when the user U utters a voice that expresses an explicit emotion, the user device 1a estimates that the emotion held by the user U is an explicit emotion. It has an aspect of adjusting the correction information CI based on the explicit voice emotion information VEa for the purpose of increasing the possibility of doing so. In this aspect, the correct answer of the emotion estimated by the voice emotion information VEa is known, and the corrected emotion information CVE for the user U who has adjusted the correction information CI is the correction for the user U who has not adjusted the correction information CI. Compared with the emotion information CVE, the estimation accuracy of the emotion held by the user U can be improved.
Further, even if the user U utters a voice that expresses an explicit emotion, the intensity of expressing the emotion in the voice differs between the users U. For example, one user U may have a high intensity of expressing emotions in voice, while another user U may have a low intensity of expressing emotions in voice. The correction information CI in the second embodiment also reflects the difference in the intensity of expressing emotions in voice. For example, a user U having a high intensity of expressing emotions in voice has a voice evaluation value x close to the above-mentioned ideal voice evaluation value X, and a coefficient k close to 1. On the other hand, the user U having a low intensity of expressing emotions in voice has a voice evaluation value x that is smaller than the above-mentioned ideal voice evaluation value X, and a coefficient k that is far from 1. As described above, the lower the intensity of the user U to be exposed, the farther the coefficient k is from 1, and the difference in the intensity of expressing the emotion to the voice is reflected in the correction information CI. Can be improved.

3. 3. Third Embodiment The emotion estimation system SYS according to the third embodiment sets the user device 1b to the calibration mode by the function shown in the second embodiment, and outputs the emotion estimation result of the user U who has expressed explicit emotions. It is different from the user device 1a according to the second embodiment in that the user device 1b is not set to the calibration mode by using the user device 1b and has a configuration for adjusting the correction information CI of the user U who does not express an explicit emotion. To do. In the following description, the user U in which the user device 1b is set to the calibration mode and the explicit emotion is expressed is referred to as a "calibrated user", and the user device 1b is not set in the calibration mode and the explicit emotion is expressed. A non-user U is referred to as an "uncalibrated user".
Hereinafter, the emotion estimation system SYS according to the third embodiment will be described. Regarding the elements whose actions or functions are the same as those of the second embodiment in the third embodiment illustrated below, the reference numerals are used in the above description, and detailed description of each is appropriately omitted.

3.1. Outline of the Third Embodiment FIG. 10 is a diagram showing the overall configuration of the emotion estimation system SYS. The emotion estimation system SYS includes a user device 1b owned by the user U, a network NW, and a server device 10. The user device 1 included in the emotion estimation system SYS is from the user device 1b1 to the user device 1bm. m is an integer of 2 or more. The user U who possesses the user device 1b1 is the user U1, and the user U who possesses the user device 1bm is the user Um.

In the following, for the sake of brevity, it is assumed that the user U1 is a calibrated user and the user U2 is a non-calibrated user.

FIG. 11 is a block diagram showing the configuration of the user device 1b. The user device 1b is realized by a computer system including a processing device 2b, a storage device 3b, a display device 4, an operating device 5, a communication device 6, a sound emitting device 7, and a sound collecting device 8. The storage device 3b is a recording medium that can be read by the processing device 2b, and stores a plurality of programs including the control program PRb executed by the processing device 2b.

The processing device 2b functions as an acquisition unit 21 and an output unit 26 by reading and executing the control program PRb from the storage device 3b.

FIG. 12 is a block diagram showing the configuration of the server device 10. The server device 10 is realized by a computer system including a processing device 2B, a storage device 3B, and a communication device 6B. Each element of the server device 10 is connected to each other by a single unit or a plurality of buses 9B for communicating information. The storage device 3B is a recording medium that can be read by the processing device 2B, and stores a plurality of programs including the control program PRB executed by the processing device 2B, analysis dictionary information 31, emotion classification information 33, and learning model LM. To do.

The processing device 2B functions as the emotion estimation unit 25B by reading and executing the control program PRB from the storage device 3B. The function of the emotion estimation system SYS will be described with reference to FIG.

FIG. 13 is a diagram showing an outline of the functions of the emotion estimation system SYS. The emotion estimation unit 25B includes a noise removal unit 251, a voice evaluation unit 252B, a correction unit 253, an adjustment unit 254, a character evaluation unit 256, an estimation unit 258, and a specific unit 259.

The acquisition unit 21 of the user device 1b1 acquires the sound information SI1 output by the sound collecting device 8 that collects the sound including the voice of the user U1. FIG. 14 shows an outline of the adjustment function of the correction information CI of the non-calibration user, which is a function realized by the processing device 2B.

FIG. 14 is a diagram showing an outline of the adjustment function of the correction information CI of the non-calibrated user. FIG. 14 shows a state in which the calibrated user U1 utters “Thank you” and the acquisition unit 21 of the user device 1b1 has acquired the sound information SI1.

The explanation is returned to FIG. With respect to the user U1, the noise removing unit 251 removes noise from the sound indicated by the sound information SI1 to generate the voice information VI1. The voice evaluation unit 252B inputs the feature amount of the sound extracted from the voice information VI1 into the learning model LM, and acquires the voice emotion information VE1 from the learning model LM. The correction unit 253 generates the correction emotion information CVE1 in which the voice emotion information VE1 is corrected by using the correction information CI1 based on the voice characteristics of the user U1. Further, the voice recognition processing unit 2561 executes the voice recognition processing on the sound information SI1 and acquires the recognition character string RT1 indicating the recognition result of the voice recognition processing. Subsequently, the morphological analysis processing unit 2563 performs morphological analysis processing on the recognition character string RT1 with reference to the analysis dictionary information 31, and outputs the corrected recognition character string CRT1. The evaluation value calculation unit 2565 generates the character emotion information TE1 by comparing the character string included in the emotion classification information 33 with the corrected recognition character string CRT1.
FIG. 14 shows a state in which the server device 10 generates the corrected emotion information CVE1 and the character emotion information TE1 based on the sound information SI1.

The specific unit 259 recognizes a recognition character string when the value indicating the degree of difference between the plurality of voice evaluation values X included in the corrected emotion information CVE1 and the character evaluation value Y included in the character emotion information TE1 is equal to or less than a predetermined value. RT1 is specified as a specific character string ST. The character string that is easily specified as the specific character string ST is a character string that is often spoken in the original meaning of this character string, and is, for example, "Thank you" and "Don't be silly".
However, the word "thank you" is sometimes spoken as a social decree or ironically, and may not be spoken in the original meaning of "thank you". In this case, since the voice information VI1 does not show joy, the voice evaluation value X and the character evaluation value Y are significantly different. Therefore, the server device 10 stores the recognized character string RT of the calibrated user, the voice evaluation value X, the character evaluation value Y, and the date and time when the voice information VI1 is generated in association with each other as a log, and stores the specific unit 259. May specify the specific character string ST based on the tendency of the degree of difference between the voice evaluation value X and the character evaluation value Y with respect to the recognition character string RT with reference to this log. For example, the specific unit 259 recognizes that the ratio of the value indicating the degree of difference between the voice evaluation value X and the character evaluation value Y from the current time to a certain time in the past is equal to or less than a predetermined ratio. The character string RT is specified as a specific character string ST.
The value indicating the degree of difference has, for example, the following two aspects. The value indicating the degree of difference in the first aspect is Sum _XY, which is the sum of the squares of the differences between the voice evaluation value X and the character evaluation value Y for each of the plurality of emotions. The sum Sum _XY is obtained by, for example, the following equation (1).
Sum _XY = (X1-Y1) ² + (X2-Y2) ² + (X3-Y3) ² + (X4-Y4) ² (1)
The value indicating the degree of difference in the second aspect is the angle θ of the corrected emotion information CVE1 and the character emotion information TE1 when the corrected emotion information CVE1 and the character emotion information TE1 are regarded as a four-dimensional vector. It can be said that the larger the angle θ, the more different the corrected emotion information CVE1 and the character emotion information TE1. For example, the angle θ is obtained by the following equation (2).
θ = cos ^-1 ((CVE1 ・ TE1) / (| CVE1 | × | TE1 |)) (2)
However, CVE1 and TE1 indicate the inner product of the corrected emotion information CVE1 and the character emotion information TE1. | CVE1 | indicates the magnitude of the corrected emotion information CVE1. | TE1 | indicates the size of the character emotion information TE1.
In the following description, it is assumed that the value indicating the degree of difference is the sum Sum _XY .
FIG. 14 shows an example in which the sum Sum _XY is equal to or less than a predetermined value. Therefore, the specific unit 259 specifies "Thank you", which is the recognition character string RT1, as the specific character string ST.

Regarding the user U2, as shown in FIG. 14, it is assumed that the user U2 utters "Thank you" which is a specific character string ST. The acquisition unit 21 of the user device 1b2 acquires the sound information SI2. The noise removing unit 251 removes noise from the sound indicated by the sound information SI2 to generate the voice information VI2. The voice evaluation unit 252B inputs the feature amount of the sound extracted from the voice information VI2 into the learning model LM, and acquires the voice emotion information VE2 from the learning model LM. The correction unit 253 generates the correction emotion information CVE2 in which the voice emotion information VE2 is corrected by using the correction information CI2 based on the voice characteristics of the user U2. Further, the voice recognition processing unit 2561 executes the voice recognition processing on the sound information SI2 and acquires the recognition character string RT2 indicating the recognition result of the voice recognition processing. Subsequently, the morphological analysis processing unit 2563 executes the morphological analysis processing on the recognition character string RT2 with reference to the analysis dictionary information 31, and outputs the corrected recognition character string CRT2. The evaluation value calculation unit 2565 generates the character emotion information TE2 by comparing the character string included in the emotion classification information 33 with the corrected recognition character string CRT2.
FIG. 14 shows a state in which the server device 10 generates the corrected emotion information CVE2 and the character emotion information TE2 based on the sound information SI2.

When the user U2 who is a non-calibration user utters a specific character string ST, the adjustment unit 254 sets a plurality of voice evaluation values X included in the corrected emotion information CVE2 of the user U2 for each of the plurality of emotions. , The correction information CI2 for the user U2 is adjusted for the purpose of approaching the plurality of character evaluation values Y included in the character emotion information TE2 of the user U2. For example, the adjusting unit 254 generates the coefficients k1 to k4 according to the following equation.
k1 = Y1 / X1
k2 = Y2 / X2
k3 = Y3 / X3
k4 = Y4 / X4
However, the coefficient k does not necessarily have to match Y / X.

3.2. Operation of the Third Embodiment Similar to the second embodiment, in the third embodiment, the user device 1b can take an emotion estimation mode for estimating the emotion of the user U and a calibration mode for adjusting the correction information CI. .. When the user device 1b is set to the calibration mode, the server device 10 may acquire the sound information SI shown in step S21 from the user device 1b and execute each step after step S21. After completing the series of processes shown in FIG. 9, the server device 10 stores the identification information of the user device 1b set in the calibration mode in the storage device 3B as the user device 1b possessed by the calibrated user. The identification information of the user device 1b is, for example, a UID (User IDentifier), a MAC (Media Access Control) address, an IMSI (International Mobile Subscriber Identity) recorded in a subscriber authentication module (SIM), or a user ID. And so on. The UID is an ID assigned to each user by the service provider. The operation of the emotion estimation system SYS in the emotion estimation mode will be described with reference to FIGS. 15 and 16.

15 and 16 are flowcharts showing the operation of the emotion estimation system SYS in the emotion estimation mode. The server device 10 acquires the correction information CI from the user device 1b (step S31). Specifically, the user device 1b generates the correction information CI according to one of the two generation methods of the correction information CI described above, and transmits the correction information CI to the server device 10. Next, the server device 10 acquires the sound information SI from the user device 1b (step S32). Then, the voice recognition processing unit 2561 of the emotion estimation unit 25B executes the voice recognition processing on the sound information SI and outputs the recognition character string RT (step S33). Next, the morphological analysis processing unit 2563 of the emotion estimation unit 25B executes the morphological analysis processing on the recognition character string RT with reference to the analysis dictionary information 31, and outputs the corrected recognition character string CRT ( Step S34). Then, the evaluation value calculation unit 2565 of the emotion estimation unit 25B calculates the character evaluation value Y of each emotion by comparing the character string included in the emotion classification information 33 with the corrected recognition character string CRT, and calculates each emotion. The character emotion information TE including the character evaluation value Y of is generated (step S35).

Further, the noise removing unit 251 of the emotion estimation unit 25B removes noise from the sound indicated by the sound information SI according to the first parameter P1 and the second parameter P2 to generate the voice information VI (step S41). Then, the voice evaluation unit 252B of the emotion estimation unit 25B extracts the feature amount of the sound from the voice information VI from which the noise has been removed (step S42). Next, the voice evaluation unit 252B of the emotion estimation unit 25B inputs the feature amount of the sound into the learning model LM, and acquires the voice emotion information VE including the voice evaluation value x of each emotion from the learning model LM (step S43). ..

Next, the server device 10 determines whether or not the user who possesses the user device 1b that is the source of the correction information CI and the sound information SI is a calibrated user (step S44). As a method of determining whether the user is a calibrated user or a non-calibrated user, the user device 1b identifies the user device 1b at the time of transmitting the correction information CI and at the time of transmitting the sound information SI, or both. Send information. When the received identification information of the user device 1b matches the identification information stored as the user device 1b possessed by the calibrated user, the server device 10 outputs a positive determination result and stores it in the storage device 3B. If it does not match the identification information, a negative judgment result is output.

When the determination result in step S44 is affirmative, the correction unit 253 of the emotion estimation unit 25B uses the correction information CI to generate a correction emotion information CVE that corrects the voice evaluation value x of each emotion included in the voice emotion information VE. (Step S45). Then, in the specific unit 259 of the emotion estimation unit 25B, is the sum _{XY of the} squares of the differences between the voice evaluation value X included in the corrected emotion information CVE and the character evaluation value Y included in the character emotion information TE Sum _XY equal to or less than a predetermined value? Whether or not it is determined (step S46).
When the determination result in step S44 is affirmative and the determination result in step S46 is affirmative, the identification unit 259 of the emotion estimation unit 25B specifies the recognition character string RT as the specific character string ST (step S47). Then, the estimation unit 258 of the emotion estimation unit 25B estimates the emotion held by the user U based on the corrected emotion information CVE and the character emotion information TE (step S61). On the other hand, even when the determination result in step S44 is affirmative and the determination result in step S46 is negative, the estimation unit 258 of the emotion estimation unit 25B executes the process of step S61.

When the determination result in step S44 is negative, that is, when the user U who possesses the user device 1b that is the source of the correction information CI and the sound information SI is a non-calibrated user, the server device 10 has a specific character string ST. It is determined whether or not the recognition character string RT matches (step S50).
When the determination result in step S44 is negative and the determination result in step S50 is affirmative, the specific unit 259 of the emotion estimation unit 25B includes the voice evaluation value X included in the corrected emotion information CVE in the character emotion information TE. The correction information CI for the non-calibrated user is adjusted for the purpose of approaching the plurality of character evaluation values Y (step S51). Then, the correction unit 253 of the emotion estimation unit 25B uses the correction information CI to generate the correction emotion information CVE that corrects the voice evaluation value x of each emotion included in the voice emotion information VE (step S52).
Even if the determination result in step S44 is negative and the determination result in step S45 is negative, the correction unit 253 of the emotion estimation unit 25B executes the process of step S52.
After the process of step S52 is completed, the estimation unit 258 of the emotion estimation unit 25B executes the process of step S61.

After executing the process of step S61, the server device 10 transmits the recognition character string RT and the estimated emotion information EI which is the processing result of step S61 to the user device 1b. The output unit 26 outputs information obtained by executing processing according to the emotion indicated by the estimated emotion information EI with respect to the recognition character string RT (step S62). After the processing of step S62 is completed, the emotion estimation system SYS ends the series of processing shown in FIGS. 15 and 16.

3.3. Effect of Third Embodiment According to the above description, when the user U2 who is a non-calibrated user utters the specific character string ST, the server device 10 sets a plurality of voice evaluation values X included in the corrected emotion information CVE2. For each of the plurality of emotions, the correction information CI for the user U2 is adjusted for the purpose of approaching the plurality of character evaluation values Y included in the character emotion information TE2. The specific character string ST is a recognition character string RT when the value indicating the degree of difference between the voice evaluation value X and the character evaluation value Y is equal to or less than a predetermined value in the user U1 who is the calibration user.
The reason for adjusting the correction information CI for the user U2 will be described only when the user U2 utters the specific character string ST. Even for a calibrated user, the corrected emotion information CVE and the character emotion information TE may not be close to each other. For example, when the calibrated user utters this character string in a meaning different from the original meaning of the character string, the corrected emotion information CVE and the character emotion information TE may not be close to each other. An example in which the user U utters a meaning different from the original meaning of the character string is when the user U utters an ironic content and when the user U utters a joke. When the user U utters an ironic content and a joke, the accuracy of the character emotion information TE is lowered. Therefore, if the emotion held by the user U is estimated based only on the character emotion information TE, the accuracy is lowered. Further, when the user U utters an office communication such as "I have arrived now", the accuracy of the character emotion information TE decreases. Therefore, if the emotion held by the user U is estimated based only on the character emotion information TE, the accuracy decreases. .. The reason why the accuracy of the character emotion information TE is lowered when the utterance content is the office communication is that the utterance content indicating the office communication includes the character string representing the emotion registered in the emotion classification information 33. This is because the character evaluation values Y1 to Y4 tend to be lower than those of general utterance contents. When the user U utters the ironic content, when the user U utters a joke, and when the user U utters an office communication, the emotions that the user U has are accurately measured based only on the text emotion information TE. This is an example when it cannot be estimated. When the character string is spoken in the original meaning, the corrected emotion information CVE and the character emotion information TE tend to have close values.
Therefore, the specific character string ST is a character string that is highly likely to have been uttered in the original meaning because the value indicating the degree of difference between the voice evaluation value X and the character evaluation value Y is equal to or less than a predetermined value. It can be said that. Then, when the user U2 utters the specific character string ST, there is a high possibility that the user U2 is uttering in the original meaning of the specific character string ST. Therefore, originally, the corrected emotion information CVE and the character The emotional information TE should be close to the value.
Here, in a non-calibrated user, in general, the accuracy of the corrected emotion information CVE is likely to be lower than the accuracy of the character emotion information TE. The reason is that the character emotion information TE has a small influence from the voice characteristics of the user U, while the voice emotion information VE has a large influence from the voice characteristics of the user U, and the correction information CI of the uncalibrated user. Is not adjusted correctly.
Therefore, in the third embodiment, when the user U2 utters the specific character string ST, there is a high possibility that the character emotion information TE indicates the correct emotion, so that the server device 10 sets the voice evaluation value X. The correction information CI for the user U2 is adjusted for the purpose of approaching the character evaluation value Y. As described above, for the non-calibrated user, the estimation accuracy of the emotion held by the user U can be improved without using the calibration mode. Since the non-calibrated user can improve the emotion estimation accuracy without setting the user device 1b to the calibration mode, the user device 1b can reduce the labor of the non-calibrated user.

4. Fourth Embodiment The emotion estimation system SYSc according to the fourth embodiment adjusts the first parameter P1 and the second parameter P2 for the uncalibrated user by using the emotion estimation result of the calibrated user. This is different from the emotion estimation system SYS according to the third embodiment.
Hereinafter, the emotion estimation system SYSc according to the fourth embodiment will be described. Regarding the elements whose actions or functions are equivalent to those of the third embodiment in the fourth embodiment illustrated below, the reference numerals will be used in the above description, and detailed description of each will be omitted as appropriate.

FIG. 17 is a diagram showing the overall configuration of the emotion estimation system SYSTEM. The emotion estimation system SYSc includes a user device 1b owned by the user U, a network NW, and a server device 10C.

FIG. 18 is a block diagram showing a configuration of the server device 10C. The server device 10C is realized by a computer system including a processing device 2C, a storage device 3C, and a communication device 6B. The storage device 3C is a recording medium that can be read by the processing device 2C, and stores a plurality of programs including the control program PRC executed by the processing device 2C, analysis dictionary information 31, emotion classification information 33, and learning model LM. To do.

The processing device 2C functions as the emotion estimation unit 25C by reading and executing the control program PR from the storage device 3C. The function of the emotion estimation system SYSc will be described with reference to FIG.

4.1. Function of the 4th Embodiment FIG. 19 is a figure which shows the outline of the function of the emotion estimation system SYSc. The emotion estimation unit 25C includes a noise removal unit 251C, a voice evaluation unit 252B, a correction unit 253, an adjustment unit 254C, a character evaluation unit 256, an estimation unit 258, and a specific unit 259.

In the fourth embodiment, the first parameter P1 and the second parameter P2 used in the noise removing unit 251C are prepared for each user U. In the following description, the information including the first parameter P1 and the second parameter P2 for the user U1 will be referred to as the parameter information TI1, and the information including the first parameter P1 and the second parameter P2 for the user U2 will be referred to as the parameter information TI2. ..

FIG. 20 is a diagram showing an outline of the adjustment function of the parameter information TI of the non-calibrated user. FIG. 20 shows a state in which the calibrated user U1 utters “Thank you” and the acquisition unit 21 of the user device 1b1 has acquired the sound information SI1.

The description returns to FIG. With respect to the user U1, the noise removing unit 251C removes noise from the sound indicated by the sound information SI1 based on the first parameter P1 and the second parameter P2 included in the parameter information TI1 to generate the voice information VI1. Regarding the subsequent processing, the emotion estimation unit 25C processes in the same manner as in the third embodiment to generate the corrected emotion information CVE1 and the character emotion information TE1, and sets the recognition character string RT1 "Thank you" as the specific character string ST. Identify as.

Regarding the user U2, as shown in FIG. 20, it is assumed that the user U2 utters "Thank you" which is a specific character string ST. The adjustment unit 254C has the purpose of bringing the plurality of voice evaluation values X included in the corrected emotion information CVE2 of the user U2 closer to the plurality of character evaluation values Y included in the character emotion information TE2 of the user U2 for each of the plurality of emotions. , Adjust the parameter information TI2 for user U2. Specifically, the adjusting unit 254C causes the noise removing unit 251C to generate the voice information VI2 based on the first parameter P1 and the second parameter P2 of the current parameter information TI2. Then, the adjustment unit 254C causes the voice evaluation unit 252B and the correction unit 253 to generate the corrected emotion information CVE2, and causes the character evaluation unit 256 to generate the character emotion information TE2. Then, the adjustment unit 254C compares the voice evaluation value X included in the corrected emotion information CVE2 with the character evaluation value Y included in the character emotion information TE2. For example, the adjusting unit 254C changes the first parameter P1 and the second parameter P2 of the parameter information TI2 by a small amount. The adjusting unit 254C regenerates the corrected emotion information CVE based on the first parameter P1 and the second parameter P2 changed by a minute amount, and determines the degree of difference between the regenerated corrected emotion information CVE and the character emotion information TE2. When the indicated value is smaller than the value indicating the degree of difference between the corrected emotion information CVE before re-creation and the character emotion information TE2, the plurality of voice evaluation values X of the user U2 and the plurality of character evaluation values Y of the user U2 are used. It is judged that the purpose of approaching is achieved.

In FIG. 20, the adjusting unit 254C sets the first parameter P1 and the second parameter P2 included in the parameter information TI2 for the purpose of bringing the plurality of voice evaluation values X of the user U2 closer to the plurality of character evaluation values Y of the user U2. Indicates to be adjusted.

4.2. Operation of the Fourth Embodiment Next, the operation of the emotion estimation system SYSc in the emotion estimation mode will be described with reference to FIG.

FIG. 21 is a flowchart showing the operation of the emotion estimation system SYSc in the emotion estimation mode. In the operation of the emotion estimation system SYSc in the emotion estimation mode shown in the third embodiment and the operation of the emotion estimation system SYSc in the emotion estimation mode of the fourth embodiment, the processes from step S31 to step S35 shown in FIG. Is common. Therefore, illustration and description of the processes from step S31 to step S35 will be omitted.

After the processing in step S35 is completed, the server device 10C determines whether or not the user U who possesses the user device 1b that is the source of the correction information CI and the sound information SI is a calibrated user (step S71). When the determination result in step S71 is affirmative, the noise removing unit 251C of the emotion estimation unit 25C removes noise from the sound indicated by the sound information SI according to the first parameter P1 and the second parameter P2 of the parameter information T1 to perform voice information. VI is generated (step S72). The voice evaluation unit 252B of the emotion estimation unit 25C extracts a sound feature amount from the noise-removed voice information VI (step S73). Next, the voice evaluation unit 252B of the emotion estimation unit 25C inputs the feature amount of the sound into the learning model LM, and acquires the voice emotion information VE including the voice evaluation value x of each emotion from the learning model LM (step S74). .. Then, the correction unit 253 of the emotion estimation unit 25C uses the correction information CI to generate the correction emotion information CVE that corrects the voice evaluation value x of each emotion included in the voice emotion information VE (step S75).

Then, in the specific unit 259 of the emotion estimation unit 25C, is the sum _{XY of the} squares of the differences between the voice evaluation value X included in the corrected emotion information CVE and the character evaluation value Y included in the character emotion information TE Sum _XY equal to or less than a predetermined value? Whether or not it is determined (step S76).
When the determination result in step S71 is affirmative and the determination result in step S76 is affirmative, the identification unit 259 of the emotion estimation unit 25C specifies the recognition character string RT as the specific character string ST (step S77). Then, the estimation unit 258 of the emotion estimation unit 25C estimates the emotion held by the user U based on the corrected emotion information CVE and the character emotion information TE (step S91). On the other hand, even when the determination result in step S71 is affirmative and the determination result in step S76 is negative, the estimation unit 258 of the emotion estimation unit 25C executes the process of step S91.

When the determination result in step S71 is negative, that is, when the user U who possesses the user device 1b that is the source of the correction information CI and the sound information SI is a non-calibrated user, the server device 10C has a specific character string ST. It is determined whether or not the recognition character string RT matches (step S81). When the determination result in step S71 is negative and the determination result in step S81 is affirmative, the adjustment unit 254C of the emotion estimation unit 25B includes the voice evaluation value X included in the corrected emotion information CVE in the character emotion information TE. The parameter information TI for the non-calibrated user is adjusted for the purpose of approaching the plurality of character evaluation values Y (step S82). Then, the noise removing unit 251C of the emotion estimation unit 25C removes noise from the sound indicated by the sound information SI according to the first parameter P1 and the second parameter P2 of the parameter information TI to generate the voice information VI (step S83). ..
Even if the determination result in step S71 is negative and the determination result in step S81 is negative, the noise removal unit 251C of the emotion estimation unit 25C executes the process of step S83.

After the processing of step S83 is completed, the voice evaluation unit 252B of the emotion estimation unit 25C extracts the feature amount of the sound from the voice information VI from which the noise has been removed (step S84). Next, the voice evaluation unit 252B of the emotion estimation unit 25C inputs the feature amount of the sound into the learning model LM, and acquires the voice emotion information VE including the voice evaluation value x of each emotion from the learning model LM (step S85). .. Then, the correction unit 253 of the emotion estimation unit 25C uses the correction information CI to generate the correction emotion information CVE in which the voice evaluation value x included in the voice emotion information VE is corrected (step S86). After the processing of step S86 is completed, the estimation unit 258 of the emotion estimation unit 25C executes the processing of step S91.

After the processing of step S91 is completed, the server device 10C transmits the recognition character string RT and the estimated emotion information EI which is the processing result of step S61 to the user device 1b. The output unit 26 outputs information obtained by executing processing according to the emotion indicated by the estimated emotion information EI with respect to the recognition character string RT (step S92). After the processing of step S92 is completed, the emotion estimation system SYSc ends a series of processing shown in FIG.

4.3. Effect of the fourth embodiment As in the third embodiment, when the user U2 utters the specific character string ST, the character emotion information TE is likely to indicate the correct emotion. Therefore, the server device 10C adjusts the parameter information TI for the user U2 for the purpose of bringing the voice evaluation value X closer to the character evaluation value Y. As described above, for the non-calibrated user, the estimation accuracy of the emotion held by the user U can be improved without using the calibration mode. The non-calibrated user can improve the emotion estimation accuracy without setting the user device 1b to the calibration mode, and the user device 1b can reduce the labor of the non-calibrated user.
The performance of the sound collecting device 8 differs between the user devices 1b. For example, if the manufacturers of the sound collecting devices 8 are different, the performances of the sound collecting devices 8 are generally different from each other. Further, since the performance of the sound collecting device 8 tends to deteriorate due to aged deterioration, even if the sound collecting device 8 is manufactured by the same manufacturer, the performance of the sound collecting device 8 is different from each other when the number of days from the time of manufacture is different. There is a tendency. As a result of the performance of the sound collecting device 8 being different between the user devices 1b, the amount of noise included in the sound information SI is also different. Therefore, by adjusting the parameter information TI, the emotion held by the user U can be estimated accurately.
For example, when noise processing to which the learned parameter information TI is applied is executed, the information necessary for emotion estimation may be missing from the voice information VI due to the difference in the performance of the sound collecting device 8. Therefore, by adjusting the parameter information TI according to the performance of the sound collecting device 8, the emotion held by the user U can be estimated accurately.

5. Fifth Embodiment The emotion estimation system SYSd according to the fifth embodiment is the first embodiment in that the processing of the emotion estimation unit 25 shown in the first embodiment is distributed between the server device 10D and the user device 1d. It is different from the user device 1. Hereinafter, the emotion estimation system SYSd according to the fifth embodiment will be described. Regarding the elements whose actions or functions are the same as those of the first embodiment in the fifth embodiment illustrated below, the reference numerals are used in the above description, and detailed description of each is appropriately omitted.

5.1. Outline of the 5th Embodiment FIG. 22 is a diagram showing the overall configuration of the emotion estimation system SYSTEM. The emotion estimation system SYSd includes a user device 1d owned by the user U, a network NW, and a server device 10D.

FIG. 23 is a block diagram showing the configuration of the user device 1d. The user device 1d is realized by a computer system including a processing device 2d, a storage device 3d, a display device 4, an operating device 5, a communication device 6, a sound emitting device 7, and a sound collecting device 8. The storage device 3d is a recording medium that can be read by the processing device 2d, and stores a plurality of programs including the control program PRd executed by the processing device 2d. The communication device 6 is an example of the “second communication device”.

The processing device 2d functions as an acquisition unit 21, a first emotion estimation unit 25d, and an output unit 26 by reading and executing the control program PRd from the storage device 3d.

FIG. 24 is a block diagram showing the configuration of the server device 10D. The server device 10D is realized by a computer system including a processing device 2D, a storage device 3D, and a communication device 6B. The storage device 3D is a recording medium that can be read by the processing device 2D, and stores a plurality of programs including a control program PRD executed by the processing device 2D, analysis dictionary information 31, emotion classification information 33, and a learning model LM. To do. The communication device 6B is an example of the “first communication device”.

The processing device 2D functions as a second emotion estimation unit 25D by reading and executing the control program PRD from the storage device 3D. The function of the emotion estimation system SYSd will be described with reference to FIG. 25.

FIG. 25 is a diagram showing an outline of the functions of the emotion estimation system SYSd. The first emotion estimation unit 25d includes a correction unit 253 and an estimation unit 258. The second emotion estimation unit 25D includes a noise removal unit 251, a voice evaluation unit 252, and a character evaluation unit 256.

The acquisition unit 21 acquires the sound information SI1 output by the sound collecting device 8 that collects the sound including the voice of the user U1. The communication device 6 transmits the sound information SI to the server device 10D. The second emotion estimation unit 25D generates voice emotion information VE, character emotion information TE, and recognition character string RT based on the sound information SI. The communication device 6B transmits the voice emotion information VE, the character emotion information TE, and the recognition character string RT to the user device 1d.

The correction unit 253 uses the correction information CI to generate the correction emotion information CVE. The estimation unit 258 estimates the emotion held by the user U based on the corrected emotion information CVE and the character emotion information TE. The output unit 26 outputs the data obtained by executing the processing according to the emotion indicated by the estimated emotion information EI with respect to the recognition character string RT.

5.2. Effect of Fifth Embodiment According to the above description, in the emotion estimation system SYSd, the load of the user device 1d can be reduced as compared with the user device 1 of the first embodiment.

6. The sixth embodiment The emotion estimation system SYSSe according to the sixth embodiment is the second embodiment in that the processing of the emotion estimation unit 25 shown in the second embodiment is distributed between the server device 10D and the user device 1e. It is different from the user device 1a. Hereinafter, the emotion estimation system SYSTEM according to the sixth embodiment will be described. Regarding the elements whose actions or functions are equivalent to those of the second embodiment or the fifth embodiment in the sixth embodiment illustrated below, the reference numerals are used in the above description, and detailed description of each is appropriately described. Omitted to.

6.1. Outline of the Sixth Embodiment FIG. 26 is a diagram showing the overall configuration of the emotion estimation system SYSTEM. The emotion estimation system SYSTEM includes a user device 1e owned by the user U, a network NW, and a server device 10D.

FIG. 27 is a block diagram showing the configuration of the user device 1e. The user device 1e is realized by a computer system including a processing device 2e, a storage device 3e, a display device 4, an operating device 5, a communication device 6, a sound emitting device 7, and a sound collecting device 8. The storage device 3e is a recording medium that can be read by the processing device 2e, and stores a plurality of programs including the control program PRE executed by the processing device 2e.

The processing device 2e functions as an acquisition unit 21a, a first emotion estimation unit 25e, and an output unit 26 by reading and executing the control program PRE from the storage device 3e. The function of the emotion estimation system SYSTEM will be described with reference to FIG. 28.

FIG. 28 is a diagram showing an outline of the functions of the emotion estimation system SYSTEM. The first emotion estimation unit 25e includes a correction unit 253, an adjustment unit 254, and an estimation unit 258.

The acquisition unit 21a acquires sound information SIa indicating a sound including a voice in which the user U expresses an explicit emotion. The server device 10D generates voice emotion information VEa based on the sound information SIa. Then, the communication device 6B transmits the voice emotion information VEa to the user device 1.

The adjustment unit 254 adjusts the correction information CI based on the explicit voice emotion information VEa for the purpose of increasing the possibility that the estimation unit 258 estimates that the emotion held by the user U is an explicit emotion.

6.2. Effect of the Sixth Embodiment According to the above description, in the emotion estimation system SYS, the load of the user device 1d can be reduced as compared with the user device 1 of the second embodiment.

7. Seventh Embodiment The emotion estimation system SYSf according to the seventh embodiment has the third embodiment in that the processing of the emotion estimation unit 25 shown in the third embodiment is distributed between the server device 10F and the user device 1f. It differs from the emotion estimation system SYS. Hereinafter, the emotion estimation system SYSf according to the seventh embodiment will be described. Regarding the elements whose actions or functions are equivalent to those of the third embodiment or the fifth embodiment in the seventh embodiment illustrated below, the reference numerals are used in the above description, and detailed description of each is appropriately described. Omitted to.

7.1. Outline of the 7th Embodiment FIG. 29 is a diagram showing the overall configuration of the emotion estimation system SYSf. The emotion estimation system SYSf includes a user device 1f owned by the user U, a network NW, and a server device 10F. User U1 is an example of a "first user". User U2 is an example of a "second user".

FIG. 30 is a block diagram showing a user device 1f. The user device 1f is realized by a computer system including a processing device 2f, a storage device 3f, a display device 4, an operating device 5, a communication device 6, a sound emitting device 7, and a sound collecting device 8. The storage device 3f is a recording medium that can be read by the processing device 2f, and stores a plurality of programs including the control program PRf executed by the processing device 2f.

The processing device 2f functions as an acquisition unit 21, a first emotion estimation unit 25f, and an output unit 26 by reading and executing the control program PRf from the storage device 3f.

FIG. 31 is a block diagram showing the configuration of the server device 10F. The server device 10F is realized by a computer system including a processing device 2F, a storage device 3F, and a communication device 6B. The storage device 3F is a recording medium that can be read by the processing device 2F, and stores a plurality of programs including a control program PRF executed by the processing device 2F, analysis dictionary information 31, emotion classification information 33, and a learning model LM. To do.

The processing device 2F functions as a second emotion estimation unit 25F by reading and executing the control program PRD from the storage device 3F. The function of the emotion estimation system SYSf will be described with reference to FIG. 32.

FIG. 32 is a diagram showing an outline of the function with the emotion estimation system SYSf. The first emotion estimation unit 25f includes a correction unit 253, an adjustment unit 254, and an estimation unit 258. The second emotion estimation unit 25F includes a noise removal unit 251, a voice evaluation unit 252, a character evaluation unit 256, and a specific unit 259. The user device 1f1 is an example of the "first terminal device". The user device 1f2 is an example of the "second terminal device". In order to prevent the drawings from becoming complicated, the functions realized by the processing device 2f of the user device 1f1 are not shown.

The acquisition unit 21 of the user device 1f1 acquires the sound information SI1 output by the sound collecting device 8 of the user device 1f1. The sound collecting device 8 of the user device 1f1 is an example of the “first sound collecting device”. The communication device 6 of the user device 1f1 transmits the sound information SI1 to the server device 10F. The noise removing unit 251 removes noise from the sound indicated by the sound information SI1 to generate the voice information VI1. Regarding the user U1, the subsequent processing is the same as the voice evaluation unit 252B, the correction unit 253, the voice recognition processing unit 2561, the morphological analysis processing unit 2563, the evaluation value calculation unit 2565, and the specific unit 259 shown in FIG. Is omitted. Further, although not shown, the second emotion estimation unit 25F transmits the recognition character string RT1, the voice emotion information VE1 of the user U1, and the character emotion information TE1 of the user U1 to the user device 1f1. Then, in order for the specific unit 259 to specify the specific character string ST, the communication device 6 of the user device 1f1 transmits the corrected emotion information CVE1 of the user U1 to the server device 10F.
Regarding the user U2, the acquisition unit 21 of the user device 1f2 acquires the sound information SI2 output by the sound collecting device 8 of the user device 1f2. The sound collecting device 8 of the user device 1f2 is an example of the “second sound collecting device”. The communication device 6 of the user device 1f2 transmits the sound information SI2 to the server device 10F. The communication device 6 of the user device 1f2 is an example of the “third communication device”. Regarding the user U2, the subsequent processing is the same as the voice evaluation unit 252B, the correction unit 253, the voice recognition processing unit 2561, the morphological analysis processing unit 2563, and the evaluation value calculation unit 2565 shown in FIG. 13, so the description thereof will be omitted. The communication device 6B transmits the specific character string ST, the voice emotion information VE2 of the user U2, and the character emotion information TE2 of the user U2 to the user device 1f2.

When the user U2 who is a non-calibration user utters a specific character string ST, the adjustment unit 254 sets a plurality of voice evaluation values X included in the corrected emotion information CVE2 of the user U2 for each of the plurality of emotions. , The correction information CI2 for the user U2 is adjusted for the purpose of approaching the plurality of character evaluation values Y included in the character emotion information TE2 of the user U2.

7.2. Effect of the 7th Embodiment According to the above description, in the emotion estimation system SYSf, the load of the server device 10F can be reduced as compared with the server device 10 of the 3rd embodiment.

8. Modifications The present invention is not limited to the embodiments exemplified above. A specific mode of modification is illustrated below. Two or more aspects arbitrarily selected from the following examples may be merged.

(1) As a first modification, the estimation unit 258 has explained that the emotion held by the user U is estimated based on the corrected emotion information CVE and the character emotion information TE, but the present invention is not limited to this. An example in which the estimation unit 258 estimates one or more emotions held by the user U based on the corrected emotion information CVE will be described with reference to FIG. 33.

FIG. 33 is a diagram showing an outline of the function of the user device 1g in the first modification. The processing device 2 of the user device 1g functions as an acquisition unit 21, an emotion estimation unit 25g, and an output unit 26g by reading and executing a control program from the storage device 3 of the user device 1g. The emotion estimation unit 25g includes a noise removal unit 251, a voice evaluation unit 252, a correction unit 253, and an estimation unit 258g. The estimation unit 258g estimates one or more emotions held by the user U based on the corrected emotion information CVE. For example, the estimation unit 258g compares the voice evaluation values X1 to X4 of the corrected emotion information CVE with the threshold value, and identifies the voice evaluation value X exceeding the threshold value. The estimation unit 258g estimates one or more emotions corresponding to the specified voice evaluation value X as one or more emotions held by the user U. The emotion estimation unit 25g outputs an estimated emotion information EI indicating one or more emotions held by the estimated user U.
The output unit 26g outputs the estimated emotion information EI. For example, the output unit 26g outputs a character string indicating the emotion indicated by the estimated emotion information EI to the display device 4.

(2) The processing of the emotion estimation unit 25C shown in the fourth embodiment may be distributed between the server device 10 and the user device 1. For example, the second emotion estimation unit 25 in the server device 10 has a noise removal unit 251C, a voice evaluation unit 252B, a character evaluation unit 256, and a specific unit 259. The first emotion estimation unit 25 in the user device 1 includes a correction unit 253, an adjustment unit 254, and an estimation unit 258.

(3) Although the emotion estimation unit 25 has explained that one or more emotions are estimated among joy, anger, sadness, and normality, one emotion may be estimated. For example, the estimation unit 258 adds the voice evaluation values X1 to X4 of the corrected emotion information CVE and the character evaluation values Y1 to Y4 of the character emotion information TE for each emotion, and calculates the addition value for each emotion. The estimation unit 258 may estimate the emotion having the largest value among the added values for each emotion as the emotion held by the user U.

(4) In the third embodiment, the emotion estimation unit 25B is realized by the server device 10, but it may be applied to one user device 1. For example, the user device 1 is possessed by a plurality of users U. When the user U1 possesses the user device 1 in a certain period, the user device 1 is set to the calibration mode, and the user U2 possesses the user device 1 in a period after a certain period, the third embodiment is performed. May be applied.

(5) In the fifth embodiment, the sixth embodiment, and the seventh embodiment, the communication device 6B uses the recognition character string RT, the voice emotion information VE2 of the user U2, and the character emotion information TE2 of the user U2. , Is transmitted to the user device 1f2, but the recognition character string RT may not be transmitted. For example, the user device 1f outputs a character string indicating the emotion indicated by the estimated emotion information EI to the display device 4.

(6) The user device 1 does not have to have the sound collecting device 8. When the sound collecting device 8 is not provided, the user device 1 may acquire the sound information SI via the communication device 6 or may acquire the sound information SI stored in the storage device 3.

(7) The user device 1 does not have to have the sound emitting device 7.

(8) The user device 1 may be a smart speaker. When the user device 1 is a smart speaker, the user device 1 does not have to have the display device 4 and the operation device 5.

(9) As shown in FIG. 4, the emotion classification information 33 is either joy, anger, sadness, or normal for each of a plurality of morphemes utilized by a certain word, such as "win" and "win". However, it is not limited to this. For example, the emotion classification information 33 may classify the character string registered in the prototype data of the analysis dictionary information 31 into any of joy, anger, sadness, and normal. For example, the emotion classification information 33 classifies the character strings “happy”, “pass”, and “win” registered in the prototype data of the analysis dictionary information 31 into joy. The evaluation value calculation unit 2565 decomposes the corrected recognition character string CRT for each morpheme, and converts the decomposed morpheme into a character string registered in the prototype data of the analysis dictionary information 31. Then, the evaluation value calculation unit 2565 corresponds to the character string included in the corrected recognition character string CRT when the character string obtained by conversion and the character string included in the emotion classification information 33 match. Increases the character evaluation value Y of emotions.

(10) The evaluation value calculation unit 2565 calculated the character evaluation value Y for each emotion with respect to the corrected recognition character string CRT, but calculated the character evaluation value Y for each emotion with respect to the recognition character string RT. May be good. However, the recognition character string RT includes a character string that is unnecessary for estimating emotions. Therefore, by calculating the character evaluation value Y for each emotion for the corrected recognition character string CRT, the emotion is estimated as compared with the case where the character evaluation value Y for each emotion is calculated for the recognition character string RT. The accuracy can be improved.

(11) The value indicating the degree of difference in the first aspect was the sum of the squares of the differences between the voice evaluation value X and the character evaluation value Y, but the difference between the voice evaluation value X and the character evaluation value Y. It may be a value obtained by an arbitrary evaluation function that defines the distance between evaluation values, such as the sum of absolute values of.

(12) Although the example in which the user U speaks Japanese is used, each of the above aspects can be applied to any language spoken by the user. For example, even when the user U speaks English, French, Chinese, or the like other than Japanese, each of the above aspects can be applied. For example, when the user U speaks English, the analysis dictionary information 31 is information related to English morphemes, and the emotion classification information 33 is data in which English words are classified into joy, anger, sadness, or normal. It should be.

(13) The block diagram used in the description of each of the above-described aspects shows a block of functional units. These functional blocks (components) are realized by any combination of hardware and / or software. Further, the means for realizing each functional block is not particularly limited. That is, each functional block may be realized by one physically and / or logically coupled device, or directly and / or indirectly by two or more physically and / or logically separated devices. (For example, wired and / or wireless) may be connected and realized by these plurality of devices.

(14) The order of the processing procedures, sequences, flowcharts, etc. in each of the above-described aspects may be changed as long as there is no contradiction. For example, the methods described herein present elements of various steps in an exemplary order, and are not limited to the particular order presented.

(15) In each of the above-described aspects, the input / output information and the like may be stored in a specific place (for example, a memory) or may be managed by a management table. Input / output information and the like can be overwritten, updated, or added. The output information and the like may be deleted. The input information or the like may be transmitted to another device.

(16) In each of the above-described aspects, the determination may be made by a value represented by 1 bit (0 or 1) or by a boolean value (Boolean: true or false). , May be done by numerical comparison (eg, comparison with a given value).

(17) In each of the above-described aspects, a portable information processing device such as a smartphone is illustrated as the user device 1, but the specific form of the user device 1 is arbitrary and is not limited to the above-mentioned examples of each form. .. For example, a portable or stationary personal computer may be used as the user device 1.

(18) In each of the above-described aspects, the storage device 3 is a recording medium that can be read by the processing device 2, and examples thereof include a ROM and a RAM. Discs, Blu-ray® disks, smart cards, flash memory devices (eg cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppies (registered trademarks) ) Disks, magnetic strips, databases, servers and other suitable storage media. The program may also be transmitted from the network. The program may also be transmitted from the communication network via a telecommunication line.

(19) Each of the above-described embodiments includes LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G, 5G, FRA (Future Radio Access), and W-CDMA (registered trademark). , GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark) ), Other systems that utilize suitable systems and / or next-generation systems that are extended based on them.

(20) In each of the above aspects, the information, signals, and the like described may be represented using any of a variety of different techniques. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description are voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these. It may be represented by a combination of.
In addition, the terms described in the present specification and / or the terms necessary for understanding the present specification may be replaced with terms having the same or similar meanings.

(21) FIG. 2, FIG. 5, FIG. 7, FIG. 8, FIG. 11, FIG. 12, FIG. 13, FIG. 14, FIG. 18, FIG. 19, FIG. 20, FIG. 23, FIG. 24, FIG. 25, FIG. 27, FIG. 28. , 30, 31, 32, and 33, each of which is illustrated by any combination of hardware and software. Further, each function may be realized by a single device, or may be realized by two or more devices configured as separate bodies from each other.

(22) The programs exemplified in each of the above-described embodiments are called instructions, instruction sets, codes, code segments regardless of whether they are called software, firmware, middleware, microcode or hardware description language, or by other names. , Program code, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, execution threads, procedures or functions, etc. should be broadly interpreted to mean.
Further, software, instructions, and the like may be transmitted and received via a transmission medium. For example, the software uses wired technology such as coaxial cable, fiber optic cable, twist pair and digital subscriber line (DSL) and / or wireless technology such as infrared, wireless and microwave to websites, servers, or other When transmitted from a remote source, these wired and / or wireless technologies are included within the definition of transmission medium.

(23) In each of the above-described embodiments, the information, parameters, etc. may be represented by absolute values, relative values from a predetermined value, or other corresponding information. May be good.

(24) The names used for the above-mentioned parameters are not limited in any respect. Further, mathematical formulas and the like using these parameters may differ from those expressly disclosed herein.

(25) In each of the above-described embodiments, the user device 1 includes a case where it is a mobile station. Mobile stations can be subscriber stations, mobile units, subscriber units, wireless units, remote units, mobile devices, wireless devices, wireless communication devices, remote devices, mobile subscriber stations, access terminals, mobile terminals, wireless, depending on the trader. It may also be referred to as a terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable term.

(26) In each of the above embodiments, the statement "based on" does not mean "based on" unless otherwise stated. In other words, the statement "based on" means both "based only" and "at least based on".

(27) Any reference to elements using designations such as "first", "second" as used herein does not generally limit the quantity or order of those elements. These designations can be used herein as a convenient way to distinguish between two or more elements. Thus, references to the first and second elements do not mean that only two elements can be adopted there, or that the first element must somehow precede the second element.

(28) As long as "inclusion," "comprising," and variations thereof in each of the embodiments described above are used herein or within the scope of the claims, these terms are used. As with the term "prepare", it is intended to be inclusive. Furthermore, the term "or" as used herein or in the claims is intended not to be an exclusive OR.

(29) In the whole of the present application, if articles are added by translation, for example, a, an and the in English, unless the context clearly indicates that these articles are not. Including multiple.

(30) It will be apparent to those skilled in the art that the present invention is not limited to the embodiments described herein. The present invention can be implemented as modifications and modifications without departing from the spirit and scope of the invention, which is determined based on the description of the claims. Therefore, the description herein is for illustrative purposes and has no limiting implications for the present invention. In addition, a plurality of aspects selected from the aspects illustrated in the present specification may be combined.

1, 1a, 1b, 1d, 1e, 1f, 1g ... User device, 10, 10C, 10D, 10F ... Server device, 21, 21a ... Acquisition unit, 26 ... Output unit, 31 ... Analysis dictionary information, 33 ... Emotion Classification information, 251, 251C ... Noise removal unit, 252, 252B ... Voice evaluation unit, 253 ... Correction unit, 254, 254C ... Adjustment unit, 256 ... Character evaluation unit, 258 ... Estimating unit, 259 ... Specific unit, CI ... Correction Information, CVE ... Corrected emotion information, LM ... Learning model, P1 ... 1st parameter, P2 ... 2nd parameter, TE ... Character emotion information, U ... User, VE ... Voice emotion information, VI ... Voice information, X ... Voice evaluation Value, Y ... Character evaluation value.

Claims (9)

Voice information indicating the user's voice for a learning model that has been learned for multiple humans, regarding the relationship between multiple feature quantities according to human voice and the intensity of each of the multiple emotions held by the person who emitted the voice. A voice evaluation unit that inputs a plurality of feature quantities based on the above and acquires voice emotion information including a voice evaluation value indicating the intensity of each of the plurality of emotions held by the user from the learning model.
A correction unit that generates corrected emotion information that corrects the voice emotion information using correction information based on the characteristics of the user's voice, and a correction unit.
An estimation unit that estimates one or more emotions held by the user from the plurality of emotions based on the corrected emotion information.
Emotion estimator equipped with. An acquisition unit that acquires sound information output by a sound collecting device that collects sounds including the user's voice, and an acquisition unit.
A noise removing unit that removes noise from the sound indicated by the sound information to generate the voice information,
The emotion estimation device according to claim 1. The voice evaluation unit inputs a plurality of feature quantities based on voice information indicating a voice in which the user explicitly expresses one of the plurality of emotions to the learning model, and the user Explicit voice emotional information is obtained from the learning model and
An adjustment unit that adjusts the correction information based on the explicit voice emotion information for the purpose of increasing the possibility that the estimation unit estimates that the emotion held by the user is the one emotion.
The emotion estimation device according to claim 1 or 2. A voice recognition process for recognizing the utterance content of the voice uttered by the human is executed for the sound information, and based on the recognition character string indicating the recognition result of the voice recognition process, the plurality of emotions held by the user It is equipped with a character evaluation unit that generates character emotion information including a character evaluation value indicating the strength for each.
The estimation unit estimates one or more emotions held by the user based on the corrected emotion information and the character emotion information.
The emotion estimation device according to claim 3. When the value indicating the degree of difference between the plurality of voice evaluation values included in the corrected emotion information of the user and the plurality of character evaluation values included in the character emotion information is equal to or less than a predetermined value, the recognition character string is specified. It has a specific part to be specified as a character string,
When another user who has not spoken a voice that expresses an explicit emotion speaks the specific character string,
The voice evaluation unit learns the voice emotion information of the other user by inputting a plurality of feature quantities corresponding to the voice of the other user uttering the specific character string into the learning model. Get from the model
The character evaluation unit
Generates the character emotion information of the other user based on the voice that the other user utters the specific character string.
The purpose of the adjusting unit is to bring a plurality of voice evaluation values included in the corrected emotion information of the other user closer to a plurality of character evaluation values included in the character emotion information of the other user for each of the plurality of emotions. To adjust the correction information for the other user,
The emotion estimation device according to claim 4. A noise removing unit that removes noise from the sound indicated by the sound information based on a predetermined threshold value to generate the voice information, and a noise removing unit.
When the value indicating the degree of difference between the plurality of voice evaluation values included in the corrected emotion information of the user and the plurality of character evaluation values included in the character emotion information is equal to or less than a predetermined value, the recognition character string is specified. It has a specific part to be specified as a character string,
When another user who has not spoken a voice that expresses an explicit emotion speaks the specific character string,
The voice evaluation unit learns the voice emotion information of the other user by inputting a plurality of feature quantities corresponding to the voice of the other user uttering the specific character string into the learning model. Get from the model
The character evaluation unit generates character emotion information of the other user based on the voice that the other user utters the specific character string.
The purpose of the adjusting unit is to bring a plurality of voice evaluation values included in the corrected emotion information of the other user closer to a plurality of character evaluation values included in the character emotion information of the other user for each of the plurality of emotions. To adjust the predetermined threshold value for the other user.
The emotion estimation device according to claim 4. An emotion estimation system including a server device and a terminal device capable of communicating with the server device.
The server device is
The first communication device that receives sound information indicating sound including the user's voice, and
A noise removing unit that removes noise from the sound indicated by the sound information and generates voice information indicating the user's voice.
A plurality of learning models based on the voice information regarding the relationship between a plurality of feature quantities corresponding to the human voice and the intensity of each of the plurality of emotions held by the person who emitted the voice for each of the plurality of humans. A voice evaluation unit that inputs a feature amount of the above and acquires voice emotion information including a voice evaluation value indicating the intensity of each of the plurality of emotions held by the user from the learning model.
A voice recognition process for recognizing the utterance content of a voice uttered by a human is executed on the sound information, and based on a recognition character string indicating a recognition result of the voice recognition process, the plurality of emotions held by the user It is equipped with a character evaluation unit that generates character emotion information including a character evaluation value indicating the strength for each.
The first communication device is
The character emotion information and the voice emotion information are transmitted to the terminal device,
The terminal device is
A sound collecting device that collects sounds including the user's voice, and
A second communication device that transmits the sound information output by the sound collecting device to the server device and receives the character emotion information and the voice emotion information from the server device.
A correction unit that generates corrected emotion information that corrects the voice emotion information using correction information based on the characteristics of the user's voice, and a correction unit.
It includes an estimation unit that estimates one or more emotions held by the user based on the corrected emotion information and the character emotion information.
Emotion estimation system. The voice evaluation unit inputs a plurality of feature quantities based on voice information indicating a voice in which the user explicitly expresses one of the plurality of emotions to the learning model, and the user Explicit voice emotional information is obtained from the learning model and
The terminal device is
An adjustment unit that adjusts the correction information based on the explicit voice emotion information for the purpose of increasing the possibility that the estimation unit estimates that the emotion held by the user is the one emotion.
7. The emotion estimation system according to claim 7. The terminal device is a first terminal device and is
The user is a first user and
The sound collecting device is the first sound collecting device, and is
The server device can communicate with a second terminal device different from the first terminal device.
The second communication device is
The corrected emotion information of the first user is transmitted to the server device,
The server device is
When the value indicating the degree of difference between the plurality of voice evaluation values included in the corrected emotion information of the first user and the plurality of character evaluation values included in the character emotion information is equal to or less than a predetermined value, the recognition character string It has a specific part that identifies as a specific character string,
When the second user possessing the second terminal device does not utter a voice that expresses an explicit emotion and utters the specific character string.
The voice evaluation unit learns the voice emotion information of the second user by inputting a plurality of feature quantities corresponding to the voice spoken by the second user into the learning model. Get from the model
The character evaluation unit generates character emotion information of the second user based on the voice that the second user utters the specific character string.
The first communication device is
The specific character string, the voice emotion information of the second user, and the character emotion information of the second user are transmitted to the second terminal device.
The second terminal device is
A second sound collecting device that collects sounds including the voice of the second user, and
A third communication device that transmits sound information output by the second sound collecting device to the server device and receives the character emotion information of the second user and the voice emotion information of the second user from the server device.
A correction unit that generates corrected emotional information of the second user, which corrects the voice emotional information of the second user by using correction information based on the characteristics of the voice of the user.
The second, for the purpose of bringing a plurality of voice evaluation values included in the corrected emotion information of the second user closer to a plurality of character evaluation values included in the character emotion information of the second user for each of the plurality of emotions. The adjustment unit for adjusting the correction information for the user is provided.
The emotion estimation system according to claim 8.

Images