JP2021012285A - Emotion estimation device, and emotion estimation system - Google Patents

Abstract

To precisely estimate emotion born by a user, even in a state that user voice has little expression of the emotion.SOLUTION: An emotion estimation device comprises: a generation unit which generates a feature amount for user voice, based on voice information on a user; a first evaluation unit which generates a first voice evaluation value indicating the intensity of first emotion born by the user, and a second voice evaluation value indicating the intensity of second emotion born by the user, based on the feature amount; a recognition unit which generates a recognition character string indicating an utterance content of the user, based on the voice information; a second evaluation unit which generates a first character evaluation value indicating the intensity of the first emotion born by the user, and a second character evaluation value indicating the intensity of the second emotion born by the user, based on the recognition character string; a correction unit which corrects the first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the second character evaluation value, based on situation information indicating the situation of the user; and an estimation unit which estimates one or more pieces of emotion born by the user, based on the correction result of the correction unit.SELECTED DRAWING: Figure 6

Description

The present invention relates to an emotion estimation device and an emotion estimation system.

In recent years, there has been known a technique for estimating joy, anger, sadness, and emotions held by a normal user in consideration of the user's situation. For example, in Patent Document 1, a feature amount corrected according to the user's situation is input to a learning model in which the relationship between the feature amount of the user's voice information and emotions has been learned, and the user holds the feature amount from the learning model. A technique for outputting emotions is disclosed.

Japanese Unexamined Patent Publication No. 2018-072876

However, in a situation where emotions are poorly expressed in the user's voice, it is difficult to determine whether it is difficult to express emotions in the voice or whether the user does not express emotions in the first place. , It is difficult to accurately judge the emotions that the user has.

The emotion estimation device according to a preferred embodiment of the present invention includes a generation unit that generates a feature amount for the user's voice based on the user's voice information, and the user has a first emotion based on the feature amount. Based on the voice information, the first evaluation unit that generates the first voice evaluation value indicating the intensity and the second voice evaluation value indicating the intensity that the user has the second emotion, and the speech content of the user are shown. A recognition unit that generates a recognition character string, a first character evaluation value indicating the strength with which the user has the first emotion based on the recognition character string, and a first character evaluation value indicating the strength with which the user has the second emotion. The first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the above, based on the second evaluation unit that generates the two-character evaluation value and the situation information indicating the situation of the user. It includes a correction unit that corrects the second character evaluation value, and an estimation unit that estimates one or more emotions held by the user based on the correction result of the correction unit.

The emotion estimation system according to a preferred embodiment of the present invention is an emotion estimation system including the above-mentioned emotion estimation device and a terminal device capable of communicating with the emotion estimation device, and the terminal device uses the voice of the user. The sound collecting device for collecting sound, the situation information generation unit for generating the situation information indicating the user's situation, the voice information indicating the user's voice, and the situation information are transmitted to the emotion estimation device to estimate the emotion. The recognition character is a communication device that receives the recognition character string and emotion information indicating one or more emotions held by the user estimated by the estimation unit, and processing according to the emotion indicated by the emotion information. It includes an output unit that outputs information obtained by executing the column.

According to the present invention, it is possible to accurately estimate the emotions held by the user even in a situation where the emotions of the user's voice are poorly expressed.

The figure which shows the outline of the user apparatus 1. The block diagram which shows the structure of the user apparatus 1 which concerns on 1st Embodiment. The figure which shows an example of the storage contents of the dictionary information 31 for analysis. The figure which shows an example of the memory content of the emotion classification information 33. The figure which shows an example of the storage contents of the schedule information 35. The figure which shows the outline of the function of the user apparatus 1. The figure which shows an example of the memory content of the situation relation information 37. The flowchart which shows the operation of the user apparatus 1. The block diagram which shows the user apparatus 1a which concerns on 2nd Embodiment. The figure which shows the outline of the function of the user apparatus 1a in 2nd Embodiment. The figure which shows an example of the storage contents of the character string relation information 38. The flowchart which shows the operation of the user apparatus 1a. The figure which shows the whole structure of the emotion estimation system SYS. The block diagram which shows the structure of the user apparatus 1b. The block diagram which shows the structure of the server apparatus 10.

1. 1. 1st Embodiment FIG. 1 is a diagram showing an outline of a user device 1. The user device 1 is assumed to be a smartphone. The user device 1 is an example of an “emotion estimation device”. However, as the user device 1, any information processing device can be adopted, and for example, it may be a terminal-type information device such as a personal computer, or a portable type such as a notebook computer, a wearable terminal, or a tablet terminal. It may be an information terminal of.

The user device 1 has a function of transmitting a recognition character string obtained by executing a voice recognition process on voice information indicating a sound including a sound of a user U who owns the user device 1 to a device used by another person. Or, it has a function of emitting a sound indicating a recognition character string in order to let another person located near the user U hear it. Further, the user device 1 estimates the emotion held by the user U based on the voice of the user U, adds a figure corresponding to the estimated emotion to the recognition character string, or adds the estimated emotion to the recognition character string. Emotional expressions necessary for communication can be added by emitting a sound indicating a recognition character string with intonation according to.
In the example of FIG. 1, a user U is say "Hello", the user device 1, a graphic PI according to the estimated emotions, appended to the image representing the recognized character string.

FIG. 2 is a block diagram showing a configuration of the user device 1 according to the first embodiment. The user device 1 is realized by a computer system including a processing device 2, a storage device 3, an input device 4, an output device 5, a communication device 6, an inertial sensor 7, and a GPS (Global Positioning System) device 8. Each element of the user device 1 is connected to each other by a single unit or a plurality of buses 9 for communicating information. The term "device" in the present specification may be read as another term such as a circuit, a device, or a unit. Further, each element of the user device 1 may be composed of a single device or a plurality of devices, and some elements of the user device 1 may be omitted.

The processing device 2 is a processor that controls the entire user device 1, and is composed of, for example, a single chip or a plurality of chips. The processing device 2 is composed of, for example, a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, registers, and the like. Part or all of the functions of the processing device 2 are realized by hardware such as DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), FPGA (Field Programmable Gate Array), etc. You may. The processing device 2 executes various processes in parallel or sequentially.

The storage device 3 is a recording medium that can be read by the processing device 2, and is a plurality of programs including a control program PR executed by the processing device 2, analysis dictionary information 31, emotion classification information 33, schedule information 35, and situation-related information. 37 and the learning model LM are stored. The storage device 3 is composed of, for example, one or more types of storage circuits such as ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), and RAM (Random Access Memory).

FIG. 3 is a diagram showing an example of the stored contents of the analysis dictionary information 31. The analysis dictionary information 31 is information in which part of speech, part of speech subclassification, and original form information are associated with each other for each morpheme. A morpheme is a character string that is the smallest unit of a meaningful expression element. Part of speech is a type of word classified according to its grammatical nature, such as nouns, verbs, and adjectives. Part of speech subclassification is an item in which part of speech is further subdivided. The original form information is a character string indicating the original form of the word when the word is utilized by the corresponding morpheme, and is the same character string as the corresponding morpheme when the word is not utilized by the relevant morpheme.

FIG. 4 is a diagram showing an example of the stored contents of the emotion classification information 33. The emotion classification information 33 is information in which the character string is classified into any of joy, anger, sadness, and normal. In the example of FIG. 4, the character string group 331 classified as joy includes "happy", "pass", "win", "win", and the like. Similarly, the character string group 332 classified as anger includes "irritated", "mucked up" and the like. Similarly, the character string group 333 classified as sad includes "sad", "losing", and the like. Similarly, the character string group 334 normally classified includes "safety" and the like.

FIG. 5 is a diagram showing an example of the stored contents of the schedule information 35. Schedule information 35 indicates the schedule of user U. The schedule information 35 shown in FIG. 5 has records 35_1 to 35_3. Record 35_1 indicates that User U's schedule from 10:00 to 11:00 on April 10, 2019 is a meeting with the client. Record 35_2 indicates that User U's schedule from 15:00 to 16:00 on April 12, 2019 is to attend an internal meeting. Record 35_3 indicates that User U's schedule from 18:00 to 20:00 on April 15, 2019 is to attend the alumni association.

The explanation is returned to FIG. The situation-related information 37 is used when estimating emotions. In the learning model LM, the relationship between the feature amount corresponding to the human voice and the intensity for each of the plurality of emotions has been learned. Further, it is preferable that the learning model LM has learned the relationship between the feature amount corresponding to the human voice and the intensity for each of the plurality of emotions for a plurality of humans.

The input device 4 is a device for inputting information used by the user device 1 into the processing device 2. The input device 4 includes a sound collecting device 41 and a touch panel 43. The output device 5 is a device for outputting information. The output device 5 includes a display device 51 and a sound emitting device 53.

The sound collecting device 41 is composed of, for example, a microphone and an AD converter, and collects sounds including the voice of the user U under the control of the processing device 2. The microphone converts the collected voice into an electric signal. The AD converter AD-converts the electric signal converted by the microphone and converts it into the voice information VI shown in FIG. The sound indicated by the voice information VI may include noise emitted from the surroundings of the speaker in addition to the voice of the speaker. The touch panel 43 detects contact with the display surface of the display device 51. The touch panel 43 may include a plurality of controls that can be operated by the user U.

The display device 51 displays various images under the control of the processing device 2. For example, various display panels such as a liquid crystal display panel and an organic EL (electro-luminescence) display panel are preferably used as the display device 51. The sound emitting device 53 is composed of, for example, a speaker, and emits sound under the control of the processing device 2.

The communication device 6 is hardware (transmission / reception device) for communicating with another device via a network. The communication device 6 is also called, for example, a network device, a network controller, a network card, a communication module, or the like.

The inertial sensor 7 measures the inertial force applied to the user device 1 and outputs the inertial information IFI shown in FIG. 6 obtained from the measurement result. For example, the inertial sensor 7 is one or both of an acceleration sensor and an angular velocity sensor.

The GPS device 8 receives radio waves from a plurality of satellites and generates the position information PoI shown in FIG. The position information PoI may be in any format as long as the position can be specified. The position information PoI indicates, for example, the latitude and longitude of the user device 1. Further, in the present embodiment, it is illustrated that the position information PoI is obtained from the GPS device 8, but the user device 1 may acquire the position information PoI by any method. For example, the user device 1 acquires a cell ID (IDentifier) assigned to a base station that is a communication destination of the user device 1 as position information PoI. The cell ID is identification information that uniquely identifies the base station. Further, when the user device 1 communicates with the access point of the wireless LAN (Local Area Network), the identification address (MAC (Media Access Control) address) on the network assigned to the access point and the actual address (location). The location information PoI may be acquired by referring to the database associated with.

1.1. The function processing device 2 of the first embodiment functions as an acquisition unit 21, a situation information generation unit 23, an emotion information generation unit 25, and an output unit 27 by reading and executing the control program PR from the storage device 3.
The function realized by the processing apparatus 2 will be described with reference to FIG.

FIG. 6 is a diagram showing an outline of the functions of the user device 1. The acquisition unit 21 acquires the voice information VI output by the sound collecting device 41.

The status information generation unit 23 generates status information SiI indicating the status of the user U based on the position information PoI obtained from the GPS device 8, the inertial information IFI obtained from the inertial sensor 7, and the schedule information 35. More specifically, the situation information generation unit 23 identifies the location of the user U based on the location information PoI. The location includes, for example, the home of the user U, the company of the user U, and the commuting route from the home of the user U to the company. Further, the situation information generation unit 23 determines whether the user U is stopped, the user U is walking, or the user U is riding in the vehicle and moving based on the inertial information IFI. Then, when the specified place is home, the situation information generation unit 23 generates identification information indicating the situation where the user U is at home as the situation information SiI. Further, when the specified place is in the commuting route and the user U is moving in the vehicle, the situation information generation unit 23 indicates the situation where the user U is using the train as the situation information SiI. Generate identification information. The situation of using a train is an example of "a situation of using public transportation". Further, when the schedule information 35 indicates a meeting or a meeting regarding the schedule of the user U at the current time, the status information generation unit 23 indicates a situation in which the user U is in a meeting as the status information SiI. Generate identification information.

The emotion information generation unit 25 estimates one or more emotions held by the user U from a plurality of emotions held by the user U. In the first embodiment, the plurality of emotions held by the user U will be described as four emotions: joy, anger, sadness, and normality. Below, joy, anger, sadness, and normality are examples of multiple emotions.

The emotion information generation unit 25 includes a feature amount generation unit 251, a first evaluation unit 252, a recognition unit 254, a second evaluation unit 255, a correction unit 257, and an estimation unit 258. The feature amount generation unit 251 is an example of the “generation unit”.

The feature amount generation unit 251 generates a feature amount for the voice of the user U from the voice information VI. Features include, for example, MFCC (Mel-Frequency Cepstrum Coefficients) 12 dimensions, loudness, fundamental frequency (F0), loudness, zero crossover ratio, HNR (Harmonics-to-Noise-Ratio), and their first derivative, MFCC. And a total of 47 second-order differentials of loudness, part or all. Loudness is the loudness of a sound, which indicates the intensity of the sound felt by human hearing. The voice probability indicates the probability that the sound indicated by the voice information VI includes voice. The zero crossing rate is the number of times the sound pressure becomes zero. Further, the feature amount generation unit 251 may execute the correction process on the voice information VI and extract the feature amount from the corrected voice information obtained by executing the correction process. The correction process is, for example, one or both of a process of removing silent portion data from the voice information VI and a process of removing noise contained in the sound indicated by the voice information VI.

The first evaluation unit 252 generates voice evaluation values EV1 to EV4 corresponding to each of the plurality of emotions based on the feature amount. More specifically, the first evaluation unit 252 sets the voice evaluation value EV1 corresponding to joy, the voice evaluation value EV2 corresponding to anger, the voice evaluation value EV3 corresponding to sadness, and the voice evaluation value EV4 corresponding to normal times. Generate. The voice evaluation value EV indicates the intensity with which the user U has emotions.
In the following description, when distinguishing the same kind of elements, reference codes are used such as the voice evaluation value EV1 corresponding to joy and the voice evaluation value EV2 corresponding to anger. On the other hand, when the same type of elements are not distinguished, only the common number among the reference codes is used as in the voice evaluation value EV.
In addition, joy, anger, sadness, and normality are each examples of "first emotion". The "second emotion" is any one of joy, anger, sadness, and normal emotions that are different from the "first emotion". The voice evaluation value EV1 corresponding to joy, the voice evaluation value EV2 corresponding to anger, the voice evaluation value EV3 corresponding to sadness, and the voice evaluation value EV4 corresponding to normal times are examples of the "first voice evaluation value". A voice evaluation value corresponding to a second emotion different from the first emotion is an example of a “second voice evaluation value”.
For example, the first evaluation unit 252 generates the voice evaluation values EV1 to EV4 by one of the two modes shown below.

In the first aspect, the first evaluation unit 252 generates the voice evaluation values EV1 to EV4 by comparing the feature amount with the predetermined value. For example, when joy or anger is expressed in voice, the fundamental frequency tends to be higher and the loudness tends to be higher than when joy or anger is not expressed in voice. For example, in the first evaluation unit 252, when the fundamental frequency is larger than the predetermined value and the loudness is larger than the predetermined value, the fundamental frequency is smaller than the predetermined value and the loudness is smaller than the predetermined value. The voice evaluation value EV1 and the voice evaluation value EV2 are set to large values.

In the second aspect, the first evaluation unit 252 inputs the feature amount generated by the feature amount generation unit 251 into the learning model LM in order to generate the voice evaluation values EV1 to EV4, and the voice evaluation values EV1 to EV4. Is obtained from the learning model LM.

The recognition unit 254 generates a recognition character string RT indicating the utterance content of the user U based on the voice information VI. More specifically, the recognition unit 254 executes a voice recognition process including a method of recognizing a character string from a voice using, for example, an acoustic model and a language model prepared in advance, and outputs a recognition character string RT. ..

The second evaluation unit 255 generates a character evaluation value ET corresponding to each of the plurality of emotions based on the recognition character string RT. More specifically, the second evaluation unit 255 sets the character evaluation value ET1 corresponding to joy, the character evaluation value ET2 corresponding to anger, the character evaluation value ET3 corresponding to sadness, and the character evaluation value ET4 corresponding to normal times. Generate. The character evaluation value ET indicates the intensity with which the user U has emotions.
The character evaluation value ET1 corresponding to joy, the character evaluation value ET2 corresponding to anger, the character evaluation value ET3 corresponding to sadness, and the character evaluation value ET4 corresponding to normal times are examples of the "first character evaluation value". .. The character evaluation value corresponding to the second emotion different from the first emotion is an example of the “second character evaluation value”.

More specifically, the second evaluation unit 255 includes an analysis unit 2552 and a calculation unit 2554. The analysis unit 2552 refers to the analysis dictionary information 31 to execute morphological analysis processing on the recognition character string RT, and outputs the corrected recognition character string CRT. The morphological analysis process is a process of decomposing the recognition character string RT for each morpheme. In the morphological analysis process, the part of speech and the part of speech subclassification of the analysis dictionary information 31 are used. The corrected recognition character string CRT is a character string excluding a character string such as a filler, which is unnecessary for estimating the emotion held by the user U. Filler is a word that is inserted between utterances, such as "um", "that", and "well".

The calculation unit 2554 calculates the character evaluation value ET of each emotion by comparing the character string included in the emotion classification information 33 with the corrected recognition character string CRT. Regarding the calculation of the character evaluation value ET of each emotion, the calculation unit 2554 calculates the character string included in the corrected recognition character string CRT when the corrected recognition character string CRT includes the character string included in the emotion classification information 33. Increases the character evaluation value ET of the emotion corresponding to.
For example, if the corrected recognition character string CRT is "winning the game today", the calculation unit 2554 outputs the character evaluation value ET of each emotion as follows.

Character evaluation value corresponding to joy ET1 1
Character evaluation value ET20 corresponding to anger
Character evaluation value ET30 corresponding to sadness
Character evaluation value ET40 corresponding to normal times

In the above example, since the corrected recognition character string CRT includes the "win" included in the emotion classification information 33, the calculation unit 2554 increases the joy character evaluation value ET1 corresponding to the "win" by 1. Let me. The amount of increase in the character evaluation value ET to be increased is not limited to 1, and may differ for each character string included in the emotion classification information 33. For example, the amount of increase in the character evaluation value ET for a character string that more strongly indicates joy may be set to 2. Further, when the corrected recognition character string CRT includes the character string included in the emotion classification information 33 and the character string that emphasizes the content, the calculation unit 2554 may increase the increase amount of the character evaluation value ET. Good. For example, if the corrected recognition character string CRT is "very happy to win the game today", the corrected recognition character string CRT contains "happy" included in the emotion classification information 33 and is "very". Since the character string emphasizing the content is included, the calculation unit 2554 increases, for example, the character evaluation value ET1 of joy by 2. Which character string of the corrected recognition character string CRT is a character string that emphasizes the content can be determined by the morpheme obtained by the morphological analysis process. In the following example, for the sake of simplicity, it is assumed that the amount of increase in the character evaluation value ET to be increased is 1.
Further, when the corrected recognition character string CRT includes a character string included in the emotion classification information 33 and a character string denying the content, the calculation unit 2554 determines the character string included in the corrected recognition character string CRT. A process different from the process of increasing the character evaluation value ET corresponding to the above may be executed. For example, if the corrected recognition character string CRT is "could not win the game today", the corrected recognition character string CRT includes "win" included in the emotion classification information 33, but is said to be "not". Since the character string denying the content is included, the calculation unit 2554 increases, for example, the character evaluation value ET3 of sadness by 1. Which character string of the corrected recognition character string CRT is a character string whose content is negated can be determined by the morpheme obtained by the morphological analysis process. In this way, it is possible to estimate whether the corrected recognition character string CRT has positive content or negative content by the morphological analysis process. In the following example, if the corrected recognition character string CRT includes the character string included in the emotion classification information 33, the character corresponding to the character string included in the corrected recognition character string CRT is included in the following example. The explanation will be given as increasing the evaluation value ET.

The correction unit 257 corrects the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 based on the situation information SiI. More specifically, the correction unit 257 corrects the voice evaluation value EV1, the voice evaluation value EV2, the voice evaluation value EV3, and the voice evaluation value EV4, and corrects the voice evaluation value CEV1 and the corrected voice evaluation value CEV2. The corrected voice evaluation value CEV3 and the corrected voice evaluation value CEV4 are output. Similarly, the correction unit 257 corrects the character evaluation value ET1, the character evaluation value ET2, the character evaluation value ET3, and the character evaluation value ET4, and corrects the corrected character evaluation value CET1, the corrected character evaluation value CET2, and the corrected character evaluation value CET2. The character evaluation value CET3 and the corrected character evaluation value CET4 are output.

More specifically, the correction unit 257 outputs the corrected voice evaluation values CEV1 to CEV4 and the corrected character evaluation values CET1 to CET4 by executing the following calculation.

CEV1 = kv1 x EV1
CEV2 = kv2 x EV2
CEV3 = kv3 x EV3
CEV4 = kv4 x EV4
CET1 = kt1 x ET1
CET2 = kt2 x ET2
CET3 = kt3 x ET3
CET4 = kt4 x ET4

However, the weighting coefficients kv1 to kv4 and the weighting coefficients kt1 to kt4 are real numbers of 0 or more and 1 or less. Further, the weighting coefficients kv1 to kv4 and the weighting coefficients kt1 to kt4 have the following conditions.

kv1 + kt1 = kv2 + kt2 = kv3 + kt3 = kv4 + kt4

For example, the correction unit 257 sets the weighting coefficients kv1 to kv4 and the weighting coefficients kt1 to kt4 with reference to the situation-related information 37.

FIG. 7 is a diagram showing an example of the stored contents of the situation-related information 37. The situation-related information 37 shows the relationship between the identification information indicating a situation that a human can take, each of the plurality of voice evaluation values EV set according to the situation, and each of the plurality of character evaluation values ET.

The situations that humans can take include a situation in which the user is in a private space where entry is prohibited without the permission of the user, and a situation in which the user is in a non-private space where access is possible without the permission of the user. The private space is, for example, in the home. Non-private spaces are, for example, in public transportation and in the workplace.

In the situation-related information 37 shown in FIG. 7, "home", "train", and "meeting" are registered as identification information indicating a situation that a human can take. The identification information "home" indicates a situation in which the user U is at home. The identification information "train" indicates a situation in which the user U is using the train. The identification information "meeting" indicates a situation in which the user U is in a meeting at work.

The situation-related information 37 indicates that the weighting coefficient kv1 is larger than the weighting coefficient kt1 for the weighting coefficient related to the identification information “home”. Similarly, the situation-related information 37 indicates that the weighting coefficient kv2 is larger than the weighting coefficient kt2, the weighting coefficient kv3 is larger than the weighting coefficient kt3, and the weighting coefficient kv4 is larger than the weighting coefficient kt4. Specifically, the weighting coefficient related to the identification information "home" shown in FIG. 7 is as follows.
Weighting coefficient kv1 = 0.7
Weighting coefficient kt1 = 0.3
Weighting coefficient kv2 = 0.8
Weighting coefficient kt2 = 0.2
Weighting coefficient kv3 = 0.7
Weighting coefficient kt3 = 0.3
Weighting coefficient kv4 = 0.8
Weighting coefficient kt4 = 0.2

Further, the situation-related information 37 indicates that the weighting coefficient kt1 is larger than the weighting coefficient kv1 for the weighting coefficient related to the identification information “train”. Similarly, the situation-related information 37 indicates that the weighting coefficient kt2 is larger than the weighting coefficient kv2, the weighting coefficient kt3 is larger than the weighting coefficient kv3, and the weighting coefficient kt4 is larger than the weighting coefficient kv4. Specifically, the weighting coefficients related to the identification information "train" shown in FIG. 7 are as follows.
Weighting coefficient kv1 = Weighting coefficient kv2 = Weighting coefficient kv3 = Weighting coefficient kv4 = 0.1
Weighting coefficient kt1 = Weighting coefficient kt2 = Weighting coefficient kt3 = Weighting coefficient kt4 = 0.9

Further, the situation-related information 37 indicates that the weighting coefficient kt1 is larger than the weighting coefficient kv1 for the weighting coefficient related to the identification information “meeting”. Similarly, the situation-related information 37 indicates that the weighting coefficient kt2 is larger than the weighting coefficient kv2, the weighting coefficient kt3 is larger than the weighting coefficient kv3, and the weighting coefficient kt4 is larger than the weighting coefficient kv4. Specifically, the weighting coefficients related to the identification information "meeting" shown in FIG. 7 are as follows.
Weighting coefficient kv1 = 0.4
Weighting coefficient kt1 = 0.6
Weighting coefficient kv2 = 0.2
Weighting coefficient kt2 = 0.8
Weighting coefficient kv3 = 0.3
Weighting coefficient kt3 = 0.7
Weighting coefficient kv4 = 0.2
Weighting coefficient kt4 = 0.8

The explanation is returned to FIG. The estimation unit 258 estimates one or more emotions held by the user U based on the correction result of the correction unit 257, that is, the corrected voice evaluation values CEV1 to CEV4 and the corrected character evaluation values CET1 to CET4. To do. For example, the estimation unit 258 executes the following calculation to generate an evaluation value E1 corresponding to joy, an evaluation value E2 corresponding to anger, an evaluation value E3 corresponding to sadness, and an evaluation value E4 corresponding to normal times. To do.
E1 = CEV1 + CET1
E2 = CEV2 + CET2
E3 = CEV3 + CET3
E4 = CEV4 + CET4

Then, the estimation unit 258 estimates the emotion held by the user U according to, for example, one of the two modes shown below. In the first aspect, the estimation unit 258 outputs an emotion information EI indicating an emotion corresponding to the largest evaluation value E among the evaluation value E1, the evaluation value E2, the evaluation value E3, and the evaluation value E4. In the second aspect, the estimation unit 258 compares each of the evaluation values E1 to E4 with the threshold value, and outputs an emotion information EI indicating an emotion corresponding to the evaluation value E exceeding the threshold value. In the second aspect, the emotional information EI may exhibit multiple emotions, such as exhibiting anger and sadness.
The emotional information EI has, for example, the following two aspects. The first aspect of the emotion information EI is an identifier indicating one or more emotions held by the estimated user U. The identifier indicating emotion includes an identifier indicating joy, an identifier indicating anger, an identifier indicating sadness, and an identifier indicating normality. The second aspect of the emotion information EI is an identifier indicating one or more emotions held by the estimated user U and an evaluation value E of the emotions held by the estimated user U.

The output unit 27 outputs the information obtained by executing the processing corresponding to one or more emotions indicated by the emotion information EI with respect to the recognition character string RT obtained by the recognition unit 254. There are two modes of processing according to emotions, for example, as shown below.
The first aspect of the process according to the emotion is a process of adding a figure embodying the emotion to the recognition character string RT. The figures that embody emotions are, for example, pictograms that embody emotions and emoticons that embody emotions. A pictogram is an image associated with a character code. The character code is, for example, Unicode. An emoticon is a character string that expresses a face by combining symbols and characters. In the following description, a figure that embodies emotions will be described as a pictogram that embodies emotions. The pictogram that embodies joy is, for example, a pictogram that shows a smile. The pictogram that embodies anger is, for example, a pictogram that shows the face of anger. The pictogram that embodies sadness is, for example, a pictogram that shows a crying face. Further, when the emotion information EI is the second aspect, the output unit 27 is a pictogram that embodies the emotion indicated by the emotion information EI and has an intensity corresponding to the evaluation value E included in the emotion information EI. May be determined as a pictogram to be added to the recognition character string RT. For example, when the emotion indicated by the emotion information EI is sadness and the evaluation value E included in the emotion information EI is equal to or less than a predetermined threshold value, the output unit 27 recognizes a pictogram indicating a face spilling tears. Determined as a pictogram to be added to. On the other hand, when the emotion indicated by the emotion information EI is sad and the evaluation value E included in the emotion information EI is larger than a predetermined threshold value, the output unit 27 adds a pictogram indicating a crying face to the recognition character string RT. Decide as a pictogram to do. The emoji showing a crying face embodies a higher degree of sadness than the emoji showing a tearful face.
The output unit 27 outputs a character string with a pictogram obtained by adding a pictogram to the recognition character string RT. For example, there are two positions for adding pictograms as shown below. The first position is the end of the recognition string RT. The second position is next to the character string included in the emotion classification information 33 in the recognition character string RT. The display device 51 displays an image based on the character string with pictograms output by the output unit 27.

The second aspect of the emotion-based process is a process of generating a synthetic voice read aloud by adding emotion-based intonation. Inflection is, for example, increasing or decreasing the reading speed, or increasing or decreasing the volume. Pleasure-based intonation is, for example, speeding up reading. Anger-based intonation is, for example, increasing the volume. Sadness-based intonation is, for example, reducing the volume. The output unit 27 outputs information indicating a synthetic voice read aloud with an emotion-based intonation added. Then, the output unit 27 adds emotion-based intonation to the synthetic voice indicated by the generated data, and outputs information indicating the synthetic voice read by adding emotion-based intonation. The sound emitting device 53 emits a synthetic voice indicated by the data output by the output unit 27.

1.2. Operation of the First Embodiment Next, the operation of the user device 1 will be described with reference to FIG.

FIG. 8 is a flowchart showing the operation of the user device 1. The acquisition unit 21 acquires the voice information VI (step S1). The feature amount generation unit 251 generates a feature amount from the voice information VI (step S2). Then, the first evaluation unit 252 inputs the feature amount into the learning model LM and acquires the voice evaluation value EV of each emotion (step S3).

On the other hand, the recognition unit 254 outputs the recognition character string RT based on the voice information VI (step S4). Next, the analysis unit 2552 executes the morphological analysis process and outputs the corrected recognition character string CRT (step S5). Then, the calculation unit 2554 generates the character evaluation values ET1 to ET4 of each emotion from the corrected recognition character string CRT (step S6).

The status information generation unit 23 generates status information SiI based on the position information PoI obtained from the GPS device 8, the inertial information IFI obtained from the inertial sensor 7, and the schedule information 35 (step S7). Then, the correction unit 257 corrects the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 based on the weighting coefficients kv1 to kv4 and kt1 to kt4 according to the situation indicated by the situation information SiI (step S8). ..

The estimation unit 258 estimates one or more emotions held by the user U based on the corrected voice evaluation values CEV1 to CEV4 and the corrected character evaluation values CET1 to CET4, and outputs emotion information EI ( Step S9). The output unit 27 outputs the information obtained by executing the processing corresponding to the emotion indicated by the emotion information EI on the recognition character string RT (step S10). After the processing in step S10 is completed, the user device 1 ends a series of processing shown in FIG.

1.3. Effect of 1st Embodiment According to the above description, the user apparatus 1 corrects the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 based on the situation information SiI. When the situation information SiI indicates a situation in which it is difficult to express emotions in voice, it can be said that the accuracy of the voice evaluation values EV1 to EV4 is lowered. Therefore, when the situation information SiI indicates a situation in which it is difficult to express emotions to the voice, the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 are corrected by a method of lowering the voice evaluation values EV1 to EV4. Therefore, the emotion held by the user U can be corrected with high accuracy.

Further, in the user device 1, the feature amount generation unit 251 uses a learning model LM that has learned the relationship between a plurality of feature quantities corresponding to human voice and the intensity of each of the plurality of emotions held by the human who emits the voice. The generated feature amount is input, and the voice evaluation values EV1 to EV4 are acquired from the learning model LM. By using the learning model LM, the user device 1 can acquire accurate voice evaluation values EV1 to EV4.

In addition, the learning model LM has learned the relationship between a plurality of features corresponding to human voices and the intensity of each of the plurality of emotions held by the human being who emits the voices for a plurality of humans. In other words, the learning model LM is generated by teacher data based on voice information VIs of a plurality of users. Therefore, the learning model LM is a general purpose model that is not tailored for a particular individual. In the first embodiment, since the general-purpose learning model LM can be used, it is easy to introduce as compared with the case of using a learning model adjusted for a specific individual.

Further, the correction unit 257 sets a weighting coefficient for each of the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 according to the situation indicated by the situation information SiI with reference to the situation-related information 37. By referring to the situation-related information 37, it is possible to accurately identify whether or not the user U is placed in a situation in which it is difficult to express emotions.

The situation-related information 37 indicates that the weighting coefficient kv1 is larger than the weighting coefficient kt1 with respect to the weighting coefficient related to the identification information indicating the situation where the user U is in the private space. Further, the situation-related information 37 indicates that the weighting coefficient kv2 is larger than the weighting coefficient kt2, the weighting coefficient kv3 is larger than the weighting coefficient kt3, and the weighting coefficient kv4 is larger than the weighting coefficient kt4.
When the user U is in a private space such as home, the emotions are easily expressed in the voice because the user U does not catch the eyes of others. Therefore, the weighting coefficient kv1 is larger than the weighting coefficient kt1, the weighting coefficient kv2 is larger than the weighting coefficient kt2, the weighting coefficient kv3 is larger than the weighting coefficient kt3, and the weighting coefficient kv4 is larger than the weighting coefficient kt4. , Joy, anger, sadness, and normality can be estimated accurately.

The situation-related information 37 indicates that the weighting coefficient kt1 is larger than the weighting coefficient kv1 with respect to the weighting coefficient related to the identification information indicating the situation where the user U is in the non-private space. Further, the situation-related information 37 indicates that the weighting coefficient kt2 is larger than the weighting coefficient kv2, the weighting coefficient kt3 is larger than the weighting coefficient kv3, and the weighting coefficient kt4 is larger than the weighting coefficient kv4.
As an example of having User U in a non-private space, when User U is using a train, speaking with emotion will attract the attention of others, so it is common to speak without feeling. It can be said that. The situation where the user U is using the train is one of the situations where it is difficult for the user U to express his emotions in the voice. As another example in which the user U is in a non-private space, it can be said that in a situation where the user U is in a meeting, it is rare to utter with emotion. The situation in which the user U is in a meeting is one of the situations in which the user U does not easily express his / her emotions in the voice.
Therefore, when the user U is in the non-private space, the weighting coefficient kt1 is larger than the weighting coefficient kv1, the weighting coefficient kt2 is larger than the weighting coefficient kv2, the weighting coefficient kt3 is larger than the weighting coefficient kv3, and the weighting coefficient kt4 is. By being larger than the weighting coefficient kv4, the user device 1 can accurately estimate joy, anger, sadness, and normality.

2. 2. Second Embodiment The user device 1a according to the second embodiment sets the voice evaluation values CEV1 to CEV4 and the character evaluation values CET1 to CET4 based on the degree to which emotions are expressed in the voice according to the recognition character string RT. It differs from the user device 1 according to the first embodiment in that it is corrected. Regarding the elements whose actions or functions are the same as those of the first embodiment in the second embodiment illustrated below, the reference numerals are used in the above description, and detailed description of each is appropriately omitted.

2.1. Function of the second embodiment FIG. 9 is a block diagram showing a user device 1a according to the second embodiment. The user device 1a is realized by a computer system including a processing device 2a, a storage device 3a, an input device 4, an output device 5, a communication device 6, an inertial sensor 7, and a GPS device 8. The storage device 3a is a recording medium that can be read by the processing device 2a, and is a plurality of programs including the control program PRa executed by the processing device 2a, analysis dictionary information 31, emotion classification information 33, schedule information 35, and situation-related information. 37, the character string relation information 38, and the learning model LM are stored.

The character string-related information 38 is used for determining the degree to which emotions are expressed in the voice according to the recognition character string RT. The processing device 2a functions as an acquisition unit 21, a situation information generation unit 23, an emotion information generation unit 25a, and an output unit 27 by reading and executing the control program PRa from the storage device 3a.

FIG. 10 is a diagram showing an outline of the functions of the user device 1a in the second embodiment. The emotion information generation unit 25a includes a feature amount generation unit 251, a first evaluation unit 252, a recognition unit 254, a second evaluation unit 255, a correction unit 257a, and an estimation unit 258.

The correction unit 257a has voice evaluation values EV1 to EV4 and character evaluation values based on the degree to which emotions are expressed in the voice of the user U when the user U utters the recognition character string RT and the situation information SiI. Correct ET1 to ET4. For example, the correction unit 257a corrects the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 according to one of the two modes shown below.

The correction unit 257a in the first aspect classifies the character string into a character string having a low degree of emotional expression in the voice of the user U when uttered and a character string having a high degree of emotional expression. Then, when the recognition character string RT includes a character string having a low degree of emotional expression, the correction unit 257a determines that the degree of emotional expression in the voice of the user U is low when the recognition character string RT is uttered. judge. On the other hand, when the recognition character string RT does not include a character string having a low degree of emotional expression, the correction unit 257a indicates that the emotion is expressed in the voice of the user U when the recognition character string RT is uttered. Judged as high. Then, when the correction unit 257a utters the recognition character string RT, the emotion is expressed in the voice of the user U when the degree of emotion is expressed in the voice of the user U is low, and when the recognition character string RT is uttered, the emotion is expressed in the voice of the user U. Compared with the case where the degree is high, the weighting coefficients kv1 to kv4 corresponding to the voice evaluation values EV1 to EV4 are lowered.
For example, "thank you" has a general meaning of gratitude. When the user U utters "Thank you", the emotions that the user U has become more joyful. However, since character strings that are used for general purposes are familiar to people, they are often uttered without undulations, and it can be said that the degree to which emotions are expressed in the voice is low. Therefore, when the recognition character string RT includes "thank you", the correction unit 257a lowers the weighting coefficients kv1 to kv4 corresponding to the voice evaluation values EV1 to EV4. For example, the correction unit 257a subtracts a predetermined value from the weighting coefficients kv1 to kv4 corresponding to the voice evaluation values EV1 to EV4, and adds the predetermined value to the weighting coefficients kt1 to kt4 corresponding to the character evaluation values ET1 to ET4.

The correction unit 257a in the second aspect is included in the recognition character string RT for each of the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 based on the character string relation information 38 and the situation relation information 37. The weighting coefficients kv1 to kv4 and kt1 to kt4 are set according to the situation indicated by the character string and the situation information SiI. Then, the correction unit 257a corrects the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 based on the weighting coefficient according to the situation indicated by the recognition character string RT and the situation information SiI.

FIG. 11 is a diagram showing an example of the stored contents of the character string-related information 38. The character string-related information 38 includes voice evaluation values EV1 to EV4 and character evaluation values ET1 set based on the character string uttered by a human being and the degree to which emotions are expressed in the human voice when the character string is uttered. The relationship between the increase / decrease values Δkv1 to Δkv4 and Δkt1 to Δkt4 of the weighting coefficients kv1 to kv4 and kt1 to kv4 with respect to ET4 is shown.
The increase / decrease values Δkv1 to Δkv4 and the increase / decrease values Δkt1 to Δkt4 have the following conditions.

Δkv1 + Δkt1 = Δkv2 + Δkt2 = Δkv3 + Δkt3 = Δkv4 + Δkt4 = 0

The character string-related information 38 shown in FIG. 10 indicates that when a human utters the character string “thank you”, the increase / decrease values Δkv1 to Δkv4 and Δkt1 to Δkt4 are set as shown below.
Increase / decrease value Δkv1 = -0.2
Increase / decrease value Δkt1 = 0.2
Increase / decrease value Δkv2 = -0.2
Increase / decrease value Δkt2 = 0.2
Increase / decrease value Δkv3 = -0.2
Increase / decrease value Δkt3 = 0.2
Increase / decrease value Δkv4 = -0.1
Increase / decrease value Δkt4 = 0.1
Of the increase / decrease values Δkv1 to Δkv4 and Δkt1 to Δkt4, a value of 0 may be included. For example, the increase / decrease values Δkv1 and Δkt1 for the character string “Thank you” are the above-mentioned values, and the increase / decrease values Δkv2 to Δkv4 and Δkt2 to Δkt4 may be 0.

When the character string included in the character string-related information 38 is included in the recognition character string RT, the correction unit 257a executes the following calculation to perform the corrected voice evaluation values CEV1 to CEV4 and the corrected character evaluation. The values CET1 to CET4 are output.

CEV1 = (kv1 + Δkv1) x EV1
CEV2 = (kv2 + Δkv2) x EV2
CEV3 = (kv3 + Δkv3) x EV3
CEV4 = (kv4 + Δkv4) x EV4
CET1 = (kt1 + Δkt1) × ET1
CET2 = (kt2 + Δkt2) × ET2
CET3 = (kt3 + Δkt3) x ET3
CET4 = (kt4 + Δkt4) x ET4

The increase / decrease values Δkv1 to Δkv4 and the increase / decrease values Δkt1 to Δkt4 are values corresponding to the character strings included in the recognition character string RT in the character string relation information 38. When the sum of the weighting coefficient kvx and the increase / decrease value Δkvx is less than 0, the correction unit 257a outputs the corrected voice evaluation value CEVx as 0, and the sum of the weighting coefficient kvx and the increase / decrease value Δkvx is more than 1. If it is large, the corrected voice evaluation value CEVx is output as the same value as the voice evaluation value EVx. Similarly, when the sum of the weighting coefficient ktx and the increase / decrease value Δktx is less than 0, the correction unit 257a outputs the corrected character evaluation value CETx as 0, and the sum of the weighting coefficient ktx and the increase / decrease value Δktx is 1. If it is larger than, the corrected character evaluation value CETx is output as the same value as the character evaluation value ETx. x is an integer from 1 to 4.
In the following description, the correction unit 257a will be described as a second aspect.

2.2. Operation of the Second Embodiment Next, the operation of the user device 1a will be described with reference to FIG.

FIG. 12 is a flowchart showing the operation of the user device 1a. Since the processes of steps S21 to S27 shown in FIG. 12 are the same as the processes of steps S1 to S7 shown in FIG. 8, the description thereof will be omitted.

After the processing in step S27 is completed, the correction unit 257a determines whether or not the character string in the character string-related information 38 is included in the recognition character string RT (step S28). When the determination result in step S28 is affirmative, the correction unit 257a sets the voice evaluation values EV1 to EV4 based on the weighting coefficients kv1 to kv4 according to the situation indicated by the character string included in the recognition character string RT and the situation information SiI. The character evaluation values ET1 to ET4 are corrected (step S29). On the other hand, when the determination result in step S28 is negative, the correction unit 257a sets the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 based on the weighting coefficients kv1 to kv4 according to the situation indicated by the situation information SiI. Correct (step S30).

After the processing of step S29 or the processing of step S30 is completed, the estimation unit 258 has one or more of the voice evaluation values CEV1 to CEV4 after the correction and the character evaluation values CET1 to CET4 after the correction. The emotion is estimated and the emotion information EI is output (step S31). The output unit 27 outputs the information obtained by executing the processing corresponding to the emotion indicated by the emotion information EI on the recognition character string RT (step S32). After the processing in step S10 is completed, the user device 1 ends a series of processing shown in FIG.

2.3. Effect of the Second Embodiment According to the above description, the user device 1a is based on the degree to which emotions are expressed in the voice of the user U when the user U utters the recognition character string RT and the situation information SiI. , Voice evaluation values EV1 to EV4, and character evaluation values ET1 to ET4 are corrected. In general, there are character strings that express emotions when they are uttered and characters that express emotions less when they are uttered. Therefore, the degree to which emotions are expressed when they are uttered. By correcting the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 according to the above, the emotion held by the user U can be estimated accurately.

Further, the user device 1a refers to the situation-related information 37 and the character string-related information 38, and refers to the characters included in the recognition character string RT for each of the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4. The weighting coefficients kv1 to kv4 and kt1 to kt4 are set according to the situation indicated by the column and the situation information SiI. By setting the weighting coefficients kv1 to kv4 and kt1 to kt4 according to the situation indicated by the character string included in the recognition character string RT and the situation information SiI, the emotion held by the user U can be estimated accurately.

3. 3. Modifications The present invention is not limited to the embodiments exemplified above. A specific mode of modification is illustrated below. Two or more aspects arbitrarily selected from the following examples may be merged.

(1) In each of the above aspects, the processing device 2 of the user device 1 functions as an acquisition unit 21, a situation information generation unit 23, an emotion information generation unit 25, and an output unit 27, but the present invention is not limited to this. In the first modification, the acquisition unit 21, the situation information generation unit 23, the emotion information generation unit 25, and the output unit 27 are distributed by the user device 1b and the server device 10.

FIG. 13 is a diagram showing the overall configuration of the emotion estimation system SYS. The emotion estimation system SYS includes a user device 1b owned by the user U, a network NW, and a server device 10. The user device 1b is an example of a “terminal device”. The server device 10 is an example of the “emotion estimation device” in the first modification.

FIG. 14 is a block diagram showing the configuration of the user device 1b. The user device 1b is realized by a computer system including a processing device 2b, a storage device 3b, an input device 4, an output device 5, a communication device 6, an inertial sensor 7, and a GPS device 8. The storage device 3b is a recording medium that can be read by the processing device 2b, and stores a plurality of programs including the control program PRb executed by the processing device 2b, and schedule information 35.

The processing device 2b functions as an acquisition unit 21, a status information generation unit 23, and an output unit 27 by reading and executing the control program PRb from the storage device 3b.

The communication device 6 transmits the voice information VI and the status information SiI to the server device 10, and receives the recognition character string RT and the emotion information EI from the server device 10.

FIG. 15 is a block diagram showing the configuration of the server device 10. The server device 10 is realized by a computer system including a processing device 2B, a storage device 3B, and a communication device 6B. Each element of the server device 10 is connected to each other by a single unit or a plurality of buses 9B for communicating information. The storage device 3B is a recording medium that can be read by the processing device 2B, and includes a plurality of programs including a control program PRB executed by the processing device 2B, analysis dictionary information 31, emotion classification information 33, situation-related information 37, and Store the learning model LM.

The processing device 2B functions as the emotion information generation unit 25 by reading and executing the control program PRB from the storage device 3B.

The communication device 6B receives the voice information VI and the status information SiI from the user device 1b, and transmits the recognition character string RT and the emotion information EI to the user device 1b.

As described above, according to the first modification, the acquisition unit 21, the situation information generation unit 23, the emotion information generation unit 25, and the output unit 27 can be distributed between the user device 1 and the server device 10.

(2) In the second embodiment, the correction unit 257a evaluates the voice based on the degree to which emotions are expressed in the voice of the user U when the user U utters the recognition character string RT and the situation information SiI. EV1 to EV4 and character evaluation values ET1 to ET4 have been corrected, but the present invention is not limited to this. For example, the correction unit 257a sets the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 based on the degree to which emotions are expressed in the voice of the user U when the user U utters the recognition character string RT. It may be corrected.

(3) Although it is described that an example of a private space is in the home, it is not limited to this. For example, the private space may be in the hotel room where the user U stays. For example, it is assumed that the user device 1 has a function of controlling the locking and unlocking of the door of a hotel room. Based on this premise, the movement range of the user U is within the predetermined range from the time when the user device 1 instructs the unlocking of the door of the hotel room and the predetermined time elapses from the time when the unlocking is instructed. In some cases, the user device 1 determines that the user U is in the hotel room where the user U is staying.

(4) Non-private space is described as being in public transportation and in the workplace, but it is not limited to these. For example, non-private spaces include schools, hospitals, libraries, and so on.

(5) As an example of the situation of using public transportation, the situation of using a train was mentioned, but the situation of using public transportation is not limited to the situation of using a train. For example, the situation of using public transportation may include the situation of being in a station yard. For example, when the user device 1 has a transportation IC (Integrated Circuit) card function, the user U uses public transportation when the user U enters the station yard by the transportation IC card function. You may judge that. In addition, public transportation includes not only trains but also fixed-route buses, taxis, ferries, and passenger planes.

(6) The output unit 27 outputs information obtained by executing the processing corresponding to the emotion indicated by the emotion information EI on the recognition character string RT, but the present invention is not limited to this. For example, the output unit 27 may output a character string indicating the emotion indicated by the emotion information EI to the display device 51, or output the pictogram indicated by the emotion information EI to the display device 51.

(7) The user device 1 does not have to have the sound collecting device 41. When the sound collecting device 41 is not provided, the user device 1 may acquire the voice information VI via the communication device 6 or may acquire the voice information VI stored in the storage device 3.

(8) The user device 1 does not have to have the sound emitting device 53.

(9) The user device 1 may be a smart speaker. When the user device 1 is a smart speaker, the user device 1 does not have to have the touch panel 43 and the display device 51.

(10) As shown in FIG. 4, the emotion classification information 33 is either joy, anger, sadness, or normal for each of a plurality of morphemes utilized by a certain word, such as "win" and "win". However, it is not limited to this. For example, the emotion classification information 33 may classify the character string registered in the prototype data of the analysis dictionary information 31 into any of joy, anger, sadness, and normal. For example, the emotion classification information 33 classifies the character strings “happy”, “pass”, and “win” registered in the prototype data of the analysis dictionary information 31 into joy. The calculation unit 2554 decomposes the corrected recognition character string CRT for each morpheme, and converts the decomposed morpheme into a character string registered in the original form data of the analysis dictionary information 31. Then, when the character string obtained by conversion and the character string included in the emotion classification information 33 match, the calculation unit 2554 determines the emotion corresponding to the character string included in the corrected recognition character string CRT. Increase the character evaluation value ET.

(11) Although the calculation unit 2554 calculated the character evaluation value ET for each emotion with respect to the corrected recognition character string CRT, the character evaluation value ET for each emotion may be calculated with respect to the recognition character string RT. .. However, the recognition character string RT includes a character string that is unnecessary for estimating emotions. Therefore, by calculating the character evaluation value ET for each emotion for the corrected recognition character string CRT, the emotion is estimated as compared with the case where the character evaluation value ET for each emotion is calculated for the recognition character string RT. The accuracy can be improved.

(12) Although the example in which the user U speaks Japanese is used, each of the above aspects can be applied regardless of the language spoken by the user. For example, even when the user U speaks English, French, Chinese, or the like other than Japanese, each of the above aspects can be applied. For example, when the user U speaks English, the analysis dictionary information 31 is information related to English morphemes, and the emotion classification information 33 is information that classifies English words into any of joy, anger, sadness, and normal. It should be.

(13) The block diagram used in the description of each of the above-described aspects shows a block of functional units. These functional blocks (components) are realized by any combination of hardware and / or software. Further, the means for realizing each functional block is not particularly limited. That is, each functional block may be realized by one physically and / or logically coupled device, or directly and / or indirectly by two or more physically and / or logically separated devices. (For example, wired and / or wireless) may be connected and realized by these plurality of devices.

(14) The order of the processing procedures, sequences, flowcharts, etc. in each of the above-described aspects may be changed as long as there is no contradiction. For example, the methods described herein present elements of various steps in an exemplary order, and are not limited to the particular order presented.

(15) In each of the above-described aspects, the input / output information and the like may be stored in a specific place (for example, a memory) or may be managed by a management table. Input / output information and the like can be overwritten, updated, or added. The output information and the like may be deleted. The input information or the like may be transmitted to another device.

(16) In each of the above-described aspects, the determination may be made by a value represented by 1 bit (0 or 1) or by a boolean value (Boolean: true or false). , May be done by numerical comparison (eg, comparison with a given value).

(17) In each of the above-described aspects, a portable information processing device such as a smartphone is illustrated as the user device 1, but the specific form of the user device 1 is arbitrary and is not limited to the above-mentioned examples of each form. .. For example, a portable or stationary personal computer may be used as the user device 1.

(18) In each of the above-described aspects, the storage device 3 is a recording medium that can be read by the processing device 2, and examples thereof include a ROM and a RAM. Discs, Blu-ray® discs), smart cards, flash memory devices (eg cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppies (registered trademarks) ) Disks, magnetic strips, databases, servers and other suitable storage media. The program may also be transmitted from the network. The program may also be transmitted from the communication network via a telecommunication line.

(19) Each of the above-described aspects includes LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G, 5G, FRA (Future Radio Access), and W-CDMA (registered trademark). , GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark) ), Other systems that utilize suitable systems and / or next-generation systems that are extended based on them.

(20) In each of the above aspects, the information, signals, and the like described may be represented using any of a variety of different techniques. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description are voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these. It may be represented by a combination of.
In addition, the terms described in the present specification and / or the terms necessary for understanding the present specification may be replaced with terms having the same or similar meanings.

(21) Each of the functions illustrated in FIGS. 2, 6, 9, 10, 14, and 15 is realized by any combination of hardware and software. Further, each function may be realized by a single device, or may be realized by two or more devices configured as separate bodies from each other.

(22) The programs exemplified in each of the above-described embodiments are called instructions, instruction sets, codes, code segments regardless of whether they are called software, firmware, middleware, microcode or hardware description language, or by other names. , Program code, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, execution threads, procedures or functions, etc. should be broadly interpreted to mean.
Further, software, instructions, and the like may be transmitted and received via a transmission medium. For example, the software uses wired technology such as coaxial cable, fiber optic cable, twist pair and digital subscriber line (DSL) and / or wireless technology such as infrared, wireless and microwave to websites, servers, or other When transmitted from a remote source, these wired and / or wireless technologies are included within the definition of transmission medium.

(23) In each of the above-described embodiments, the information, parameters, etc. may be represented by absolute values, relative values from a predetermined value, or other corresponding information. May be good.

(24) The names used for the above-mentioned parameters are not limited in any respect. Further, mathematical formulas and the like using these parameters may differ from those expressly disclosed herein.

(25) In each of the above-described embodiments, the user device 1 includes a case where it is a mobile station. Mobile stations can be subscriber stations, mobile units, subscriber units, wireless units, remote units, mobile devices, wireless devices, wireless communication devices, remote devices, mobile subscriber stations, access terminals, mobile terminals, wireless, depending on the trader. It may also be referred to as a terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable term.

(26) In each of the above embodiments, the phrase "based on" does not mean "based on" unless otherwise stated. In other words, the statement "based on" means both "based only" and "at least based on".

(27) Any reference to elements using designations such as "first", "second" as used herein does not generally limit the quantity or order of those elements. These designations can be used herein as a convenient way to distinguish between two or more elements. Thus, references to the first and second elements do not mean that only two elements can be adopted there, or that the first element must somehow precede the second element.

(28) As long as "inclusion," "comprising," and variations thereof in each of the embodiments described above are used herein or within the scope of the claims, these terms are used. As with the term "prepare", it is intended to be inclusive. Furthermore, the term "or" as used herein or in the claims is intended not to be an exclusive OR.

(29) In the whole of the present application, if articles are added by translation, for example, a, an and the in English, unless the context clearly indicates that these articles are not. Including multiple.

(30) It will be apparent to those skilled in the art that the present invention is not limited to the embodiments described herein. The present invention can be implemented as modifications and modifications without departing from the spirit and scope of the present invention, which is determined based on the description of the scope of claims. Therefore, the description herein is for illustrative purposes and has no limiting implications for the present invention. In addition, a plurality of aspects selected from the aspects illustrated in the present specification may be combined.

1, 1a, 1b ... user device, 6 ... communication device, 10 ... server device, 21 ... acquisition unit, 23 ... situation information generation unit, 25, 25a ... emotion information generation unit, 27 ... output unit, 37 ... situation-related information , 38 ... Character string related information, 41 ... Sound collector, 251 ... Feature amount generation unit, 252 ... First evaluation unit 254 ... Recognition unit, 255 ... Second evaluation unit, 257 ... 257a ... Correction unit, 258 ... Estimating unit , CET1 ... Corrected character evaluation value, CET2 ... Corrected character evaluation value, CET3 ... Corrected character evaluation value, CET4 ... Corrected character evaluation value, CEV1 ... Corrected voice evaluation value, CEV2 ... After correction Voice evaluation value, CEV3 ... Corrected voice evaluation value, CEV4 ... Corrected voice evaluation value, EI ... Emotion information, ET1 ... Character evaluation value, ET2 ... Character evaluation value, ET3 ... Character evaluation value, ET4 ... Character evaluation Value, EV1 ... Voice evaluation value, EV2 ... Voice evaluation value, EV3 ... Voice evaluation value, EV4 ... Voice evaluation value, kt1 ... Weighting coefficient, kt2 ... Weighting coefficient, kt3 ... Weighting coefficient, kt4 ... Weighting coefficient, kv1 ... Weighting coefficient , Kv2 ... Weighting coefficient, kv3 ... Weighting coefficient, kv4 ... Weighting coefficient.

Claims (9)

A generation unit that generates a feature amount for the user's voice based on the user's voice information,
Based on the feature amount, a first evaluation unit that generates a first voice evaluation value indicating the intensity with which the user has a first emotion and a second voice evaluation value indicating the intensity with which the user has a second emotion. ,
A recognition unit that generates a recognition character string indicating the utterance content of the user based on the voice information,
Based on the recognition character string, a second character evaluation value indicating the strength with which the user has the first emotion and a second character evaluation value indicating the strength with the user having the second emotion are generated. Evaluation department and
A correction unit that corrects the first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the second character evaluation value based on the situation information indicating the user's situation.
An estimation unit that estimates one or more emotions held by the user based on the correction result of the correction unit, and an estimation unit.
Emotion estimator equipped with. The first evaluation unit
The feature amount generated by the generation unit is applied to a learning model in which the relationship between the feature amount corresponding to the human voice and the intensity of each of the first emotion and the second emotion held by the person who emits the voice is learned. Input and acquire the first voice evaluation value and the second voice evaluation value from the learning model.
The emotion estimation device according to claim 1. The learning model has already learned the relationship between a plurality of features corresponding to the human voice and the intensity of each of the first emotion and the second emotion held by the person who emitted the voice for a plurality of humans. is there,
The emotion estimation device according to claim 2. The correction unit
Identification information indicating a situation that a human can take, and each of the first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the second character evaluation value set according to the situation. With reference to the contextual information that shows the relationship with the weighting factor for
A weighting coefficient for each of the first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the second character evaluation value according to the situation indicated by the situation information is set.
The first voice evaluation value and the second voice evaluation value are based on the weighting coefficients for each of the first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the second character evaluation value. Correct the value, the first character evaluation value, and the second character evaluation value.
The emotion estimation device according to any one of claims 1 to 3. The situation-related information is
Regarding the weighting coefficient related to the identification information indicating the situation where the user is in the private space where entry is prohibited without the permission of the user, the weighting coefficient for the first voice evaluation value is larger than the weighting coefficient for the first character evaluation value. And, the weighting coefficient for the second voice evaluation value is larger than the weighting coefficient for the second character evaluation value.
Show that
The emotion estimation device according to claim 4. The situation-related information is
Regarding the weighting coefficient related to the identification information indicating the situation where the user is in the non-private space that can be entered without the permission of the user, the weighting coefficient for the first character evaluation value is larger than the weighting coefficient for the first voice evaluation value. And, the weighting coefficient for the second character evaluation value is larger than the weighting coefficient for the second voice evaluation value.
Show that
The emotion estimation device according to claim 4. The correction unit
The first voice evaluation value, the second voice evaluation value, and the first voice evaluation value are based on the degree to which emotions are expressed in the user's voice when the user utters the recognition character string and the situation information. Correct the character evaluation value and the second character evaluation value.
The emotion estimation device according to any one of claims 1 to 3. The correction unit
Identification information indicating a situation that a human can take, and the first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the second character evaluation value set according to the situation. Situation-related information showing the relationship with the weighting coefficient for each,
The first voice evaluation value, the second voice evaluation value, and the first character evaluation set based on the character string uttered by a human and the degree to which emotions are expressed in the human voice when the character string is uttered. Character string relationship information indicating the relationship between the value and the increase / decrease value of the weighting coefficient for each of the second character evaluation values, and
Refer to
The character string included in the recognition character string and the situation indicated by the situation information for each of the first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the second character evaluation value. Set the weighting coefficient according to
The first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the first character evaluation value, based on the character string included in the recognition character string and the weighting coefficient according to the situation indicated by the situation information. Correct the two-character evaluation value,
The emotion estimation device according to any one of claims 1 to 3. An emotion estimation system including the emotion estimation device according to any one of claims 1 to 8 and a terminal device capable of communicating with the emotion estimation device.
The terminal device is
A sound collecting device that collects the user's voice and
The situation information generation unit that generates the situation information and
The voice information indicating the user's voice and the situation information are transmitted to the emotion estimation device, and the recognition character string and one or more emotions held by the user estimated by the estimation unit are transmitted from the emotion estimation device. A communication device that receives emotional information indicating
An output unit that outputs information obtained by executing processing according to the emotion indicated by the emotion information on the recognition character string, and an output unit.
Emotion estimation system with.

Images