JP6638435B2 - Personal adaptation method of emotion estimator, emotion estimation device and program - Google Patents

Description

  The present invention relates to a personal adaptation method of an emotion estimator, an emotion estimation device, and a program.

  A technology for estimating a speaker's emotional state using an emotion estimation device generated by machine learning using a group of voice data in which emotions are labeled as teacher data is being developed. For example, Patent Literature 1 discloses a technique in which a change amount of each of a sound intensity, a sound tempo, and a intonation of a sound is obtained, and an emotional state of a speaker is estimated based on the obtained change amounts. Patent Document 1 discloses an emotion state of an arbitrary speaker at the time of uttering using an emotion estimation device generated as teacher data using voice data in which an unspecified number of speakers uttered in emotion states such as emotions and sorrows. A technology for adaptively learning an emotion estimation device by estimating the emotion data, adding a label such as estimated emotion, emotion, and so on to the voice data and adding the label to teacher data is disclosed.

JP 2005-352154 A

  By the way, the emotion estimating device generated by using the voice data of the unspecified number of speakers as the teacher data uses the common feature of the feature amount of the voice data of the unspecified number of speakers and the feature amount of the voice data of the speaker to be analyzed. The speaker's emotional state is estimated based on the characteristics. Therefore, when the characteristic amount of the audio data to be analyzed is close to the average value of the characteristic amounts of the unspecified number of audio data, the estimation accuracy by the emotion estimation device becomes higher. However, as the difference between the feature value of the voice data to be analyzed and the average value of the feature values of the unspecified number of voice data increases, the estimation accuracy of the emotion estimation device decreases.

  In the technology disclosed in Patent Literature 1, in an adaptive process for adapting an emotion estimation device as an emotion estimation device for a specific individual, a characteristic amount of voice data to be analyzed and an average value of the characteristic amount of an unspecified number of audio data are compared. Speech data to which labels such as emotions and sorrows are given in a state where the estimation accuracy is poor due to a large difference is also added as it is as teacher data. Therefore, it is difficult to improve the estimation accuracy as an emotion estimation device for a specific individual when the difference between the characteristic amount of the audio data of the specific individual and the average value of the characteristic amounts of an unspecified number of audio data is large. There's a problem.

  The present invention has been made in view of such a situation, and provides an individual adaptation method, an emotion estimation device, and a program of an emotion estimator that can improve the estimation accuracy of an emotion estimation device for a specific individual. With the goal.

In order to achieve the above object, a personal adaptation method of the emotion estimator according to the first aspect of the present invention includes:
An emotion estimator that estimates the emotional state of a speaker when speaking, which is generated as teacher data using voice data spoken by an unspecified number of speakers, as an emotion estimator that estimates the emotional state of a specific individual when speaking A personal adaptation method of an emotion estimator for personal adaptation,
An obtaining step of obtaining voice data spoken by the specific individual;
A feature extraction step of extracting features of the audio data;
A frequency analysis step of classifying the extracted features into a plurality of patterns for each specific extraction section and analyzing an appearance frequency for each pattern;
A first label assigning step of assigning a neutral label indicating that the emotional state at the time of speech is a calm state to the audio data of the specific extraction section in which the appearance frequency of the pattern is determined to be equal to or greater than a threshold value;
Generating the teacher data for the specific individual by adding the voice data to which the neutral label has been provided in the first label providing step to the teacher data composed of the voice data spoken by the unspecified number of speakers; By constructing an emotion estimator for the specific individual based on the generated teacher data for the specific individual, an emotion estimator generated as teacher data using the voice data spoken by the unspecified number of speakers, Individual adaptation step of personal adaptation as an emotion estimator for estimating the emotional state of the specific individual at the time of speech,
It is characterized by including.

Further, the emotion estimation device according to the second aspect of the present invention includes:
Acquiring means for acquiring voice data spoken by a specific individual;
Feature extraction means for extracting features of the audio data;
Frequency analysis means for classifying the extracted features into a plurality of patterns for each specific extraction section, and analyzing an appearance frequency for each pattern;
A first label assigning unit that assigns a neutral label indicating that the emotional state at the time of speech is a calm state to the audio data of the specific extraction section in which the appearance frequency of the pattern is determined to be equal to or greater than a threshold value;
Generating the teacher data for the specific individual by adding the voice data to which the neutral label is provided by the first label providing unit to the teacher data composed of the voice data spoken by the unspecified number of speakers; By constructing an emotion estimator for the specific individual based on the generated teacher data for the specific individual, an emotion estimator generated as teacher data using the voice data spoken by the unspecified number of speakers, Personal adaptation means for personal adaptation as an emotion estimator for estimating the emotional state of the specific individual when speaking,
It is characterized by having.

Further, the emotion estimation device according to the third aspect of the present invention includes:
Acquiring means for acquiring voice data spoken by the speaker;
Feature extraction means for extracting features of the audio data;
Frequency analysis means for classifying the extracted features into a plurality of patterns for each specific extraction section, and analyzing an appearance frequency for each pattern;
A first label assigning unit that assigns a neutral label indicating that the emotional state at the time of speech is a calm state to the audio data of the specific extraction section in which the appearance frequency of the pattern is determined to be equal to or greater than a threshold value;
An evaluation value obtained by multiplying the number of the specific extraction sections to which the neutral label is added by a weighting coefficient, and an evaluation value obtained by multiplying the number of the specific extraction sections to which the neutral label is not added by the weighting coefficient If the evaluation value of the specific extraction section with the neutral label is higher than the evaluation value of the specific extraction section without the neutral label, the utterance of the speaker is compared. Emotion estimation means for determining the emotional state at the time as neutral,
It is characterized by having.

Further, a program according to a fourth aspect of the present invention includes:
Acquisition means for acquiring voice data spoken by a specific individual using a computer;
Feature extracting means for extracting features of the audio data,
Frequency analysis means for classifying the extracted features into a plurality of patterns for each specific extraction section and analyzing the appearance frequency of each pattern;
A first label assigning unit that assigns a neutral label indicating that the emotional state at the time of speech is a calm state to the audio data of the specific extraction section in which the appearance frequency of the pattern is determined to be equal to or greater than a threshold value;
Generating the teacher data for the specific individual by adding the voice data to which the neutral label is provided by the first label providing unit to the teacher data composed of the voice data spoken by the unspecified number of speakers; By constructing an emotion estimator for the specific individual based on the generated teacher data for the specific individual, an emotion estimator generated as teacher data using the voice data spoken by the unspecified number of speakers, Personal adaptation means for personal adaptation as an emotion estimator for estimating the emotional state of the specific individual when speaking,
It is characterized by functioning as

  According to the present invention, it is possible to improve the estimation accuracy of the emotion estimation device for a specific individual.

1 is a block diagram illustrating a physical configuration of an emotion estimation device according to a first embodiment of the present invention. FIG. 2 is a block diagram illustrating a functional configuration of the emotion estimation device according to the first embodiment of the present invention. FIG. 4 is a diagram for describing a method for creating power time-series data of audio data and pitch time-series data of audio data. FIG. 3 is a diagram illustrating a spectrum distribution of audio data. It is a figure for explaining an extraction section. FIG. 5 is a diagram for describing a method of extracting a power time-series change pattern of audio data. FIG. 7 is a diagram for describing a method of extracting a pitch time-series change pattern of audio data. (A) is a figure for demonstrating the frequency analysis method of a power time series change pattern. (B) is a figure for explaining the frequency analysis method of a pitch time series change pattern. It is a figure for explaining an image of a discrimination threshold of an emotion estimator. FIG. 6 is a diagram for describing an emotion estimation method by an emotion estimation unit. FIG. 4 is a diagram for describing a weight coefficient. It is a flowchart for demonstrating the individual adaptation process for adapting an emotion estimator for a specific individual. It is a flow chart for explaining details of personal adaptation processing of an emotion estimator. It is a flowchart for demonstrating the extraction process of the characteristic of audio | voice data. It is a flowchart for demonstrating an emotion estimation process. FIG. 9 is a diagram for describing a frequency analysis method of the emotion estimation device 100 according to Modification 1. (A) shows a case where a neutral section is set using the average value of the appearance frequency, (b) shows a case where the neutral section is set using the variance of the appearance frequency, and (c) shows a median value of the appearance frequency. Is used to set a neutral section.

  Hereinafter, a personal adaptation method, an emotion estimation device, and a program of an emotion estimator according to an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same or corresponding portions are denoted by the same reference numerals.

(Embodiment 1)
In the present embodiment, a case will be described in which the emotion estimation device 100 is mounted on a pet robot. The emotion estimation device 100 mounted on the pet robot estimates the emotional state of the user by analyzing the features of the user's voice, and the pet robot takes an action corresponding to the estimated emotion. The emotion estimation device 100 mounted on the pet robot adapts the emotion estimator as a specific individual emotion estimator by adding the voice data of the user accumulated daily to the teacher data.

  In the present embodiment, a case will be described in which the emotion estimation device 100 estimates the emotional state of the speaker as one of positive, negative, and neutral emotional states. The positive emotional state is a happy emotional state, an emotional state of comfort, an emotional state of interest, and the like. The negative emotional state includes an angry emotional state, an anxious emotional state, a boring emotional state, and the like. The neutral emotional state is an emotional state other than the positive emotional state and the negative emotional state. In general, it can be assumed that most of the uttered voices are often uttered in a neutral emotional state.

  Hereinafter, the configuration of the emotion estimation device 100, the personal adaptation process for adapting the emotion estimation device 100 as the emotion estimation device of the specific individual user, and the emotion estimation process of the specific individual by the emotion estimation device 100 will be described in detail.

  The emotion estimation device 100 according to the first embodiment physically includes a control unit 1, a storage unit 2, an input / output unit 3, and a bus 4, as illustrated in FIG.

  The control unit 1 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and a CPU (Central Processing Unit). The ROM stores an emotion estimator personal adaptation processing program according to the present embodiment, an emotion estimation program, various initial setting programs, an initial program for loading a hardware inspection program, and the like. The RAM functions as a work area for temporarily storing various software programs executed by the CPU, data necessary for executing the software programs, and the like. The CPU is a central processing unit that executes various processes and calculations by executing various software programs.

  The storage unit 2 includes a nonvolatile memory such as a hard disk drive and a flash memory. The storage unit 2 stores voice data spoken by an unspecified number of speakers as teacher data. The storage unit 2 stores voice data spoken by the user every day.

  The input / output unit 3 includes a voice input device for acquiring voice data spoken by the user. In addition, the input / output unit 3 includes a CD (Compact Disc) drive and a USB (Universal Serial Bus) interface to acquire audio data to be analyzed via a storage medium. In addition, the input / output unit 3 includes a speaker, a display, an LED (Light Emitting Diode), and the like in order to output the emotional state of the user estimated by the emotion estimator. The input / output unit 3 directly outputs the estimated user's emotional state (positive, negative, or neutral) by voice or text, and also generates an utterance associated with the estimated emotional state based on a pre-programmed scenario. The estimated emotional state can also be indirectly output as the content, speech speed, and LED color. In addition, a change setting of a weight coefficient, which will be described later, can be performed from the input / output unit 3.

  The bus 4 connects the control unit 1, the storage unit 2, and the input / output unit 3.

  As shown in FIG. 2, the emotion estimation device 100 functionally includes a voice data acquisition unit 110, a voice data analysis unit 120, a feature extraction unit 130, a frequency analysis unit 150, and a first label provision unit 160. , A second label assigning section 170, an emotion estimator adaptation processing section 180, and an emotion estimating section 190. In addition, the feature extraction unit 130 includes a specific individual classification unit 131, a time length measurement unit 132, a power time series change pattern calculation unit 133, and a pitch time series change pattern calculation unit 134.

  The audio data acquisition unit 110 acquires, via the input / output unit 3, audio data of a user to be analyzed.

  The audio data analysis unit 120 analyzes the acquired audio data. Specifically, the power of the audio data and the pitch of the audio data are created as time-series data. The waveform shown by the dotted line in FIG. 3 is a waveform example of the audio data. The audio data analysis unit 120 sets an analysis window starting from the start point t0 of the audio data, and calculates the time average of the square of the audio waveform amplitude for each analysis window while shifting the analysis window by a shift width dt. Create power time series data.

  Further, the audio data analysis unit 120 sets an analysis window starting from the start point t0 of the audio data, and performs FFT (Fast Fourier Transform) conversion on the audio data in the analysis window while shifting the analysis window by a shift width dt. FIG. 4 is an example showing a spectrum distribution of audio data in each analysis window obtained by the FFT transform. The horizontal axis is frequency and the vertical axis is spectrum intensity. The peak frequency existing in the lowest frequency region in this spectrum is defined as f0. This f0 indicates the fundamental frequency unique to the speaker obtained from the audio data of the analysis window. The voice data analysis unit 120 creates pitch time-series data of voice data by extracting f0 at time tn as f0_n. In creating the power time series data and the pitch time series data, the analysis window width and the shift width dt of the analysis window are set based on the sampling frequency of the audio data.

  The feature extraction unit 130 uses the power time-series data of the extracted voice data and the pitch time-series data of the voice data as the characteristics of the voice data, and performs a power time-series change pattern, a pitch time-series A power time series change pattern calculation unit 133 and a pitch time series change pattern calculation unit 134 for calculating a change pattern are provided. Here, as a specific extraction section, an exhalation paragraph section in which audio data is divided in units of exhalation paragraphs is set as an extraction section. An exhalation paragraph is a unit of an utterance section uttered during a breath, and the reason for extracting features of voice data in units of an exhalation paragraph is that the emotional state of a speaker often changes in the exhalation paragraph. . A specific example of the setting of the exhalation paragraph will be described with reference to FIG. The upper part of FIG. 5 is a waveform example of voice data for analyzing an emotional state at the time of speech. The horizontal axis is time, and the vertical axis is the amplitude of speech. The second row shows an example in which the voice data is divided from the expiration paragraph section 1 to the expiration paragraph section n for each expiration paragraph section.

  In addition, the feature extracting unit 130 includes a specific individual classifying unit 131 so that the frequency analyzing unit 150 described later analyzes the appearance frequency of the power time-series change pattern and the pitch time-series change pattern for each specific individual. In addition, the feature extraction unit 130 includes a time length measurement unit 132, for example, to classify the exhalation paragraph section into three, that is, short, normal, and long, and to perform a frequency analysis described later.

  The specific individual classifying unit 131 classifies the voice data of a plurality of users into voice data for each specific individual by comparing the feature data of the voice data of the specific individual registered in advance. For example, when a father, mother, and child are registered as a user, the specific individual classification unit 131 classifies the audio data for each father, mother, and child. Specifically, the specific individual classifying unit 131 divides the input audio data into fathers, mothers, and children based on the correlation between the feature amounts of the audio data registered in advance and the features of the input audio data. Audio data, mother audio data, child audio data, and others. Then, the specific individual classifying unit 131 attaches a label for discriminating a father, a mother, and a child to the classified voice data, and stores the label in the storage unit 2.

  The time length measuring unit 132 measures the time length of an exhalation paragraph section, which is an extraction section for extracting features of voice data. Specifically, the time length measuring unit 132 measures the time length of the expiration paragraph section by detecting a silent section.

  The power time series change pattern calculation unit 133 extracts a power time series change pattern for each extraction section as a change pattern of the power time series data. This will be specifically described with reference to FIG. FIG. 6 is an example of power time-series data of an arbitrary exhalation paragraph section created by the audio data analysis unit 120. The angle between the line AB connecting the position A where the power is the highest and the starting point B of the waveform and the time axis is defined as θs. Θs in the expiration paragraph section 1 is assumed to be θ1s. Similarly, θs in the expiration paragraph section n is set to θns. Further, an angle between a line AC connecting the position A where the power is greatest and the end point C of the waveform and the time axis is defined as θe. Let θe in expiration paragraph section 1 be θ1e. Similarly, let θe be θne in the expiration paragraph section n. The power time-series change pattern calculation unit 133 calculates θns from θ1s and θne from θ1e.

  The power time-series change pattern calculation unit 133 similarly calculates θ1s to θns and θ1e to θne for all of the stored voice data, and associates the obtained voice data with the voice data divided for each expiration paragraph section to store the data in the storage unit 2. To memorize. These θs and θe are referred to as a power time-series change pattern.

  Returning to FIG. 2, the pitch time series change pattern calculation unit 134 extracts a pitch time series change pattern for each extraction section as a change pattern of the pitch time series data.

  FIG. 7 is a time-series graph of f0 obtained in FIG. 4 for an arbitrary exhalation paragraph section. The horizontal axis is time, and the vertical axis is the frequency of f0 shown in FIG. The time from the point B to f0_n is dt × (n−1). The angle between the line AB connecting the position A of the highest frequency and the starting point B of the waveform and the time axis is defined as θr. Θr in the expiration paragraph section 1 is assumed to be θ1r. Similarly, θr in the expiration paragraph section n is set to θnr. Further, an angle formed between a line AC connecting the position A of the highest frequency and the end point C of the waveform and the time axis is θf. Assume that θf in expiration paragraph section 1 is θ1f. Similarly, let θf in the expiration paragraph section n be θnf. The pitch time-series change pattern calculation unit 134 calculates θnr from θ1r and θnf from θ1f.

  The pitch time-series change pattern calculating unit 134 similarly calculates θ1r to θnr and θ1f to θnf for all of the stored voice data, and associates the obtained voice data with the voice data divided for each exhalation paragraph section to store the data in the storage unit 2. To memorize. These θr and θf are referred to as a pitch time-series change pattern.

  Returning to FIG. 2, the frequency analysis unit 150 classifies the extracted features into a plurality of patterns, and analyzes the appearance frequency of each pattern for each specific individual. More specifically, the frequency analysis unit 150 converts the power time series change pattern calculated by the power time series change pattern calculation unit 133 and the pitch time series change pattern calculated by the pitch time series change pattern calculation unit 134 into audio data. The frequency analysis of the time series change pattern is performed for each specific individual. Then, the frequency analysis unit 150 determines that the time-series change pattern whose appearance frequency is equal to or greater than the threshold is the time-series change pattern of the voice uttered in a normal state, and sets a section to which the time-series change pattern belongs as a neutral section. .

  This will be specifically described with reference to FIG. The frequency analysis unit 150 performs a frequency analysis on θs calculated by the power time-series change pattern calculation unit 133, and creates a frequency analysis graph of the power time-series change pattern as shown in FIG. The horizontal axis is the angle of θs, and is divided from 0 ° to 90 ° every 10 °. The vertical axis represents the appearance frequency of the power time-series change pattern belonging to the corresponding θs section in%. The frequency analysis unit 150 divides the power time-series change pattern θs extracted by the feature extraction unit 130 into sections at intervals of 10 °, counts the number of data belonging to each section, and creates a graph shown in FIG.

  Then, the frequency analysis unit 150 sets the section of θs in which the appearance frequency is equal to or larger than the threshold to the neutral section to which the power time-series change pattern of the voice data uttered in the quiet state (neutral). In the example illustrated in FIG. 8A, the frequency analysis unit 150 sets the threshold to 15%, and sets a section from 20 ° to 50 ° where the appearance frequency of θs is 15% or more as a neutral section.

  Further, the frequency analysis unit 150 performs a frequency analysis on θr calculated by the pitch time-series change pattern calculation unit 134, and creates a frequency analysis graph of the pitch time-series change pattern as shown in FIG. The horizontal axis is the angle of θr, and is divided from 0 ° to 90 ° every 10 °. The vertical axis indicates the frequency of appearance of the pitch time-series change pattern belonging to the corresponding θr section in%. The frequency analysis unit 150 sets the section of θr in which the appearance frequency is equal to or greater than the threshold to the neutral section to which the pitch time-series change pattern of the voice data uttered in a quiet state (neutral). In the example illustrated in FIG. 8B, the frequency analysis unit 150 sets the threshold to 15%, and sets a section from 30 ° to 60 ° where the appearance frequency of θr is 15% or more as a neutral section.

  The frequency analysis unit 150 similarly performs frequency analysis on θe and θf. The frequency analysis unit 150 performs this frequency analysis process for each specific individual classified by the specific individual classification unit 131. Further, the frequency analysis unit 150 converts the audio data into, for example, 2 seconds or less (short), 2 seconds to 4 seconds (normal), and 4 seconds or more (long) according to the time length measured by the time length measurement unit 132. And frequency analysis is performed for each of the classified audio data. The reason why voice data is classified by time length is that if the time length of the expiration paragraph section is significantly different, the tendency of feature changes due to changes in emotional state at the time of speech may be different, so it is better to classify and analyze by time length. This is because the analysis accuracy can be improved.

  Returning to FIG. 2, the first label assigning unit 160 has a quiet emotional state when uttering the speech data of the extraction section in which the appearance frequency of the pattern analyzed for each specific individual is determined to be equal to or higher than the threshold. A neutral label indicating that Specifically, the first label assigning section 160 assigns a neutral label to the audio data belonging to the neutral section set by the frequency analysis section 150 in the personal adaptation processing of the emotion estimator, and associates the neutral label with the corresponding audio data. In the storage unit 2. The first label assigning unit 160 outputs the teacher data that has been assigned to the neutral section based on the new frequency analysis result of the frequency analyzing unit 150 even if the teacher data has been assigned a positive or negative label before the frequency analysis. For, change the label to neutral.

  The second label assigning unit 170 uses the emotion estimator mounted on the emotion estimating device 100 for any of the voice data to which the first label assigning unit 160 does not assign the neutral label, and outputs any one of the positive and the negative. A label is given and linked to the corresponding audio data and stored in the storage unit 2. At the start of using the emotion estimating apparatus 100, positive or negative is determined by using an emotion estimator in an initial state in which voice data spoken by an unspecified number of speakers is generated as teacher data.

  The emotion estimator adaptation processing unit 180 converts the voice data of a specific individual, which has been given a positive or negative or neutral label by the first label giving unit 160 and the second label giving unit 170, into an unspecified number of Generate teacher data for a specific individual added to the teacher data composed of voice data spoken by the speaker, and construct an emotion estimator for the specific individual based on the generated teacher data for the specific individual. Thereby, the emotion estimator generated as the teacher data using the voice data uttered by the unspecified number of speakers is personally adapted as the emotion estimator for estimating the emotional state of the specific individual when speaking.

  Specifically, the emotion estimating apparatus 100 uses the voice data of the specific individual user to which the positive, negative, and neutral labels are assigned as teacher data, and classifies the teacher data into positive, negative, and neutral. Optimize the parameters of the formula that determine the characteristics of FIG. 9 is an image diagram expressing two-dimensionally a discrimination threshold for discriminating three emotional states of positive, negative, and neutral. The emotion estimation apparatus 100 adds voice data of a specific individual user to teacher data composed of voice data uttered by an unspecified number of speakers, and accumulates the data every day. The emotion estimator adaptation processing unit 180 reconstructs the emotion estimator each time the accumulated amount of the teacher data exceeds the preset increase amount.

  Since the teacher data in the initial state is composed of the voice data of an unspecified number of speakers, the characteristics of the voice data of the unspecified number of speakers and the characteristics of the voice data of the specific individual user do not always match. Not necessarily. Therefore, the accuracy of estimating the emotion of the specific individual user by the emotion estimator in the initial state is not always high. However, as the voice data of the specific individual user accumulates daily, the ratio of the voice data of the specific individual user to the teacher data increases. Therefore, the influence of the voice data of the unspecified number of speakers on the neutral labeling gradually decreases. This is because the frequency analysis unit 150 sets a neutral section without using an initial emotion estimator using the voice data of an unspecified number of speakers as teacher data. As described above, the emotion estimation device 100 is changed to an emotion estimator adapted to a specific individual user, and thus the accuracy of emotion estimation is improved as the emotion estimation device 100 for a specific individual.

  The emotion estimation unit 190 estimates the emotional state of the analysis target user at the time of utterance, using an emotion estimator mounted on the emotion estimation device 100. This will be specifically described with reference to FIGS. As shown in FIG. 10, the emotion estimating unit 190 adds the neutral, positive, and negative labels assigned by the first label assigning unit 160 and the second label assigning unit 170 based on each of the power analysis result and the pitch analysis result. It is arranged for each exhalation paragraph section that is an extraction section. As shown in FIG. 11, the numbers of exhalation paragraph sections to which neutral, positive, and negative labels are assigned based on the power analysis result are Npa, Npb, and Npc. Similarly, let Nfa, Nfb, and Nfc be the number of breath paragraph sections to which neutral, positive, and negative labels are assigned based on the pitch analysis result. Emotion estimation section 190 assigns weighting factors Wpa, Wpb, Wpc, Wfa, Wfb, and Wfc to this.

  By setting the weight coefficient, it is possible to adjust the balance between emphasizing the power analysis result and the pitch analysis result. Further, the pseudo personality of the robot can be adjusted by adjusting the positive, negative, and neutral weighting factors. For example, by increasing the neutral weighting factor, it is possible to form an office-like pseudo-personality that does not capture a slight change in the emotional state of the user. In addition, when the negative weighting factor is increased, a pseudo-personality excellent in concern for sensitively extracting the negative characteristics of the user voice can be formed.

  The emotion estimation unit 190 obtains a neutral evaluation point Eneu, a positive evaluation point Epos, and a negative evaluation point Eneg based on Npa, Npb, Npc, Nfa, Nfb, and Nfc, using Expressions 1 to 3. .

Eneu = Npa * Wpa + Nfa * Wfa (Equation 1)
Epos = Npb * Wpb + Nfb * Wfb (Equation 2)
Eneg = Npc * Wpc + Nfc * Wfc (Equation 3)

  The emotion estimation unit 190 estimates the emotion state of the user when speaking the voice data (for example, a sentence) whose analysis state is the highest in the evaluation point.

  The input / output unit 3 outputs the estimated user's emotional state via the input / output unit 3. For example, the input / output unit 3 outputs the estimated emotional state of the user (positive, negative, neutral) from the speaker, and displays it on the display unit. The input / output unit 3 can also indirectly output the utterance content, the utterance speed, and the emotion state estimated as the color of the LED associated with the emotion state based on a scenario programmed in advance.

  Next, a personal adaptation process for reconstructing an emotion estimator mounted on the emotion estimation device 100 having the above configuration for a specific individual will be described with reference to flowcharts shown in FIGS. It is assumed that the teacher data in which positive, negative, and neutral labels are added to the voice data of an unspecified number of speakers is stored in the storage unit 2 in advance. In the present embodiment, since the characteristics of the audio data are extracted in units of the exhalation paragraph, the audio data divided for each exhalation paragraph section is used as the teacher data. Further, it is assumed that the emotion estimator mounted on the emotion estimation device 100 is constructed by using voice data of an unspecified number of speakers as teacher data. It is also assumed that a father, a mother, and a child are registered in advance as specific individuals.

  The flowchart shown in FIG. 12 is started when the user activates the robot and supplies voice data to the emotion estimation device 100 mounted on the robot.

  When the audio data acquisition unit 110 acquires the audio data supplied by the user (Step S11), the audio data analysis unit 120 analyzes the acquired audio data (Step S12). Specifically, the audio data analysis unit 120 creates the power of the audio data and the pitch of the audio data as time-series data from the acquired audio data.

  The specific individual classifying unit 131 classifies the voice data for each specific individual based on the extracted characteristics of the voice data, and stores the voice data in the storage unit 2 (step S13). For example, audio data of a father, audio data of a mother, and audio data of a child are classified and stored in the storage unit 2.

  Next, emotion estimation apparatus 100 determines whether or not the amount of increase in the stored voice data exceeds a predetermined amount (step S14). When the setting of the predetermined amount is increased, the pseudo personality of the robot equipped with the emotion estimating device 100 changes greatly every time the emotion estimator performs the individual adaptation process. Further, when the setting of the predetermined amount is reduced, the pseudo personality of the robot equipped with the emotion estimation device 100 gradually changes. When the increase amount of the stored voice data does not exceed the predetermined threshold (step S14: No), emotion estimation apparatus 100 continues to store the voice data. When the increase amount of the accumulated voice data exceeds the predetermined amount (step S14: Yes), the emotion estimation device 100 performs personal adaptation processing of the mounted emotion estimator (step S15). The personal adaptation process of the emotion estimator will be described with reference to the flowchart shown in FIG.

  When the emotion estimation device 100 starts the personal adaptation process of the emotion estimator, the feature extraction unit 130 describes a section for calculating a power time series change pattern and a pitch time series change pattern of audio data with reference to FIG. As described above, the exhalation paragraph section in which the voice data is divided in the unit of the exhalation paragraph is set. The feature extracting unit 130 performs a process of extracting features of the voice data for all of the set exhalation paragraph sections (step S21). The feature extraction process of audio data will be described with reference to the flowchart shown in FIG.

  When the emotion estimation device 100 starts the feature extraction process of the voice data, first, the time length of the voice data divided into the exhalation paragraph section is measured by using the function of the time length measurement unit 132, and is measured as the voice data. The time length is associated with the time length and stored in the storage unit 2 (step S31).

  Next, as described with reference to FIG. 6, the power time-series change pattern calculation unit 133 calculates a power time-series change pattern for each extraction section (expiration paragraph section) as a power time-series data change pattern ( Step S32). Then, the calculated power time-series change pattern and the time length measured by the time length measuring unit 132 are linked and stored in the storage unit 2.

  Next, as described with reference to FIG. 7, the pitch time-series change pattern calculating unit 134 calculates a pitch time-series change pattern for each extraction section (expiration paragraph section) as a change pattern of the pitch time-series data (FIG. 7). Step S33). Then, the calculated pitch time-series change pattern and the time length measured by the time length measuring unit 132 are linked and stored in the storage unit 2.

  Returning to the flowchart of FIG. 13, when the feature extraction process is completed for all of the audio data (power time series data, pitch time series data) stored in the storage unit 2, the frequency analysis unit 150 determines the extracted features. The pattern is classified into a plurality of patterns, and the appearance frequency of each pattern is analyzed for each specific individual (step S22). Specifically, as described with reference to FIG. 8, the frequency analysis unit 150 performs the frequency analysis of the power time-series change pattern and the frequency analysis of the pitch time-series change pattern for each specific individual. At this time, the frequency analysis unit 150 classifies the time length of the extraction section measured by the time length measurement unit 132 into three types, for example, 2 seconds or less, 2 seconds to 4 seconds, and 4 seconds or more, and performs frequency analysis. . Then, the frequency analysis unit 150 sets a section in which the appearance frequency is equal to or greater than the threshold as a neutral section to which the time-series change pattern of the voice data uttered in the normal state belongs.

  The first label assigning section 160 assigns a neutral label to audio data whose characteristics (power time-series change pattern and pitch time-series change pattern) belong to a section set as a neutral section by the frequency analysis section 150 ( Step S23). Then, the first label assigning unit 160 stores the voice data to which the neutral label has been assigned in the storage unit 2 as the teacher data (step S24).

  Next, the second label assigning section 170 classifies the voice data to which the first label assigning section 160 has not assigned the neutral label as positive or negative, using the mounted emotion estimator (step S25). Then, the second label assigning unit 170 assigns a positive or negative label to the corresponding audio data, and stores the audio data in the storage unit 2 as teacher data (step S26).

  When the positive, negative, and neutral labeling is completed for all of the voice data (power time series data and pitch time series data) stored in the storage unit 2, the emotion estimator adaptation processing unit 180 sets the personality of the mounted emotion estimator. An adaptive process is performed (step S27). More specifically, the emotion estimator adaptation processing unit 180 receives the sound data to which any one of the positive, negative, and neutral labels has been given by the first label giving unit 160 and the second label giving unit 170. We reconstruct the on-board emotion estimator as a specific individual emotion estimator that classifies as positive, negative, or neutral according to the label. If three people, father, mother and child, are registered, three kinds of emotion estimators for father, mother and child are constructed. Thereby, the emotion estimator adaptation processing unit 180 converts the emotion estimator in the initial state in which voice data spoken by an unspecified number of speakers as teacher data into an emotion estimator for estimating an emotion state dedicated to a specific individual. As personal adaptation. When the personal adaptation process (step S27) of the emotion estimator is completed, the process of step S15 in FIG. 12 ends.

  The emotion estimation device 100 continues to store the voice data, and performs personal adaptation processing of the mounted emotion estimator each time the increase amount of the stored voice data exceeds a predetermined amount. This is the end of the description of the personal adaptation process of the emotion estimator performed by the emotion estimation device 100.

  Next, the emotion estimation process performed by the emotion estimation device 100 to estimate the emotional state of the user when speaking from the user's voice data will be described with reference to the flowchart shown in FIG. It is assumed that the individual adaptation process of the emotion estimation device 100 described using the flowcharts shown in FIGS. The flow chart shown in FIG. 15 is started when the user starts the robot equipped with the emotion estimation device 100 and supplies the robot with the user's voice data.

  The description from the process of the user supplying the voice data and obtaining the voice data (step S51) to the process of classifying the voice data for each specific individual (step S53) is described in steps S11 to S13 performed in the individual adaptation process. It is the same as the description.

  When the emotion estimation device 100 classifies the voice data (power time-series data, pitch time-series data) created by the voice data analysis unit 120 for each specific individual, the feature extraction unit 130 sets the voice data (power time-series Data and pitch time series data) (step S54). When extracting the voice data of the specific individual, the feature extracting unit 130 sets an exhalation paragraph section in the voice data (power time series data, pitch time series data). Then, as described with reference to FIGS. 6 and 7, the feature extraction unit 130 extracts the power time series change pattern and the pitch time series change pattern of the audio data as the characteristics of the audio data for all of the exhalation paragraph sections. (Step S55). The details of step S55 are the same as those described with reference to FIG.

  Next, the emotion estimating apparatus 100 classifies the speech data into any of positive, negative, and neutral for each extraction section (expiration paragraph section) by using an onboard emotion estimator, and labels positive, negative, and neutral. And stored in the storage unit 2 (step S56).

  Next, emotion estimation apparatus 100 determines whether or not analysis (emotion estimation) has been completed for all extraction sections (expiration paragraph sections) (step S57). When the analysis of all extraction sections has not been completed (step S57: No), emotion estimation apparatus 100 extracts another extraction section and continues the analysis (step S58). On the other hand, when the analysis has been completed for all the extracted sections (step S57: Yes), the emotion estimation unit 190 specifies the expression using Expressions 1 to 3 as described with reference to FIGS. For the entire voice data (for example, a sentence) spoken by the individual, the emotional state of the specific individual at the time of speech is estimated (step S59).

  Next, emotion estimation apparatus 100 determines whether or not emotion estimation has been completed for the acquired voice data of all persons (step S60). If the emotion estimation of all the persons has not been completed (step S60: No), the emotion estimation apparatus 100 sets the other person as the specific individual, and sets new specific individual voice data (power time series data, pitch time series data). ) Is extracted and the emotion estimation process is continued (step S61). On the other hand, when the emotion estimation has been completed for the voice data of all the persons (Step S60: Yes), the emotion estimation processing of the emotion estimation device 100 ends.

  As described above, the emotion estimation device 100 extracts the features of the audio data, adds a neutral label to the audio data based on the appearance frequency of the extracted features, and adds the neutral labels to the teacher data. As a result, as the voice data of the specific individual user accumulates daily, the ratio of the voice data of the specific individual user to the teacher data increases. The emotion estimator reconstructed based on the teacher data in which the ratio of the voice data of the specific individual user has been increased adapts as an emotion estimator for the specific individual.

  In the labeling method in the related art, labeling is performed by using an emotion estimator constructed as teacher data using voice data of an unspecified number of speakers. Therefore, even if the voice data to which the label is added is added to the teacher data to reconstruct the emotion estimator, it cannot be said that the emotion estimator optimized for the specific individual user is necessarily optimized. Since the emotion estimation device 100 according to the present embodiment assigns a neutral label based on the appearance frequency of the feature pattern of the voice data of the specific individual user accumulated every day, it is easy to adapt as an emotion estimator for the specific individual. . Thus, emotion estimation apparatus 100 can improve estimation accuracy as an emotion estimation apparatus for a specific individual.

  Further, the feature extraction unit 130 extracts a power time series change pattern and a pitch time series change pattern as features of the audio data, and the frequency analysis unit 150 analyzes the frequency of the feature of the audio data based on the time series change pattern. Do. The time-series change pattern of voice data tends to change depending on the emotional state of the speaker. The emotion estimation device 100 determines the neutral (average) emotion state of the speaker by analyzing the frequency of the time-series change pattern. Thereby, emotion estimation apparatus 100 can more accurately estimate the emotional state of the specific individual user when speaking.

  The first label assigning unit 160 determines that the frequency of occurrence of the specific pattern in the power time-series change pattern is determined to be equal to or higher than the threshold, or that the frequency of occurrence of the specific pattern in the pitch time-series change pattern is equal to or higher than the threshold. A neutral label indicating that the emotional state at the time of speech is a calm state is given to the determined voice data. As described above, the emotion estimation device 100 performs labeling on the emotional state of the specific individual user when speaking based on the statistical data of the characteristic pattern of the voice data of the specific individual user, and adds the label to the teacher data. As a result, as the voice data of the specific individual user increases, the emotion estimation device 100 can adapt the mounted emotion estimator as the emotion estimator for the specific individual.

  The extraction section of the change pattern of the feature of the voice data may be an exhalation paragraph, or a section such as a word, an accent phrase, a sentence, or the like. If a specific individual user does not have a habit of speaking a set sentence, it may be possible to construct an emotion estimation device 100 suitable for the user by shortening the extraction section such as a word unit. By setting the extraction section in accordance with the utterance feature of the specific individual user, the estimation accuracy can be improved.

  The emotion estimator adaptation processing unit 180 performs the adaptation processing of the emotion estimator every time the increase amount of the voice data of the specific individual accumulated every day exceeds a preset amount. This makes it possible to reduce the processing amount of emotion estimation apparatus 100 as compared with a method in which adaptive processing is performed each time voice data is acquired.

  The emotion estimation unit 190 obtains a neutral evaluation point Eneu, a positive evaluation point Epos, and a negative evaluation point Eneg using Expressions 1 to 3 using the weighting coefficients. By setting the weight coefficient, it is possible to adjust the balance between emphasizing the power analysis result and the pitch analysis result. Further, the pseudo personality of the robot can be adjusted by adjusting the positive, negative, and neutral weighting factors.

(Modification 1)
In the description of the first embodiment, the frequency analysis unit 150 sets the neutral section based on a preset threshold, but the frequency analysis method need not be limited to this. For example, as shown in FIG. 16A, a section having an average appearance frequency or more may be set as a neutral section. Alternatively, as shown in FIG. 16B, the variance σ in the appearance frequency distribution may be obtained, and the range of the variance σ may be set in the neutral section. As the range of the neutral section is increased from σ to 1.5σ and 2σ, the pseudo personality of the robot equipped with the emotion estimation device 100 is changed to an office-like pseudo personality that does not capture a slight change in the emotional state of the user. be able to. Further, as shown in FIG. 16C, a median in the appearance frequency distribution may be obtained, and a range of a predetermined width X from the median may be set as a neutral section.

(Modification 2)
The process of the emotion estimating unit 190 described with reference to FIG. 10 in the first embodiment is a process of obtaining an evaluation point using Expressions 1 to 3 and estimating whether the sentence as a whole is positive, negative, or neutral. Met. However, the method of the emotion estimation processing need not be limited to this. For example, it is determined whether each of the extraction sections (expiration paragraph section) corresponds to positive, negative, or neutral, and the number of sections corresponding to each of positive, negative, and neutral is compared, and the majority is determined. Is also good. Specifically, only the extraction section in which both the power analysis result and the pitch analysis result are neutral is set to neutral. Then, for the extracted section to which the neutral label has not been added, a positive or negative discrimination is performed using the mounted emotion estimator. Then, the emotion state (positive, negative, or neutral) with the largest number of sections among the number of sections determined as positive, the number of sections determined as negative, and the number of sections determined as neutral is estimated as the emotion state of the speaker. You may do so.

  If a section in which both the power analysis result and the pitch analysis result are neutral is defined as a neutral section, the section determined to be neutral becomes narrower. In this case, the pseudo-personality of the robot equipped with the emotion estimation device 100 can be a pseudo-personality excellent in concern for sensitively picking up changes in the emotional state of the user.

  As another method, a section in which either the power analysis result or the pitch analysis result is neutral may be set as a neutral section. In this case, since the section that is determined to be neutral is widened, the pseudo personality of the robot can be set to a pseudo personality that has a strong office-like image that does not capture any change in the emotional state of the user.

  In the first embodiment, the analysis is performed using the two features of the power time series change pattern and the pitch time series change pattern. However, the analysis is performed using only one of the power time series change pattern and the pitch time series change pattern. It may be. Which analysis should be omitted may be selected in consideration of emotion estimation accuracy, processing speed, manufacturing cost, and the like. Further, the pseudo personality of the robot can be adjusted depending on which analysis item is omitted.

(Modification 3)
In the description of the first embodiment, the teacher data is audio data divided for each exhalation paragraph section, and the first label assigning unit 160 and the second label assigning unit 170 generate audio data divided for the exhalation paragraph section. On the other hand, it was explained that positive, negative and neutral labels were provided. In Modification 3, a case will be described in which teacher data to which positive, negative, and neutral labels are assigned in units of sentences is used.

  The functional operations of the audio data acquisition unit 110, the audio data analysis unit 120, the feature extraction unit 130, and the frequency analysis unit 150 are the same. The audio data analysis unit 120 creates power time series data and pitch time series data from the acquired audio data. The feature extraction unit 130 sets an exhalation paragraph as a feature extraction section in the power time series data and the pitch time series data, and extracts a power time series change pattern and a pitch time series change pattern. Then, the frequency analysis unit 150 performs a frequency analysis of the power time series change pattern and the pitch time series change pattern for each specific individual.

  The first label assigning unit 160 determines whether or not the sentence to be analyzed is uttered in a neutral emotional state. Specifically, as described with reference to FIG. 10 in the first embodiment, a comparison is made with the appearance frequency distribution illustrated in FIG. 8 to determine whether or not each extraction section corresponds to a neutral section. Then, for example, when the number of sections corresponding to the neutral section is 50% or more of the number of extracted sections of the entire sentence, it is determined that the sentence is a sentence uttered in a neutral emotional state. The sentence determined to be neutral is given a neutral label and stored in the storage unit 2.

  The second label assigning unit 170 converts sentence-based speech data to which a neutral label has not been assigned using an emotion estimator generated as teacher data using sentence-based speech data uttered by an unspecified number of speakers. , Positive or negative, and a positive or negative label is given in units of sentences. Specifically, the second label assigning unit 170 uses the emotion estimating unit 190 to determine whether a positive or negative label has been assigned to the sentence unit speech data to which the first label assigning unit 160 has not assigned a neutral label. And associated with the corresponding audio data and stored in the storage unit 2.

  The emotion estimator adaptation processing unit 180 extracts the audio data for each specific individual to which the first label attaching unit 160 and the second label attaching unit 170 are assigned a positive, negative, or neutral label in units of a sentence. In addition to the teacher data composed in units of sentences with voice data spoken by a specific number of speakers, teacher data for a specific individual is generated. Then, by reconstructing the emotion estimator as an emotion estimator for a specific individual based on teacher data generated for each specific individual, voice data spoken by an unspecified number of speakers was generated as teacher data. The emotion estimator is individually adapted as an emotion estimator for estimating the emotional state of a specific individual on a sentence basis.

  Next, the emotion estimation unit 190 estimates the speaker's emotional state in units of sentences using an emotion estimator that estimates the speaker's emotion in units of sentences in the voice data. Therefore, emotion estimation section 190 does not perform the processing described using FIG. 10 and Equations 1 to 3.

  When the supply of voice data having similar content to the emotion estimation device 100 mounted on the robot is repeated every day, it is possible to improve the accuracy of estimating the emotion of the speaker by performing the labeling process in a sentence longer than the exhalation paragraph. There are cases. In some cases, the processing time required for emotion estimation can be reduced.

  In the above description, the power time series change pattern and the pitch time series change pattern have been described as examples of the characteristics of the audio data, but the characteristics of the audio data need not be limited to these. For example, the peak value or average value of the power of the audio in the extraction section, the peak frequency f0 present in the lowest frequency region when the audio data in the extraction section is subjected to the FFT conversion, and the spectrum obtained by performing the FFT conversion on the audio data in the extraction section. May be used. As the time-series change pattern, a power change pattern for each accent phrase, a pitch change pattern for each accent phrase, and the like can be used. As described above, by performing frequency analysis based on characteristics of various voice data, more accurate emotion estimation can be performed.

  In the above description, the voice data analysis unit 120 creates power time series data and pitch time series data, and the feature extraction unit 130 sets an expiration paragraph in the power time series data and pitch time series data, It has been described that the power time-series change pattern and the pitch time-series change pattern are extracted for each power. As a modification of this, audio data is divided into exhalation paragraphs based on silent sections, power time series data and pitch time series data are created from the divided audio data, and power time series data and pitch created for each exhalation paragraph are created. A power time series change pattern and a pitch time series change pattern may be extracted from the time series data.

  Further, the emotion estimator mounted on emotion estimation device 100 may be one estimator that classifies into three values of positive, negative, and neutral as described with reference to FIG. 9, or two estimators. May be used. For example, the first estimator may classify the speech data into neutral and other, and the second estimator may classify the speech data classified as other into positive and negative.

  Further, in the emotion estimation process, the on-board emotion estimator may not be used for neutral classification. Specifically, the first label assigning unit 160 determines whether or not the feature of the voice data of the user to be analyzed extracted by the feature extracting unit 130 belongs to the neutral section analyzed by the frequency analyzing unit 150, If it belongs to a neutral section, a neutral label is given to the corresponding audio data. In the example illustrated in FIG. 8A, the first labeling unit 160 determines that the power time-series change pattern θs calculated by the power time-series change pattern calculation unit 133 is in the range of 20 ° to 50 °. Add a neutral label to the audio data. The same applies to the pitch time series change pattern. The classification of the positive and the negative may use the mounted emotion estimator.

  In the description of the first embodiment, the time length measuring unit 132 is provided in the feature extracting unit 130, and the frequency analyzing unit 150 performs the frequency analysis of the time-series change pattern for each predetermined time length. However, the frequency analysis may be performed without classifying the time length. Whether to omit the classification based on the time length can be determined in consideration of the setting of the extraction section, the estimation accuracy, the processing time, the manufacturing cost, and the like.

  Further, in the description of the frequency analysis using FIG. 8 of the first embodiment, a description has been given in which nine sections are provided as divisions obtained by dividing the frequency division by 10 °, but this division can be set arbitrarily. For example, three sections such as 0 ° to 30 °, 30 ° to 60 °, and 60 ° to 90 ° may be used.

  Further, in the above description, the timing at which the emotion estimator adaptation processing unit 180 reconstructs the emotion estimator is described as each time the amount of increase in the teacher data increases by a predetermined amount. However, the timing of reconstruction of the emotion estimator may be set arbitrarily. For example, it may be set to 2:00 midnight every night, 2:00 midnight every Sunday, 2:00 midnight on the first day of every month, and the like. By setting the reestablishment timing of the emotion estimator at a specific time in this way, it is possible to enjoy that the pseudo personality of the robot equipped with the emotion estimating device 100 adaptively changes at each set timing.

  Further, as described in the first embodiment, since the emotion estimation unit 190 calculates the evaluation value using Expressions 1 to 3, the neutral evaluation value Eneu, the positive evaluation value Epos, and the negative evaluation value I know Eneg. Therefore, based on the ratio of the neutral evaluation value Eneu, the positive evaluation value Epos, and the negative evaluation value Eneg, the emotion estimating apparatus 100 determines a plurality of degrees of emotion states such as “neutral but slightly negative”. Can be used to estimate the emotion.

  In the description of the first embodiment, the emotional state of the speaker is classified into three types, that is, positive, negative, and neutral, but the emotion classification method need not be limited to this. Any method that classifies neutral (ordinary) and other emotional states may be used. For example, it may be classified into five types: happy, angry, sad, easy, and ordinary. In this case, the initial emotion estimator constructed using the voice data of an unspecified number of speakers as teacher data is also equipped with an emotion estimator that can classify the voice data into five types: happy, angry, sad, easy, and ordinary. I do. The mounted emotion estimator uses the speech data to which the first label assigning unit 160 has not assigned the neutral label based on the features of the speaker, such as the strength of the speech, the pitch of the speech, and the time length of the phoneme, based on the feature amounts. The emotional state is classified into one of four types, happy, angry, sad, and easy, and a label such as emotion, sadness, and so on is given.

  In addition, if the amount of initial teacher data uttered by an unspecified number of speakers is increased, it is necessary to input voice data of many specific individuals in order to adapt to specific individuals. On the other hand, if the amount of the initial teacher data is too small, the accuracy of the initial emotion estimation decreases. Therefore, it is preferable to set the amount of initial teacher data uttered by an unspecified number of speakers in consideration of the above.

  In addition to being able to be provided as the emotion estimation device 100 having a configuration for realizing the function according to the present invention in advance, by applying a program, an existing personal computer, an information terminal device, or the like can be converted into an emotion estimation device according to the present invention. It can also function as 100. That is, the present invention is applied by applying a program for realizing each functional configuration of the emotion estimation device 100 exemplified in the above embodiment so that a CPU or the like for controlling an existing personal computer, information terminal device, or the like can execute the program. Can function as the emotion estimation device 100 according to the above. Further, the emotion estimation method according to the present invention can be implemented using the emotion estimation device 100.

  The method of applying such a program is arbitrary. The program can be stored in a computer-readable recording medium (Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD), Magneto Optical disc (MO), etc.), and applied, for example, to the Internet, etc. Alternatively, the program can be stored in a storage on a network, and the program can be applied by downloading the program.

  As described above, the preferred embodiments of the present invention have been described, but the present invention is not limited to the specific embodiments, and the present invention includes the inventions described in the claims and equivalents thereof. It is. Hereinafter, the inventions described in the claims of the present application will be additionally described.

(Appendix 1)
An emotion estimator that estimates the emotional state of a speaker when speaking, which is generated as teacher data using voice data spoken by an unspecified number of speakers, as an emotion estimator that estimates the emotional state of a specific individual when speaking A personal adaptation method of an emotion estimator for personal adaptation,
An obtaining step of obtaining voice data spoken by the specific individual;
A feature extraction step of extracting features of the audio data;
A frequency analysis step of classifying the extracted features into a plurality of patterns for each specific extraction section and analyzing an appearance frequency for each pattern;
A first label assigning step of assigning a neutral label indicating that the emotional state at the time of speech is a calm state to the audio data of the specific extraction section in which the appearance frequency of the pattern is determined to be equal to or greater than a threshold value;
Generating the teacher data for the specific individual by adding the voice data to which the neutral label has been provided in the first label providing step to the teacher data composed of the voice data spoken by the unspecified number of speakers; By constructing an emotion estimator for the specific individual based on the generated teacher data for the specific individual, an emotion estimator generated as teacher data using the voice data spoken by the unspecified number of speakers, Individual adaptation step of personal adaptation as an emotion estimator for estimating the emotional state of the specific individual at the time of speech,
Personal adaptation method of the emotion estimator including.

(Appendix 2)
The emotion estimator generated as the teacher data using the voice data spoken by the unspecified number of speakers is an emotion estimator that estimates the emotional state of the speaker at the time of utterance as either positive or negative, and neutral. ,
The voice data to which the neutral label is not provided in the first label providing step is converted into either positive or negative voice data using an emotion estimator generated using the voice data uttered by the unspecified number of speakers as teacher data. And further comprising a second labeling step of applying a positive or negative label,
In the individual adaptation step, the specific individual voice data to which either the positive label or the negative label is assigned by the first label assigning step and the second label assigning step is converted into the unspecified number of utterances. Generating teacher data for the specific individual added to teacher data composed of voice data spoken by a person, and constructing an emotion estimator for the specific individual based on the generated teacher data for the specific individual. Thereby, the emotion estimator generated as teacher data using the voice data spoken by the unspecified number of speakers is personally adapted as an emotion estimator for estimating the emotional state of the specific individual at the time of speech.
The personal adaptation method of the emotion estimator according to claim 1, characterized in that:

(Appendix 3)
The feature of the audio data extracted in the feature extraction step is a change pattern of power time-series data of the audio data, or a change pattern of pitch time-series data of the audio data,
The personal adaptation method of the emotion estimator according to claim 1 or 2, characterized in that:

(Appendix 4)
Classified in the frequency analysis step, the pattern to analyze the appearance frequency is a change pattern of the power time series data, or a change pattern of the pitch time series data,
The personal adaptation method of the emotion estimator according to claim 3, characterized in that:

(Appendix 5)
In the first labeling step, a specific pattern in the change pattern of the power time-series data is determined for the audio data whose appearance frequency of the specific pattern is determined to be equal to or more than a threshold value, or in a change pattern of the pitch time-series data. For the voice data whose appearance frequency is determined to be equal to or higher than the threshold, a neutral label indicating that the emotional state at the time of speech is a calm state is given.
The personal adaptation method of the emotion estimator according to supplementary note 4, characterized in that:

(Appendix 6)
In the first labeling step, the frequency of occurrence of a specific pattern in the change pattern of the power time-series data is determined to be equal to or more than a predetermined value, and the frequency of occurrence of the specific pattern in the change pattern of the pitch time-series data is determined by a predetermined value. For the voice data determined as described above, a neutral label indicating that the emotional state at the time of speech is a calm state is given,
The personal adaptation method of the emotion estimator according to supplementary note 4, characterized in that:

(Appendix 7)
The specific extraction section is at least one of a breath paragraph, a word, an accent phrase, and a sentence.
7. The personal adaptation method of the emotion estimator according to any one of supplementary notes 1 to 6, characterized in that:

(Appendix 8)
In the individual adaptation step, the emotion estimator is changed to the emotion estimator for the specific individual every time the increase amount of the voice data uttered by the speaker of the specific individual acquired in the acquisition step exceeds a preset amount. As a personal adaptation,
8. The personal adaptation method of the emotion estimator according to any one of supplementary notes 1 to 7, characterized in that:

(Appendix 9)
In the individual adaptation step, at a preset time, personalize the emotion estimator as the specific individual emotion estimator,
8. The personal adaptation method of the emotion estimator according to any one of supplementary notes 1 to 7, characterized in that:

(Appendix 10)
In the obtaining step, obtains voice data spoken by the specific individual,
In the frequency analysis step, the frequency of appearance of the pattern is analyzed for each of the specific individuals,
In the first labeling step, the emotional state at the time of speech is a calm state with respect to the voice data of the specific extraction section in which the appearance frequency of the pattern analyzed for the specific individual is determined to be equal to or higher than a threshold value With a neutral label indicating
In the individual adaptation step, the speech data classified for each specific individual to which a neutral label is assigned in the first label assigning step is converted into teacher data composed of speech data spoken by the unspecified number of speakers. By generating teacher data for each specific individual added to the above, and constructing an emotion estimator for each specific individual based on the generated teacher data for each specific individual, the unspecified number of speakers spoke Emotion estimator generated as speech data teacher data, personal adaptation as an emotion estimator to estimate the emotional state at the time of speech for each specific individual,
10. The personal adaptation method of the emotion estimator according to any one of supplementary notes 1 to 9, characterized in that:

(Appendix 11)
Acquiring means for acquiring voice data spoken by a specific individual;
Feature extraction means for extracting features of the audio data;
Frequency analysis means for classifying the extracted features into a plurality of patterns for each specific extraction section, and analyzing an appearance frequency for each pattern;
A first label assigning unit that assigns a neutral label indicating that the emotional state at the time of speech is a calm state to the audio data of the specific extraction section in which the appearance frequency of the pattern is determined to be equal to or greater than a threshold value;
Generating the teacher data for the specific individual by adding the voice data to which the neutral label is provided by the first label providing unit to the teacher data composed of the voice data spoken by the unspecified number of speakers; By constructing an emotion estimator for the specific individual based on the generated teacher data for the specific individual, an emotion estimator generated as teacher data using the voice data spoken by the unspecified number of speakers, Personal adaptation means for personal adaptation as an emotion estimator for estimating the emotional state of the specific individual when speaking,
An emotion estimation device comprising:

(Appendix 12)
Acquiring means for acquiring voice data spoken by the speaker;
Feature extraction means for extracting features of the audio data;
Frequency analysis means for classifying the extracted features into a plurality of patterns for each specific extraction section, and analyzing an appearance frequency for each pattern;
A first label assigning unit that assigns a neutral label indicating that the emotional state at the time of speech is a calm state to the audio data of the specific extraction section in which the appearance frequency of the pattern is determined to be equal to or greater than a threshold value;
An evaluation value obtained by multiplying the number of the specific extraction sections to which the neutral label is added by a weighting coefficient, and an evaluation value obtained by multiplying the number of the specific extraction sections to which the neutral label is not added by the weighting coefficient If the evaluation value of the specific extraction section with the neutral label is higher than the evaluation value of the specific extraction section without the neutral label, the utterance of the speaker is compared. Emotion estimation means for determining the emotional state at the time as neutral,
An emotion estimation device comprising:

(Appendix 13)
Acquisition means for acquiring voice data spoken by a specific individual using a computer;
Feature extracting means for extracting features of the audio data,
Frequency analysis means for classifying the extracted features into a plurality of patterns for each specific extraction section and analyzing the appearance frequency of each pattern;
A first label assigning unit that assigns a neutral label indicating that the emotional state at the time of speech is a calm state to the audio data of the specific extraction section in which the appearance frequency of the pattern is determined to be equal to or greater than a threshold value;
Generating the teacher data for the specific individual by adding the voice data to which the neutral label is provided by the first label providing unit to the teacher data composed of the voice data spoken by the unspecified number of speakers; By constructing an emotion estimator for the specific individual based on the generated teacher data for the specific individual, an emotion estimator generated as teacher data using the voice data spoken by the unspecified number of speakers, Personal adaptation means for personal adaptation as an emotion estimator for estimating the emotional state of the specific individual when speaking,
Program to function as

DESCRIPTION OF SYMBOLS 1 ... Control part, 2 ... Storage part, 3 ... Input / output part, 4 ... Bus, 100 ... Emotion estimation device, 110 ... Speech data acquisition part, 120 ... Speech data analysis part, 130 ... Feature extraction part, 131 ... Specific individual Classification unit, 132: time length measurement unit, 133: power time series change pattern calculation unit, 134: pitch time series change pattern calculation unit, 150: frequency analysis unit, 160: first label assignment unit, 170: second label assignment Section, 180: emotion estimation unit adaptive processing section, 190: emotion estimation section

Claims (13)

An emotion estimator that estimates the emotional state of a speaker when speaking, which is generated as teacher data using voice data spoken by an unspecified number of speakers, as an emotion estimator that estimates the emotional state of a specific individual when speaking A personal adaptation method of an emotion estimator for personal adaptation,
An obtaining step of obtaining voice data spoken by the specific individual;
A feature extraction step of extracting features of the audio data;
A frequency analysis step of classifying the extracted features into a plurality of patterns for each specific extraction section and analyzing an appearance frequency for each pattern;
A first label assigning step of assigning a neutral label indicating that the emotional state at the time of speech is a calm state to the audio data of the specific extraction section in which the appearance frequency of the pattern is determined to be equal to or greater than a threshold value;
Generating the teacher data for the specific individual by adding the voice data to which the neutral label has been provided in the first label providing step to the teacher data composed of the voice data spoken by the unspecified number of speakers; By constructing an emotion estimator for the specific individual based on the generated teacher data for the specific individual, an emotion estimator generated as teacher data using the voice data spoken by the unspecified number of speakers, Individual adaptation step of personal adaptation as an emotion estimator for estimating the emotional state of the specific individual at the time of speech,
Personal adaptation method of the emotion estimator including. The emotion estimator generated as the teacher data using the voice data spoken by the unspecified number of speakers is an emotion estimator that estimates the emotional state of the speaker at the time of utterance as either positive or negative, and neutral. ,
The voice data to which the neutral label is not provided in the first labeling step is converted into either positive or negative voice data using an emotion estimator generated as voice data spoken by the unspecified number of speakers as teacher data. And further comprising a second labeling step of applying a positive or negative label,
In the individual adaptation step, the specific individual voice data to which either the positive label or the negative label is assigned by the first label assigning step and the second label assigning step is converted into the unspecified number of utterances. Generating teacher data for the specific individual added to teacher data composed of voice data spoken by a person, and constructing an emotion estimator for the specific individual based on the generated teacher data for the specific individual. Thereby, the emotion estimator generated as teacher data using the voice data spoken by the unspecified number of speakers is personally adapted as an emotion estimator for estimating the emotional state of the specific individual at the time of speech.
The personal adaptation method of the emotion estimator according to claim 1, wherein: The feature of the audio data extracted in the feature extraction step is a change pattern of power time-series data of the audio data, or a change pattern of pitch time-series data of the audio data,
The personal adaptation method of the emotion estimator according to claim 1 or 2, wherein: Classified in the frequency analysis step, the pattern to analyze the appearance frequency is a change pattern of the power time series data, or a change pattern of the pitch time series data,
4. The personal adaptation method of the emotion estimator according to claim 3, wherein: In the first labeling step, a specific pattern in the change pattern of the power time-series data is determined for the audio data whose appearance frequency of the specific pattern is determined to be equal to or more than a threshold value, or in a change pattern of the pitch time-series data. For the voice data whose appearance frequency is determined to be equal to or higher than the threshold, a neutral label indicating that the emotional state at the time of speech is a calm state is given.
5. The personal adaptation method of the emotion estimator according to claim 4, wherein: In the first labeling step, the frequency of occurrence of a specific pattern in the change pattern of the power time-series data is determined to be equal to or more than a predetermined value, and the frequency of occurrence of the specific pattern in the change pattern of the pitch time-series data is determined by a predetermined value. For the voice data determined as described above, a neutral label indicating that the emotional state at the time of speech is a calm state is given,
5. The personal adaptation method of the emotion estimator according to claim 4, wherein: The specific extraction section is at least one of a breath paragraph, a word, an accent phrase, and a sentence.
The personal adaptation method of the emotion estimator according to any one of claims 1 to 6, characterized in that: In the individual adaptation step, the emotion estimator is changed to the emotion estimator for the specific individual every time the increase amount of the voice data uttered by the speaker of the specific individual acquired in the acquisition step exceeds a preset amount. As a personal adaptation,
The personal adaptation method of the emotion estimator according to any one of claims 1 to 7, wherein: In the individual adaptation step, at a preset time, personalize the emotion estimator as the specific individual emotion estimator,
The personal adaptation method of the emotion estimator according to any one of claims 1 to 7, wherein: In the obtaining step, obtains voice data spoken by the specific individual,
In the frequency analysis step, the frequency of appearance of the pattern is analyzed for each of the specific individuals,
In the first labeling step, the emotional state at the time of speech is a calm state with respect to the voice data of the specific extraction section in which the appearance frequency of the pattern analyzed for the specific individual is determined to be equal to or higher than a threshold value With a neutral label indicating
In the individual adaptation step, the speech data classified for each specific individual to which a neutral label is assigned in the first label assigning step is converted into teacher data composed of speech data spoken by the unspecified number of speakers. By generating teacher data for each specific individual added to the above, and constructing an emotion estimator for each specific individual based on the generated teacher data for each specific individual, the unspecified number of speakers spoke Emotion estimator generated as speech data teacher data, personal adaptation as an emotion estimator to estimate the emotional state at the time of speech for each specific individual,
The personal adaptation method of the emotion estimator according to any one of claims 1 to 9, wherein: Acquiring means for acquiring voice data spoken by a specific individual;
Feature extraction means for extracting features of the audio data;
Frequency analysis means for classifying the extracted features into a plurality of patterns for each specific extraction section, and analyzing an appearance frequency for each pattern;
A first label assigning unit that assigns a neutral label indicating that the emotional state at the time of speech is a calm state to the audio data of the specific extraction section in which the appearance frequency of the pattern is determined to be equal to or greater than a threshold value;
Generating the teacher data for the specific individual by adding the voice data to which the neutral label is provided by the first label providing unit to the teacher data composed of the voice data spoken by the unspecified number of speakers; By constructing an emotion estimator for the specific individual based on the generated teacher data for the specific individual, an emotion estimator generated as teacher data using the voice data spoken by the unspecified number of speakers, Personal adaptation means for personal adaptation as an emotion estimator for estimating the emotional state of the specific individual when speaking,
An emotion estimation device comprising: Acquiring means for acquiring voice data spoken by the speaker;
Feature extraction means for extracting features of the audio data;
Frequency analysis means for classifying the extracted features into a plurality of patterns for each specific extraction section, and analyzing an appearance frequency for each pattern;
A first label assigning unit that assigns a neutral label indicating that the emotional state at the time of speech is a calm state to the audio data of the specific extraction section in which the appearance frequency of the pattern is determined to be equal to or greater than a threshold value;
An evaluation value obtained by multiplying the number of the specific extraction sections to which the neutral label is added by a weighting coefficient, and an evaluation value obtained by multiplying the number of the specific extraction sections to which the neutral label is not added by the weighting coefficient If the evaluation value of the specific extraction section with the neutral label is higher than the evaluation value of the specific extraction section without the neutral label, the utterance of the speaker is compared. Emotion estimation means for determining the emotional state at the time as neutral,
An emotion estimation device comprising: Acquisition means for acquiring voice data spoken by a specific individual using a computer;
Feature extracting means for extracting features of the audio data,
Frequency analysis means for classifying the extracted features into a plurality of patterns for each specific extraction section and analyzing the appearance frequency of each pattern;
A first label assigning unit that assigns a neutral label indicating that the emotional state at the time of speech is a calm state to the audio data of the specific extraction section in which the appearance frequency of the pattern is determined to be equal to or greater than a threshold value;
Generating the teacher data for the specific individual by adding the voice data to which the neutral label is provided by the first label providing unit to the teacher data composed of the voice data spoken by the unspecified number of speakers; By constructing an emotion estimator for the specific individual based on the generated teacher data for the specific individual, an emotion estimator generated as teacher data using the voice data spoken by the unspecified number of speakers, Personal adaptation means for personal adaptation as an emotion estimator for estimating the emotional state of the specific individual when speaking,
Program to function as

Images