JP2008170820A - Content provision system and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To select an advertisement etc., to be displayed according to a speaker's feeling in order to enhance advertisement effect. <P>SOLUTION: A speaker's voice is acquired (S11), and the feeling of the speaker is analyzed based upon the acquired speech (S12); and the analysis results of the feeling are stored in a database (S13), and screen display is changed based upon the feeling analysis results (S14). For example, when the feeling analysis results show "excitement", advertisement prestored while related to a feeling classification "excitement" is displayed on the screen of a monitor that the speaker is viewing. Further, the screen display is changed based upon a word spotting result (S15). For example, when a word "hot-spring" is extracted from a conversation, advertisement prestored while related to the word "hot-spring" is displayed on the screen of the monitor that the speaker is viewing. Then banner advertisement displayed on the screen is clicked on with a pointing device such as a mouse to access a link-destination Web site of the banner advertisement (S16). <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  In the present invention, when two or more speakers have a conversation using, for example, a personal computer and the Internet, the topic and the emotion of the speaker are determined from the input voice, and contents such as advertisements and comments suitable for the topic and emotion are displayed. The present invention relates to a system and method for displaying on a monitor of a personal computer.

A technique relating to a method and apparatus for detecting emotion from input speech has been proposed (see, for example, Patent Document 1).
Japanese Patent Application No. 2002-293926

In Patent Document 1, as described in paragraph [0032], emotion recognition is performed for a predetermined speaker, but the emotion of an unspecified speaker cannot be determined.
Therefore, the present invention determines a speaker's emotion even when an unspecified speaker speaks with an unspecified utterance content in an unspecified environment, and an appropriate comment or Allows advertisements to be displayed.

  A first feature of the present invention is a content providing system, in which a word and content are stored in association with each other, a word extracting unit that extracts a word from conversation speech, and a word extracted by the word extracting unit. It is provided with a content reading means for reading the content stored in association with the content transmitting means for sending the read content to the content reproduction means.

Content includes, for example, advertisements and comments.
The means for storing includes means for storing information magnetically, electrically, optically or magneto-optically, specifically, hard disk drive (HDD), random access memory (RAM), CD Includes drives, DVD drives, and MO drives.
The word extraction means includes those used in a technique called word spotting.
The content reproduction means refers to, for example, a personal computer or a mobile phone capable of displaying a GIF or JPEG format image or a website described in HTML.

  In addition to the first feature, the second feature of the present invention further includes means for storing a history of the content transmitted by the content transmission unit, and a plurality of contents are stored with priority for one word. The content reading unit refers to the history of the transmitted content from among the plurality of contents stored in association with the word extracted by the word extracting unit, and prioritizes the content transmitted most recently. It is to read the content with higher rank.

  “A plurality of contents are stored with priority for one word” means that, for example, the advertisement of travel company A is stored at the top priority for the word “Onsen”, and travel company B Is stored in the second priority, and the advertisement of the Ginza A store is stored in the first priority, and the advertisement in the Aoyama B store is the second priority in the waveform of the voice “cake”. Say that is remembered in.

“The content reading means refers to the history of the transmitted content from among the plurality of contents stored in association with the word extracted by the word extracting means, and then prioritizes the content that has been transmitted most recently. "Read content with high" means, for example, the word "Onsen"
An advertisement for travel agency A, which has the highest priority among the content associated with the word "Onsen"
After that, the word “cake” is extracted, and the advertisement of the first priority Ginza A store or the second highest Aoyama B store in the content associated with the word “cake” is transmitted,
Thereafter, when the word “Onsen” is extracted again, it means reading the advertisement of travel company B having the second highest priority among the contents associated with the word “Onsen”.

  According to a third aspect of the present invention, there is provided a content providing system, an emotion type content storage unit that stores an emotion type and content in association with each other, and a feature amount calculation unit that calculates a feature amount of a voice input from the voice input unit And an emotion type determining means for determining an emotion type based on the calculated audio feature amount, a content reading means for reading the content stored in association with the determined emotion type, and the read content Transmitting means for sending to the content reproduction means.

  The feature amount calculating means is a means that can calculate an average of voice power, a standard deviation, an average of pitch, a standard deviation, and the like, and substituting these averages and standard deviations into a predetermined formula.

According to a fourth feature of the present invention, in addition to the third feature, a means for storing the emotion type determined by the emotion type determining means at every first predetermined time and a first predetermined time are stored. An emotion type extracting unit that extracts one emotion type from the emotion types every second predetermined time, and the content reading unit is stored in association with the emotion type extracted by the emotion type extracting unit. The content is to be read from the emotion type content storage means.
For example, the emotion type extraction unit stores the emotion type every second, and extracts the emotion having the highest appearance frequency in the latest 5 seconds every 5 seconds.

  In addition to the fourth feature, the fifth feature of the present invention further comprises means for storing a history of content transmitted by the content sending means, and a plurality of contents are given priority for one emotion type. The stored content reading means refers to the history of the transmitted content from among the plurality of contents stored in association with the emotion type determined by the emotion type determining means, and Next, the content with the highest priority is read.

  A sixth feature of the present invention resides in that, in addition to the third to fifth features, the voice feature amount is an average and standard deviation of voice power and an average and standard deviation of voice pitch.

  A seventh feature of the present invention is a content providing system, which stores an advertisement associated with an emotion type, an advertisement associated with a word, and a comment associated with both the emotion type and the word, Based on the input voice, the emotion type determining means for determining the emotion type, the word extracting means for extracting a word from the conversation voice, and the emotion type determined by the emotion type determining means are stored in association with each other. The advertisement stored in association with the word extracted by the word extracting means, and the emotion type determined by the emotion type determining means and the word extracted by the word extracting means An advertisement comment reading means for reading stored comments, and the read advertisement and comment are containerized. It lies in comprising: a content transmission unit for sending to Tsu reproducing means.

  According to the present invention, it is possible to display contents such as advertisements and comments according to words included in a conversation and emotions of speakers.

  The best mode for carrying out the present invention will be described below. The following description is merely an example, and the technical scope of the present invention is not limited to the following description.

[Overall image]
The emotional voice identification tool is used to acquire the emotional data of the speaker and the word in conversation (word spotting). After using the tool, display the PC version website and the mobile version website. Various services can be browsed based on the acquired emotion data and words. Various services such as “voice fortune-telling” and “voice health” can be browsed on the browser based on the acquired emotion data.

[Voice emotion recognition tool]
Displays advertisements associated with voice emotions or words uttered during a conversation and comments associated with the advertisements, clicks on the advertisements, and jumps to the linked website associated with the advertisement Increase probability.

  FIG. 20 shows an example of the screen design of the voice emotion identification tool. The screen 27 shown in the figure includes a banner advertisement 271 associated with the voice emotion, a banner advertisement 272 associated with the word uttered during the conversation, and a comment associated with the voice emotion and the word uttered during the conversation. 273 and a voice emotion barometer 274.

  The banner advertisement 271 is selected based on the emotion of the speaker. For example, the emotion type “excited” and the “advertisement to be displayed when excited” are stored in association with each other, and the emotion type “sad” and the “advertisement to be displayed when sad” are stored in association with each other. deep. When the speaker's emotion is determined to be “excited”, “an advertisement to be displayed when excited” is displayed. When the emotion of the speaker is identified as “sad”, “an advertisement to be displayed when sad” is displayed.

  The banner advertisement 272 is selected based on words uttered during the conversation. For example, the word “hot spring” and “an advertisement to be displayed when pronounced as Onsen” are stored in association with each other, and the word “nabe” is associated with “an advertisement to be displayed when pronounced as pan”. Remember. If it is determined that “onsen” is pronounced during the conversation, “an advertisement to be displayed when it is pronounced onsen” is displayed. If it is determined that “nabe” is pronounced during the conversation, “an advertisement to be displayed when it is pronounced” is displayed.

  The comment 273 is selected based on the voice emotion and the word uttered during the conversation. For example, the emotion type “sad”, the word “hot spring” and the “comment to be displayed when it is determined to be sad and pronounced onsen” are stored in association with each other, and the emotion type “excitement” The word “hot spring” and the “comment to be displayed when it is determined to be excited and pronounced Onsen” are stored in association with each other. And if the speaker's emotion is identified as “sad” and it is determined that “onsen” is pronounced during the conversation, “when it is determined to be sad and pronounced as onsen, it is displayed. Comment "is displayed. Also, if the speaker ’s emotion is identified as “excited” and it is determined that “onsen” was pronounced during the conversation, then “excited and onsen was pronounced” "Comment to be displayed when" is displayed.

  When each of “advertising selected based on voice emotion”, “advertising selected based on word”, or “comment selected based on voice emotion and word” is displayed, a high advertising effect is exhibited. Therefore, it is preferable.

  When “advertising selected based on voice emotion”, “advertising selected based on word”, and “comment selected based on voice emotion and word” are displayed in combination, a higher advertising effect is obtained. All of these are preferably displayed in order to be demonstrated.

[Overall flow]
<When using the emotion voice recognition tool>
FIG. 1 is a flowchart showing the flow of processing when the emotional voice identification tool is used. As shown in FIG. 1, voice is acquired using a microphone or the like in step S11, emotions are analyzed in step S12, emotion analysis results are accumulated in a database in step S13, and a screen is displayed based on the analysis results in step S14. In step S15, the screen display is changed from the word spotting result. In step S16, a banner advertisement or the like is clicked, and the linked website is accessed.

<When viewing a personal site on a PC or mobile phone>
FIG. 2 is a flowchart showing the flow of processing when browsing a personal site on a personal computer or mobile phone. As shown in FIG. 2, in step S21, the user logs in to the website, browses various services based on the data accumulated when using the tool in step S22, and accesses the link destination from the banner advertisement in step S23.

[Example of overall service provision system configuration]
FIG. 3 is a block diagram illustrating an example of the overall configuration of the service providing system. In the example shown in the figure, the conversation of the registered member 21 is taken into the personal computer (PC) 23 via the microphone 22. The member authentication database 31 stores detailed information (member name, password, etc.) of the registered member 21. In response to the request from the emotion voice identification tool of the PC 23, the content distribution server 32 acquires a comment and an advertisement to be displayed determined from the emotion voice result and the member name from the comment server 33 and the advertisement management server 34, and Send comment data and advertisement data to the voice emotion identification tool.

  The comment server 33 transmits to the content distribution server 32 the comment data to be displayed, which is determined from the emotion sound result received from the content distribution server 32 and the member name, and holds the emotion sound data history for each member. The advertisement management server 34 transmits the advertisement data to be displayed determined from the emotion sound result received from the content distribution server 32 and the member name to the content distribution server 32, and holds the advertisement data history for each member.

  The emotion voice database 35 stores the feature amount data 26 transmitted from the emotion voice identification tool of the PC 23 for each member (speaker 21). The personal database 36 stores the emotional voice identification result transmitted from the emotional voice identification tool of the PC 23 and the displayed advertisement number (analysis result data 25) for each member. The analysis result data 25 is sent from the personal database 36 to the site management Web server 37.

  The site management Web server 37 constructs a site environment that can be browsed from the PC 23 and the mobile phone 24. The provided service database 38 acquires the latest emotion identification result from the personal database 36 based on the member name received from the web server 37 for site management, and “voice fortune-telling” and “voice health” from the acquired emotion identification result. Service data such as the above is transmitted to the Web server 37 for site operation.

[Basic concept of emotion recognition]
Prosodic components such as volume and pitch are used as simple features that do not depend on language or speaker. The basic statistic for a predetermined time (for example, the past one second) of the feature amount is used as the speaker's current way of speaking. The degree of each emotion is obtained from the amount of deviation from the steady state of speaking (for example, basic statistics over the past 5 seconds).

[DLL constituting emotion identification software (tool)]
The DLLs that make up the emotion identification software (tool) are EmotionMonitorDLL.dll, WaveIn.dll, Fft_C.dll, and the like. EmotionMonitorDLL.dll calculates voice features and measures emotions from the statistics. The voice feature amount includes voice input waveform data, spectral envelope data, pitch trajectory data, and power trajectory data. WaveIn.dll gets audio from the microphone. WaveIn.dll is a memory prepared by the tool from the voice input device using the multimedia API (Application Program Interface) that is generally used when recording sound in the Windows (registered trademark) operating system. The voice waveform of the length specified in the area (buffer) is updated and stored so that it can be used. Fft_C.dll is an API that can be used for fast Fourier transform, and calculates a frequency spectrum.

[Overall flow of voice emotion analysis from voice acquisition]
FIG. 4 is a flowchart showing a flow of processing from voice acquisition to voice emotion analysis. As shown in FIG. 4, voice is acquired from the microphone in step S41, A / D conversion is performed in step S42, discrete Fourier transform is performed in step S43, voice feature values are calculated in step S44, and emotion is calculated in step S45. Is measured, and emotion is output in step S46. Voices are acquired in steps S41 and S42, and voice emotions are analyzed in steps S43 to S46.

  In A / D conversion (analog / digital conversion), a speech waveform which is an analog signal is sampled and quantized and converted into digital data. For example, the sampling rate is 16 KHz and the resolution is 16 bits.

  In step S44, an audio feature amount is calculated. The voice feature amount is a trajectory (power trajectory) of the entire speech waveform obtained by obtaining power for each buffer (analysis frame) and a trajectory (pitch trajectory) of the entire speech waveform obtained by obtaining pitch for each buffer (analysis frame). ) Power means the sum of squares of each frequency component of the frequency spectrum. The pitch means the pitch of the voice (unit: Hz (Hertz)).

  In step S45, for example, “excitement degree”, “sadness degree”, “excitement degree”, and “degree of chilliness” are measured in real time every 0.128 seconds in 11 steps from 0.0 to 1.0 from the statistics of the voice feature quantity. . Then, the most frequent emotion is determined as the “current emotion” by five measurements. The “statistics of voice feature amount” refers to, for example, the average and standard deviation of the power locus for the last 5 seconds and the average and standard deviation of the power locus and pitch locus for the immediately preceding 1 second. Average means arithmetic mean (arithmetic mean).

[Audio feature calculation flow]
FIG. 5 is a flowchart showing the flow of the voice feature amount calculation process. FIG. 5 explains step S44 of FIG. 4 in detail. As shown in FIG. 5, the sum of squares is calculated in step S51, and the average / standard deviation for 5 seconds and the average / standard deviation for 1 second are calculated in step S52.

  Further, logarithmic transformation is performed in step S54, discrete Fourier transformation is performed in step S55, a peak is detected in step S56, and an average / standard deviation for one second of the pitch is calculated in step S57. Steps S43 and S54 to S56 are referred to as cepstrum analysis.

[Response to unspecified speakers]
The difference between speakers of speech (speaker property) is defined as “statistic of inflection in steady state of speech”. The short-time average power of the input speech is monitored for 5 seconds, and the basic statistic (average and standard deviation) is identified as speaker characteristics. By recognizing emotions based on the identified speaker characteristics, differences in speaker's voice volume are absorbed. The short-time average power corresponds to the sound volume, and the square sum of the Fourier spectrum of 2048 sample analysis frames (16 KHz sampling 0.128 seconds) is calculated.

[Response to unspecified audio environment]
The difference depending on the voice environment is defined as “basic statistic of power of background noise”. When the emotion identification software is activated, the short-time average power of the input voice is automatically monitored for 5 seconds in a state where the user does not speak, and the voice environment is identified with the basic statistics. By recognizing emotion based on the identified voice environment, it is possible to avoid misrecognizing ambient noise as voice inflection due to emotion. The voice environment can be identified manually at any time.

[Response to unspecified utterance contents]
The difference depending on the utterance content is defined as “short-term fluctuation of speech”, and it is assumed that the difference is canceled out for a long time and the difference disappears. Instead of performing emotion recognition for each analysis frame, the basic statistic of the feature amount for the past 1 second is used. The short-term average power (volume) is used as the feature amount. For “sadness” and “excitement”, the pitch (pitch) is used simultaneously for the voiced interval. Emotion recognition is performed by comparison with the basic statistics of the voice environment identified from the past 5 seconds.

[Summary from the viewpoint of processing]
In order to cope with an unspecified voice environment, the environment is monitored for 5 seconds after the emotion identification software is activated (at the time of no speech). Also, in order to deal with unspecified speakers, the voice for the past 5 seconds is monitored retroactively from the time of emotion identification (at the time of utterance). In addition, in order to deal with unspecified utterance content, basic statistics (power average value and power standard deviation for the last 5 seconds) calculated from voice feature values (power, pitch) for the past 1 second from the time of emotion identification The power average value, power standard deviation, pitch average value, pitch standard deviation for the last one second) are compared with basic statistics of the voice environment.

[Flow of voice environment identification]
The trajectory of short-term average power (referred to as “power” as appropriate) for 5 seconds after the emotion identification software is activated is stored. Then, basic statistics (average and standard deviation) are calculated for the stored power trajectory. The basic statistic is referred to as “basic statistic of voice environment”.

[Flow of speaker identification]
Always keep track of the short-term average power over the past 5 seconds. Then, basic statistics (average and standard deviation) are calculated for the stored power trajectory. The basic statistics are used as speaker characteristics.

[Each emotion recognition algorithm]
FIG. 6 is a flowchart showing a flow of excitement level measurement processing. The degree of excitement increases as the loudness (power) of the voice is kept large.
FIG. 8 is a flowchart showing the flow of sadness degree measurement processing. The degree of sadness becomes lower when the loudness of the voice is kept large, and becomes lower when the pitch (pitch) of the voice exceeds the threshold. In other words, if you speak with a low voice, your sadness will increase.
FIG. 9 is a flowchart showing the flow of the exciting degree measurement process. The degree of excitement increases when the loudness of the voice is high and the pitch of the voice is high.
Equation 6 shows an equation for calculating the degree of relaxation. The degree of relaxation increases as silence in the conversation increases.

[Emotion measurement flow]
Below, the flow of the measurement process of each emotion is demonstrated. In addition, each voice feature is represented by the following symbol.
Pitch: f,
1 second pitch average value: μ _1f
Pitch standard deviation for the last 1 second: σ _1f ,
Power average value for 1 second immediately before: μ _1p
Power standard deviation for the last 1 second: σ _1p ,
Average power for the last 5 seconds: μ _5p
Power standard deviation for the last 5 seconds: σ _5p

[Excitation level measurement flow]
FIG. 6 shows the flow of excitement level measurement processing. FIG. 7 shows the relationship between the average power value in the past 1 second and the excitement level. The power average value of the past 5 seconds is regarded as environmental noise, and if the power average value of the past 1 second is higher than that (YES in step S61), the Y path for evaluating the degree of excitement is passed. If it falls below (NO in step S61), the excitement level is set to zero. If YES in step S61, the degree of excitement is calculated by equation 1 (step S62).

[Sadness measurement flow]
FIG. 8 shows the flow of sadness degree measurement processing. If the power exceeds the power average value of the past 5 seconds (YES in step S81), the path of Y for evaluating the degree of sadness is passed. If the pitch is μ _fT (for example, 150 Hz) or less (NO in step S83), only power evaluation is used. If the pitch exceeds μ _fT (YES in step S83), the evaluation from the pitch is multiplied.

That is, if “YES in step S81” and “NO in step S83”, “calculated value of Formula 2” = “degree of sadness” is set (step S82). If “YES in step S81” and “YES in step S83”, “calculated value of equation 2” × “calculated value of equation 3” = “degree of sadness” is set (step S85). Σ _{fT in Equation} 3 is, for example, 100 Hz.

[Exciting measurement flow]
FIG. 9 shows the flow of the excitement degree measurement process. When the pitch average value in the past one second exceeds μ _fT (for example, 100 Hz) (YES in step S91), the product of the calculated value of formula 4 calculated in step S92 and the calculated value of formula 5 calculated in step S93. Is the degree of excitement. When the pitch average value in the past 1 second is less than μ _fT , the exciting degree is set to 0. Σ _{fT in Equation} 4 is, for example, 50 Hz.

[Degree of measurement flow]
The degree of looseness is calculated by Equation 6. The power average value in the past 5 seconds is regarded as environmental noise, and when the power average value in the past 1 second coincides with it, the degree of clogging is maximized, and decreases with an exponential function as the distance from it is increased.

[Emotion judgment]
The degree of 4 emotions in the past 5 frames (corresponding to 0.64 seconds in the case of 16 KHz and frame length 2048) is stored. Next, the number of occurrences is increased by 1 for the emotion having the maximum degree in each frame. Then, the emotion having the maximum cumulative number in the past five frames is determined as the current emotion.

[Emotion measurement monitor TypeA screen]
FIG. 10 shows an example of the emotion measurement monitor screen. The spectrum of speech has a structure in which a fine structure that is periodic fine irregularities is superimposed on an envelope that is a gentle undulation.

<Cepstrum>
The result of converting the power spectrum of the speech waveform to a logarithm and further Fourier transforming is called a cepstrum. The horizontal axis of the cepstrum is called the quefrency axis. Since the dimension of the horizontal axis of the spectrum is the frequency, the horizontal axis dimension of the cepstrum obtained by Fourier transforming this is the time axis. The component corresponding to the envelope appears in the low quefrency portion, and the portion corresponding to the fine structure appears in the high quefrency portion. The former corresponds to the vocal tract characteristics (voice color), and the latter corresponds to the characteristics of the vocal cord sound source (pitch: voice pitch).

  The process of dividing the low and high quefrency parts according to the threshold is called liftering through filtering. The pitch period (voice pitch) can be obtained by extracting a peak from the high quefrency portion by liftering. This series of processing is called cepstrum analysis.

[Emotion measurement monitor TypeB screen]
FIG. 11 shows another example of the emotion measurement monitor screen. As shown in the figure, the statistic of the voice feature amount (power average value, power standard deviation for the last 5 seconds. Power average value, power standard deviation, pitch average value, pitch standard deviation for the previous 1 second.) Is calculated, and the degree of each emotion such as “excitement”, “sad”, “exciting”, “marvel” is calculated based on these statistics, and the emotion is determined.

[Accumulating emotion analysis results in database]
FIG. 12 shows a flow until the analysis result of emotion is accumulated in the database. As shown in the figure, the emotion analysis result and the voice feature amount are acquired in step S131. In step S132, the database is connected via the Internet. In step S133, the feature data 26 is registered in the emotion voice database 35 every second. In step S134, the emotion analysis result data 25 is registered in the personal database 36 every second.

[Change of screen display from emotion analysis results]
FIG. 13 shows a flow from the emotion analysis result to changing the screen display. As shown in the figure, the emotional voice analysis result is transmitted to the content distribution server 32 in step S141. In step S142, the content distribution server 32 acquires the emotion history and the advertisement display history from the personal database 36, and acquires the advertisement data from the advertisement management server 34 based on the values. In step S143, the content distribution server 32 acquires comment data from the comment server 33 based on the advertisement display history acquired in step S142. In step S144, the acquired comment data and advertisement data are displayed on the screen of the PC 23.

  The advertisement display history in steps S142 to S143 is the advertisement display history displayed based on the emotion history, and is stored in the personal database 36. For example, a hot spring banner ad is displayed when the emotional result of “sadness” has been achieved in the past second. Based on the display history, when the emotional result of “sadness” is obtained next time, another advertisement is displayed. The sequence to be displayed will be described later.

  The emotion history in step S143 refers to the history of emotion analysis result data for each user accumulated in the personal database 36 in step S134. For example, it refers to a data history indicating whether the emotion determined in the past one second is “excitement”, “sadness”, “excitement”, or “slowness”. More specifically, “excitement” from 4 seconds to 3 seconds ago, “sadness” from 3 seconds to 2 seconds ago, “excitement” from 2 seconds to 1 second ago, and from 1 second ago Up to now, it is “married”.

[Ad display interval]
It is preferable that the advertisement classified by emotion is changed every 5 to 10 seconds, and the advertisement classified by word is changed at the timing when the registered word is uttered.

[Emotion-specific advertisement and comment display sequence]
FIG. 14 shows each table necessary for displaying emotion-specific advertisements and comments and the flow of processing. As shown in the figure, the A to C tables are stored in the personal database 36. A personal ID and a name are memorize | stored in A table. In the B table, personal ID, emotion history time, and emotion result are stored. In the C table, a personal ID, a display time, an advertisement number, a previous display advertisement, a comment, and a disclosure time are stored. The D table is stored in the advertisement management server 34. The D table stores an advertisement number, advertisement data (binary data), emotion type, priority, and comment number. In the table of the comment server 33, comment numbers and comments are stored.

A process necessary for displaying an emotion-specific advertisement and a comment will be described.
Step S1411: Emotion result data is registered in the B table of the personal database 36 every second. In the example shown in FIG. 14, the emotion result “1” indicates “excitement”, “2” indicates “sad”, “3” indicates “exciting”, and “4” indicates “slow”.

  Step S1412: The most prominent emotional result is extracted from the emotional results in Table B every 5 seconds. In the example illustrated in FIG. 14, the emotion result “1” appears three times and is the most frequent occurrence, so “1” is extracted. When there are a plurality of emotional results having the same appearance frequency, for example, the one with a smaller number is extracted.

  Step S1413: The advertisement having an emotion extracted from the B table as the first element and the second element having the highest value among the immediately preceding display advertisements stored in the C table is acquired. In the example shown in FIG. 14, at this time, the data on the first row (the record “1 and 1” in the “previous display advertisement” column) and the data on the second row (the “display advertisement immediately before” column) are “1,”. 2 ”) is stored, and the data in the third row (the“ previous display advertisement ”column is“ 1,3 ”data) is not stored. Under such circumstances, the maximum value of the second element in the immediately preceding display advertisement having the emotion extracted from the B table as the first element and stored in the C table is “2” in the data in the second row. “2” in “1, 2” in the “immediate display advertisement” column.

  Step S1414: The row data from the D table of the advertisement management server 34 using the emotion type and priority as the extraction conditions from the two values “value acquired in step S1412” and “value obtained by adding 1 to the value acquired in step S1413”. Narrow down. In the example illustrated in FIG. 14, the row data is “emotion identification: 1, priority: 3”. If the priority is at the end, return to 1. For example, in the case of emotion type 1, since the priority order is from 1 to 3, when the priority order becomes 3, next, the row data of priority order 1 is narrowed down.

  Step S1415: Data of each column in the row narrowed down in step S1414 is acquired from the D table.

  Step S1416: A comment is acquired from the comment server 33 based on the comment number (“eje0019” in the example shown in FIG. 14) in the column data acquired in Step S1415 (“Winter is a hot spring in the example shown in FIG. 14). !))

  Step S1417: The advertisement data acquired in step S1415 and the comment data acquired in step S1416 are registered in the C table.

  Step S1418: Each data acquired in step S1415 and step S1416 is transmitted to the tool of the PC 23 via the content distribution server 32.

  Step S1419: Each data transmitted in step S1418 is displayed in the display area of the tool.

[Change of screen display based on word spotting result]
FIG. 15 shows the flow of processing for changing the screen display based on the word spotting result. As shown in the figure, in step S151, the word spotting result is transmitted from the PC 23 to the content distribution server 32.

  In step S152, the content distribution server 32 accesses the comment server 33, and acquires comment data to be displayed based on the word spotting result from the comment server 33.

  In step S153, the content management server 34 accesses the advertisement management server 34, and acquires advertisement data to be displayed based on the word spotting result from the advertisement management server 34.

  In step S154, the comment data acquired in step S152 and the advertisement data acquired in step S153 are transmitted from the content distribution server 32 to the PC 23 and displayed on the screen of the PC 23.

  The “word spotting” in step S152 is a technique that can determine that a specific word is emitted when the speech waveform of a word uttered during a conversation and the speech waveform of a word to be extracted are similar. . For example, when “Onsen I want you!” Is pronounced, it is possible to extract the speech waveform of the “Onsen” portion and determine that the word “Onsen” has been pronounced. And in this embodiment, the banner advertisement regarding a hot spring trip is displayed.

[Word-based advertisement and comment display sequence]
FIG. 16 shows each table necessary for displaying advertisements and comments by word and the flow of processing. As shown in the figure, the E table is stored in the personal database 36. The E table stores a personal ID, display time, advertisement number, immediately preceding display advertisement, comment, disclosure time, and word ID. The F table is stored in the advertisement management server 34. The F table stores an advertisement number, advertisement data (binary data), emotion type, priority, comment number, and word ID. In the table of the comment server 33, comment numbers and comments are stored.

Processing necessary for displaying the word-by-word advertisement and the comment will be described.
Step S1511: Using the word spotting technique, the digital data of the speech waveform during conversation is compared with the “word speech waveform data” in the word speech data, and the corresponding waveform is monitored.

  Step S1512: If there is a corresponding waveform, the word ID of the corresponding waveform is acquired.

  Step S1513: The same row as the word ID acquired in Step S1512 is extracted from the E table of the personal database 36, and the second element of the immediately preceding display advertisement in the extracted row has the highest value (in the example shown in FIG. 16, “1 , 2 "" 2 ").

  Step S1514: Word ID from three values: “value acquired in step S1512”, “value acquired in step S1513 plus 1” and “most prominent emotion value extracted from table B in FIG. 14”. Then, the row data is narrowed down from the F table using the emotion type and priority as extraction conditions (in the example shown in FIG. 16, the row of “emotion type: 1, priority: 3, word ID: 001” is extracted). If the priority is at the end, return to 1.

  Step S1515: Each column data of the row narrowed down in step S1514 is acquired.

  Step S1516: Comment data is acquired from the comment server 33 based on the comment number in the column data acquired in step S1515 (“oke5009” in the example shown in FIG. 16) (in the example shown in FIG. It's exciting with a cake from Roppongi C! ”).

  Step S1517: The advertisement data acquired in step S1515 and the comment data acquired in step S1516 are registered in the E table of the personal database 36.

  Step S1518: Each data acquired in step S1515 and step S1516 is transmitted to the tool of the PC 23 via the content distribution server 32.

  Step S1519: The data transmitted to the PC 23 via the content distribution server 32 in step S1518 is displayed in the display area of the tool.

[Login to website]
FIG. 17 shows the flow of processing until login to the Web site. In step S211, a voice emotion identification tool or Internet browsing software (IE or the like) is activated. Step S212 is a case where the voice emotion identification tool is activated, and the link button displayed in the display area of the voice emotion identification tool is pressed. Thereafter, as an operation in the system, automatic authentication is performed in the member authentication database 31 using “member name” and “password” stored in the tool. Step S213 is a case where the Internet browsing software is started, and the login is executed by inputting “member name” and “password” in the member login area displayed on the screen of the PC 23 by the Internet browsing software. In step S214, authentication is completed and login to the Web site (personal site) is completed.

[Browse various services based on data results accumulated when using the tool]
FIG. 18 shows the flow of processing for browsing various services based on the data results accumulated when using the tool. As shown in the figure, in step S221, the personal database 36 is accessed from the web server 37 for site operation, and the emotion voice analysis result of the login user is acquired. In step S222, an advertisement to be displayed is acquired from the advertisement management server 34 from the acquired analysis result. In step S223, various service contents based on the acquired analysis result are acquired from the provided service database 38. In step S224, the acquired data is embedded in the html file and transmitted to the member PC 23 or the mobile phone 24.

  In step S222, the most prominent emotion during voice acquisition using the tool is extracted from the personal database 36, and an advertisement is displayed in accordance with the emotion. For example, if the conversation is sad, an advertisement such as “Hawaii trip” is displayed. The “process for judging the most prominent emotion during the conversation” will be described later.

  In step S223, emotions that are most prominent during voice acquisition using the tool are extracted from the personal database, and services are displayed in accordance with the emotions. The displayed services are, for example, voice fortune telling (display fortune telling based on emotion history), voice health check (display health level from emotion history), recommended life (display recommended lifestyle from emotion history) Etc.).

[Process to judge emotions that were most noticeable during conversation]
The most numerical value is extracted from the statistics of the first element of the immediately preceding display advertisement, and “the most prominent emotion” is derived from the value. If the order is the same, priority is given to the smaller number.

  FIG. 19 shows a second example of the table C stored in the personal database 36, and the “process for judging the emotion most noticeable during the conversation” will be described with reference to FIG. The “first element of the immediately preceding display advertisement” refers to the first element of each data in the “immediately displayed advertisement” column in the example of FIG. For example, “1” in “1, 2” in the “advance display advertisement” column is referred to. In the example of FIG. 19, since “1” is “6” and “2” is 3 in the “first element of the immediately preceding display advertisement”, “the most numerical value from the statistics of the first element of the immediately preceding display advertisement” is “ 1 ”. The emotion corresponding to this (for example, “excitement”) becomes “the most prominent emotion”. If the “first element of the immediately preceding display advertisement” is the same number of “1” and “2”, the smaller numerical value “1” becomes “the most prominent emotion”.

It is a flowchart which shows the flow of a process at the time of emotion voice identification tool utilization. It is a flowchart which shows the flow of a process at the time of personal site browsing with a personal computer or a mobile telephone. It is a block diagram which shows an example of the whole structure of a service provision system. It is a flowchart which shows the flow of a process from an audio | voice acquisition to an audio | voice emotion analysis. It is a flowchart which shows the flow of an audio | voice feature-value calculation process. It is a flowchart which shows the flow of an excitement degree measurement process. It is a graph which shows the relationship between the power average value of the past 1 second, and excitement degree. It is a flowchart which shows the flow of a sadness degree measurement process. It is a flowchart which shows the flow of an exciting degree measurement process. It is a figure which shows an example of an emotion measurement monitor screen. It is a figure which shows the other example of an emotion measurement monitor screen. It is a figure which shows the flow until it accumulate | stores the analysis result of an emotion in a database. It is a figure which shows the flow until it changes a screen display from an emotion analysis result. It is a figure which shows each table required in order to display the advertisement according to emotion, and a comment, and the flow of a process. It is a figure which shows the flow of the process which changes a screen display based on a word spotting result. It is a figure which shows each table required in order to display the advertisement according to word, and a comment, and the flow of a process. It is a figure which shows the flow of a process until it logs in to a Web site. It is a figure which shows the flow of the process which browses the various services based on the data result accumulated at the time of tool utilization. It is a figure which shows the 2nd example of the table C memorize | stored in the personal database. It is a figure which shows an example of the screen design of an audio | voice emotion identification tool.

Explanation of symbols

22 ... Microphone 23 ... Personal computer 25 ... Analysis result data 26 ... Feature data 31 ... Member authentication database 32 ... Content distribution server 33 ... Comment server 34 ... Advertisement management server 35 ... Emotion voice database 36 ... Personal database 37 ... For site management Web server 38 ... provided service database

Claims (9)

Means for associating and storing words and content;
Word extraction means for extracting words from conversational speech;
Content reading means for reading the content stored in association with the word extracted by the word extracting means;
A content providing system comprising: content transmitting means for sending the read content to content reproducing means. Means for storing a history of contents transmitted by the contents transmitting means;
A plurality of contents are stored with priority for one word,
The content reading unit refers to a history of the transmitted content from among the plurality of contents stored in association with the word extracted by the word extracting unit, and next to the content transmitted most recently. The content providing system according to claim 1, wherein content having a high priority is read. Emotion type content storage means for storing the emotion type and the content in association with each other;
Feature quantity calculating means for calculating the feature quantity of the voice input from the voice input means;
Emotion type determination means for determining an emotion type based on the calculated audio feature amount;
Content reading means for reading the content stored in association with the determined emotion type;
A content providing system comprising: transmission means for sending the read content to content reproduction means. Means for storing the emotion type determined by the emotion type determination means for each first predetermined time;
Emotion type extraction means for extracting one emotion type for every second predetermined time from the emotion types stored for each first predetermined time;
The content providing system according to claim 1, wherein the content reading unit reads content stored in association with the emotion type extracted by the emotion type extraction unit from the emotion type content storage unit. Means for storing a history of contents transmitted by the contents transmitting means;
A plurality of the contents are stored with priority for one emotion type,
The content reading unit refers to a history of the transmitted content and is transmitted most recently from the plurality of contents stored in association with the emotion type determined by the emotion type determination unit. The content providing system according to claim 4, wherein content having the next highest priority is read from the content.   The content distribution system according to any one of claims 3 to 5, wherein the audio feature amount is an average and standard deviation of audio power and an average and standard deviation of audio pitch. Means for respectively storing advertisements associated with emotion types, advertisements associated with words, and comments associated with both emotion types and words;
Emotion type determination means for determining an emotion type based on the input voice;
Word extraction means for extracting words from conversational speech;
The advertisement stored in association with the emotion type determined by the emotion type determination unit is read, the advertisement stored in association with the word extracted by the word extraction unit is read, and the determination by the emotion type determination unit An advertisement comment reading means for reading a comment stored in association with the extracted emotion type and the word extracted by the word extraction means;
A content providing system comprising: content transmitting means for sending the read advertisement and comment to content reproducing means. Pre-store the audio waveform and content in association with each other,
Compare the stored speech waveform with the speech waveform input from the speech input means, determine whether they are similar,
Reading the content stored in association with the speech waveform determined to be similar to the input speech waveform;
A content providing method for sending the read content to a content reproduction means. Associating emotion types with content in advance,
Calculate the feature value of the voice input from the voice input means,
Based on the calculated voice feature, determine the emotion type,
Read the content stored in association with the determined emotion type,
A content providing method for sending read content to a content reproduction means.