JP2017009825A - Conversation state analyzing device and conversation state analyzing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To analyze relationship of utterance by a plurality of speakers.SOLUTION: The present invention relates to a conversation state analyzing device which analyzes a state of a conversation by a plurality of speakers, the conversation state analyzing device comprising: acquisition means of acquiring conversation speeches by the plurality of speakers; separation means of separating the conversation speeches into a plurality of utterances by the speakers and utterance sections; recognition means of recognizing utterance contents using speech recognition processing for each of the plurality of utterances; and analyzing means of analyzing relationship among the utterances and speakers based upon the utterance contents, the analyzing means specifying, as a series utterance group, utterances estimated to belong to a conversation of the same theme based upon the contents by the utterances.SELECTED DRAWING: Figure 9

Description

  The present invention relates to a technique for analyzing a situation of conversation between a plurality of speakers.

  In recent years, research and development of techniques for performing various interventions such as proposals and assistance from humans to computers have been underway. In order for a computer to appropriately intervene in a situation where a plurality of people are talking, it is necessary to grasp the current situation based on conversational speech by a plurality of speakers.

  Patent Document 1 discloses a technique for analyzing and comparing voice characteristics of a plurality of speakers. In Patent Document 1, the first and second person classification information is obtained from the voice characteristics of the first and second persons, and the compatibility of the first and second persons is determined based on the combination of the two classification information. Analyzing.

  Patent Documents 2 and 3 disclose techniques for specifying a section in which a speaker represents a specific emotion (dissatisfaction, satisfaction, apology, etc.) from telephone conversation. In Patent Documents 2 and 3, speaker emotions in each section are individually detected, and a section representing a specific emotion is specified according to the emotion change pattern between the speakers.

Japanese Patent No. 3280825 International Publication No. 2014/069076 International Publication No. 2014/069120

  The method of Patent Document 1 analyzes the compatibility of speakers based on the voice characteristics of a plurality of speakers, but does not analyze the situation of conversation. In addition, the methods of Patent Documents 2 and 3 only specify a section where the speaker shows a specific emotion, and do not specify the relationship of each utterance in the conversation. Neither method can analyze the relationship between each utterance in a conversation or the entire utterance.

  In view of the above problems, an object of the present invention is to provide a technique capable of analyzing the relationship between utterances by a plurality of speakers.

  In order to achieve the above object, a first aspect of the present invention is a conversation situation analyzing apparatus for analyzing a conversation situation by a plurality of speakers, and acquiring means for acquiring conversation voices by a plurality of speakers, Separation means for separating the conversational voice into a plurality of utterances for each speaker and each utterance section; a recognition means for recognizing the utterance content using speech recognition processing for each of the plurality of utterances; and Analyzing the relationship between utterances, estimating the conversation theme of each utterance based on the content of each utterance, and identifying the utterances estimated as the same conversation theme as a series of utterances Analyzing means.

  In this way, utterance groups can be specified for the same conversation theme. Moreover, even when a plurality of speakers are divided into different groups and are talking about different themes, a series of utterance groups can be appropriately specified.

  Here, the content of the utterance means the text of the utterance. Therefore, the analysis means estimates the identity of the conversation theme of each utterance from the utterance text. Note that a series of utterance groups need not be determined based on the content of utterances alone, but may be performed based on other information such as the timing of utterances. For example, when the conversation theme cannot be estimated only from the content of the utterance, it may be the same as the theme of the conversation to which the utterance immediately before or immediately after the utterance belongs or the conversation theme spoken immediately by the speaker of the utterance.

  In the present invention, the recognizing unit recognizes the utterance content by comparing the utterance text obtained by voice recognition processing with a dictionary, and the analyzing unit recognizes the utterance content text and the dictionary recognized by the recognition unit. , The utterance intention and topic can be obtained, and the conversation theme of the utterance can be estimated based on the utterance intention and topic. Examples of utterance intentions include topic extraction, proposals, approval / disapproval of proposals, and aggregation of opinions. The topic of the utterance includes the genre of the utterance and the location or thing that is the topic. Examples of utterance genres include eating and drinking, travel, music, and weather. Examples of places and things that are talked about include place names, landmarks, store / facility names, and the like. Thus, the conversation theme can be estimated more appropriately by considering the intention and topic of the utterance based on the utterance content (text).

  The conversation state analyzing apparatus according to the present invention further includes a feature amount calculating unit that calculates a speech feature amount for each of the plurality of utterances, and the analysis unit is configured to determine each speaker based on a change in the speech feature amount. It is also preferable to estimate a speaker's emotion at the time of utterance and to estimate the intention of the utterance in consideration of the emotion. By considering the emotion of the speaker, the intention of the utterance can be estimated more accurately or in detail. For example, even if the content of the utterance is in favor of the proposal, if the emotion represents dissatisfaction or irritation, it can be estimated that the intention of the utterance is unwilling to agree.

  In the present invention, the analysis means acquires correspondence between utterances or relation between speakers in the series of utterance groups based on the intention of the utterance, the feature amount of the utterance, the emotion of the speaker at the time of utterance, and the like. It is also preferable. The correspondence between utterances and the relationship between speakers is, for example, which utterance of a certain speaker is a response to which utterance of which speaker, and how a certain speaker is connected to which speaker in the conversation. It indicates whether or not Since the intention of each utterance is obtained as described above, the correspondence between utterances and the relationship between speakers on the conversation can be obtained with high accuracy. Note that the correspondence between utterances and the correspondence between speakers do not need to be determined based solely on the intention of the utterance, but based on other information such as the topic of the utterance, the timing of the utterance, and changes in the feature amount of each utterance. May be determined. For example, when the correspondence between utterances is not sure from the intention of the utterance, or when the intention of the utterance cannot be acquired, the utterance may be associated with the utterance immediately before or after the same conversation. Also, for example, immediately after an utterance intended for a speaker's proposal, if there is a short utterance of another speaker whose intent cannot be extracted, the feature amount of the utterance is analyzed and the utterance is compared with the immediately preceding utterance. It can be associated with an utterance that shows a mutual opinion (consent) or grief (denial). In this way, correspondence between utterances (such as what utterances are connected to which utterances), and relationships between speakers in a conversation (what utterances are between certain speakers) It is possible to obtain the degree of occurrence and how the relationship between the speakers and the intimacy is estimated.

The conversation state analysis apparatus according to the present invention further includes an imaging unit that images a speaker, and the analysis unit takes into account the direction of the body, face, or line of sight of the speaker in the image captured by the imaging unit, It is also preferable to acquire a correspondence relationship between utterances in the series of utterance groups. In conversation, speaking with the body, face, or line of sight toward the person you are talking to, the direction of the speaker's body, face, or line of sight is acquired from the image as described above, so that the correspondence between utterances can be improved. Can be obtained accurately.

  In addition, the conversation state analysis apparatus according to the present invention further includes an imaging unit that captures a speaker, and the analysis unit changes the facial feature amount calculated from the face image of the speaker in the image captured by the imaging unit. It is also preferable to estimate the emotion of the speaker according to the situation and analyze the relationship between the utterances in consideration of the emotion. Since the speaker's emotions also appear in the facial expression, a more accurate analysis can be performed by photographing the speaker's face image, estimating the emotions, and analyzing the relationship of the utterances using the estimated emotions.

  Further, in the present invention, the analysis means obtains a relationship between speakers based on the relationship between utterances and at least one of the content of the utterance, the feature amount of the utterance, and the emotion of the speaker at the time of utterance. Is also preferable. The relationship between speakers includes, for example, intimacy between speakers, a hierarchical relationship, a parent-child relationship, and the like. Analyzing means includes the utterance content (eg, politeness and intimacy required by wording), utterance features (number of utterances, time, overlap) in utterances between speakers (related utterances), The relationship as described above can be obtained from emotion.

  Moreover, it is preferable that the conversation situation analysis apparatus according to the present invention further includes output means for outputting conversation situation data that is data relating to the series of utterance groups. The conversation status data includes, for example, the speakers of each utterance, the correspondence between utterances, the meaning and intention of each utterance, the emotion of the speaker at each utterance, the utterance frequency of each speaker in the utterance group, and the voice characteristics of each utterance Including at least one of a quantity and a relationship between speakers.

  The second aspect of the present invention is a support apparatus that intervenes and supports a conversation by a plurality of speakers. The support device according to this aspect includes a state of a group of a plurality of speakers participating in a series of utterance groups based on the above-described conversation state analysis device and conversation state data output from the conversation state analysis device. And a group state determining unit for determining the content of the intervention in the conversation based on the state of the group and intervening in the conversation. The group status includes the group type, the relationship between speakers in the group, and the group status change. In this way, more accurate support can be achieved by determining the group status based on the results of an accurate analysis of the conversations between speakers and the relationship between the speakers, and performing intervention according to the determined group status. Yes. The intervention in the conversation may be performed by any method such as voice output, character output, and image output, and the mode is not particularly limited.

  Note that the present invention can be understood as a conversation state analysis apparatus or a support apparatus including at least a part of the above means. The present invention can also be understood as a conversation situation analysis method or support method for executing at least a part of the processing performed by the above means. The present invention can also be understood as a computer program for causing a computer to execute these methods, or a computer-readable storage medium in which this computer program is stored non-temporarily. Each of the above means and processes can be combined with each other as much as possible to constitute the present invention.

  According to the present invention, the relationship of utterances by a plurality of speakers can be analyzed.

It is a figure which shows the structural example of the conversation intervention assistance system which concerns on 1st Embodiment. It is a figure which shows the example of a functional block of the conversation intervention assistance system which concerns on 1st Embodiment. It is a flowchart which shows the example of the flow of the whole process of the conversation intervention assistance method which the conversation intervention assistance system which concerns on 1st Embodiment performs. It is a flowchart which shows the example of the flow of the conversation situation analysis process (S303) in a conversation intervention support method. It is a figure which shows the example of the speech isolate | separated for every speaker and every speech area. It is a figure which shows the example of the genre, the place of a topic, and the intent extracted for every utterance. It is a figure which shows the example of a series of utterance groups which have the same conversation theme. It is a figure which shows the example of conversation status data. (A) Correspondence between utterances included in conversation status data, conversation theme of each utterance, intention of utterance, emotion of speaker, and (B) relationship between utterance occurrence between speakers in conversation and speaker It is a figure explaining the example of. It is a flowchart which shows the example of the flow of the group state determination process (S304) in a conversation intervention assistance method. It is a figure which shows the example of the estimation conditions of (A) group classification and (B) group classification. It is a flowchart which shows the example of the flow of the intervention content determination process (S305) in a conversation intervention support method. It is a figure explaining the example of the intervention method according to the (A) intervention policy according to group classification, and the (B) group state change.

(First embodiment)
<System configuration>
The present embodiment is a conversation intervention support system that intervenes with respect to conversations of a plurality of persons in a vehicle to provide information and support decision making. This embodiment is configured so that appropriate intervention can be performed for conversations of a plurality of persons, particularly three or more persons.

  FIG. 1 is a diagram showing an example of the configuration of a conversation intervention support system according to the present embodiment. The passenger's conversation voice acquired by the navigation device 111 via the microphone is sent to the server device 120 via the communication device 114. The server device 120 analyzes the conversation voice transmitted from the vehicle 110, and performs intervention such as provision of appropriate information and decision support according to the situation. The server device 120 analyzes the conversational voice and determines what kind of policy is used for intervention, and acquires information according to the policy from the recommendation system 121, the store advertisement information DB 122, and the related information WEB site 130. The server device 120 transmits an intervention instruction to the vehicle 110, and the vehicle 110 performs voice reproduction or text or image display through the speaker or display of the navigation device 111. The vehicle 110 also includes a GPS device 112 that acquires the current position and a camera 113 that captures the face and body of the occupant (speaker).

  FIG. 2 is a functional block diagram of the conversation intervention support system according to the present embodiment. The conversation intervention support system includes a microphone (speech input unit) 201, a noise removal unit 202, a sound source separation unit (speaker separation unit) 203, a conversation state analysis unit 204, a speech recognition corpus / dictionary 205, a vocabulary intention understanding corpus / Dictionary 206, group state determination unit 207, group model definition storage unit 208, intervention / arbitration unit 209, intervention policy definition storage unit 210, related information DB 211, output control unit 212, speaker (voice output unit) 213, display (image display) Part) 214. Details of processing performed by each of these functional units will be described below together with flowcharts.

In this embodiment, among the functions shown in FIG. 2, voice input by the microphone 201 and intervention content output by the output control unit 212, the speaker 213, and the display 214 are performed by the vehicle 110. Other functions are configured to be performed by the server device 120. However, how these functions are shared between the vehicle 110 and the server device 120 is not particularly limited. For example, the vehicle 110 may perform noise removal, sound source separation, and the like, and may further perform voice recognition processing. In addition, the server device 120 may determine the intervention policy, and the vehicle 110 may determine what information is to be presented according to the determined intervention policy. Furthermore, all functions may be realized in the vehicle 110.

  Note that each of the navigation device 111 and the server device 120 is a computer provided with an arithmetic device such as a CPU, a storage device such as a RAM or a ROM, an input device, an output device, a communication interface, etc., and a program stored in the storage device By executing the above, the above functions are realized. However, part or all of the above functions may be realized by dedicated hardware. The server device 120 does not have to be a single device, and may be configured by a plurality of devices (computers) coupled via a communication line and share the functions among the devices.

<Overall processing>
FIG. 3 is a flowchart showing the overall flow of the conversation intervention support method performed by the conversation intervention support system according to this embodiment. The whole conversation intervention support method will be described with reference to FIG.

  In step S <b> 301, the navigation device 111 acquires conversational voices by a plurality of passengers in the vehicle 110 via the microphone 201. In the present embodiment, subsequent processing for the acquired voice is performed in the server device 120, so the navigation device 111 transmits the acquired conversation voice to the server device 120 via the communication device 114. The number and arrangement of microphones to be used are not particularly limited, but it is preferable to use a plurality of microphones or microphone arrays.

  In step S <b> 302, the server device 120 uses the noise removing unit 202 and the sound source separation unit 203 to extract each utterance for each speaker from the conversational voice. Note that “speech” means that a language is generated as speech and the resulting speech is generated. The processing here includes noise removal by the noise removal unit 202 and sound source separation (speaker separation) by the sound source separation unit 203. For example, the noise removing unit 202 identifies and removes noise from a difference between a voice obtained from a microphone arranged near a noise generation source and a voice obtained from another microphone. The noise removal unit 202 also removes noise by using the correlation of utterances input to a plurality of microphones. The sound source separation unit 203 identifies the speaker by detecting the direction and distance of each speaker from the time difference in which sound is input to the plurality of microphones.

  In step S303, the conversation situation analysis unit 204 analyzes the conversation situation of a plurality of people. In order to analyze the situation of conversations of multiple people, especially 3 or more, it recognizes whether the utterances by each speaker are correlated, and if there is a correlation, what kind of relationship is recognized There is a need. Therefore, the conversation situation analysis unit 204 extracts utterance groups related to the same conversation theme as a series of utterance groups, further grasps the relations between utterances in the utterance groups, and considers the relations between utterances. And analyze the conversation situation and the relationship between the speakers. Specific processing contents by the conversation state analysis unit 204 will be described later.

In step S304, based on the conversation status data from the conversation status analysis unit 204, the group status determination unit 207 determines what type of speaker group is participating in the same conversation, and how this group is. It is determined whether it is in a proper state. Examples of groups include, for example, “a group that has a close relationship and high intimacy, and members can speak to each other”, “an up-and-down relationship but high intimacy and a specific member Such as “a group that leads the decision” and “a group that has a hierarchical relationship, low intimacy, and a specific member leading the group's decision”. In addition, examples of group status changes include a decrease in the utterance frequency of a specific member, a decrease in the utterance frequency of the entire group, a change in the emotion of a specific member, and a change in the leader of the group. . Specific processing contents by the group state determination unit 207 will be described later.

  In step S305, the intervention / arbitration unit 209 determines an intervention policy according to the group status by the group status determination unit 207, and determines the specific intervention timing and content according to the intervention policy and the content of the current conversation. For example, if the group has a flat relationship and high intimacy, and members can freely share their opinions with each other, an intervention policy that encourages lively discussion by presenting detailed reference information almost equally to all members. It is possible to adopt. In addition, for example, when the utterance frequency of a specific speaker or the entire group decreases, it may be possible to adopt an intervention policy that leads to a topic that activates conversation. When the intervention / mediation unit 209 determines the intervention policy, the intervention / arbitration unit 209 obtains information to be presented from the recommendation system 121, the store advertisement information DB 122, or the related information WEB site 130 according to the current topic, and issues an intervention instruction. Specific processing contents by the intervention / arbitration unit 209 will be described later.

  In step S <b> 306, the output control unit 212 generates synthesized speech or text to be output according to the intervention instruction output from the intervention / arbitration unit 209 and reproduces it on the speaker 213 or the display 214.

  As described above, intervention for conversations by a plurality of speakers in the vehicle 110 can be performed. Note that the processing shown in the flowchart of FIG. 3 is repeatedly executed. The conversation intervention support system obtains conversation voices at any time, continuously monitors the conversation status, the relationship between speakers, and the group state, and performs intervention when it is determined that intervention is necessary.

<Conversation situation analysis processing>
Next, the details of the conversation situation analysis process in step S303 will be described. FIG. 4 is a flowchart showing the flow of the conversation state analysis process. Note that the processing of the flowchart shown in FIG. 4 does not need to be performed in the order shown, and some of the processing may be omitted.

  In step S401, the conversation state analysis unit 204 detects an utterance section from the sound data separated from the sound source, and adds a section ID and a time stamp for each utterance section. Note that the utterance section is a continuous section in which voice is uttered. The end of the utterance period is, for example, until no utterance of 1500 milliseconds or more occurs. By this process, the conversational voice can be separated into a plurality of voice data for each speaker and each speech section. Hereinafter, the voice of an utterance in one utterance section is also simply referred to as an utterance. FIG. 5 shows each utterance separated in step S401.

  In step S402, the conversation state analysis unit 204 calculates an utterance feature amount (voice feature amount) for each utterance. Examples of the utterance feature amount include utterance volume, pitch, tone, duration, and utterance speed (average mora length). The utterance volume is the sound pressure level of the utterance. The tone is the pitch of the sound or the sound itself, and the pitch of the sound is specified by the number of vibrations (frequency) per second of the sound wave. The pitch is the perceived pitch of the sound and is specified by the physical pitch (fundamental frequency) of the sound. The average mora length is calculated as the utterance length (time) per mora. Note that mora is the number of beats. Here, with respect to the speech volume, pitch, tone, and speech speed, an average value, maximum value, minimum value, fluctuation range, standard deviation, and the like within the speech section may be obtained. In the present embodiment, these utterance feature amounts are calculated, but not all of the utterance feature amounts exemplified here may be calculated, or utterance feature amounts other than those exemplified here may be calculated.

In step S403, the conversation state analysis unit 204 obtains the speaker's emotion for each utterance from the change in the utterance feature amount. Satisfaction, dissatisfaction, excitement,
Examples include anger, sadness, expectation, security, and anxiety. The emotion can be obtained based on, for example, changes in the volume, pitch, and tone of the utterance from the normal time. The normal utterance feature amount of each speaker may be obtained from the utterance feature amount obtained so far, or information stored in the user information / use history DB 123 may be used. Note that the emotion of the speaker does not need to be determined based only on the utterance (voice data). The speaker's emotion can also be obtained from the content (text) of the utterance. Further, the emotion of the speaker can be obtained based on a change in the facial feature amount by calculating the facial feature amount from the facial image of the speaker photographed by the camera 113, for example.

  In step S <b> 404, the conversation state analysis unit 204 performs speech recognition processing using the speech recognition corpus / dictionary 205 for each utterance, and converts the utterance content into text. Existing technology may be applied to the speech recognition processing. The utterance content (text) shown in FIG. 5 is obtained by the process of step S404.

  In step S405, the conversation situation analysis unit 204 estimates the intention and topic of the utterance by referring to the vocabulary intention understanding corpus / dictionary 206 from the content (text) of each utterance. The intention of utterance includes, for example, topic extraction, proposal, approval / disapproval of proposal, and aggregation of opinions. The topic of utterance includes, for example, the genre, place, thing, etc. of the utterance. The genre of utterance includes, for example, eating and drinking, travel, music, weather, and the like. For example, the location that has become a topic includes a place name, a landmark, a store name, and a facility name. The vocabulary intention understanding corpus / dictionary 206 is used to specify a vocabulary and an utterance genre that are used in cases such as “cut out a topic, propose, ask a question, agree, disagree, and aggregate things”. It includes a dictionary of vocabularies relating to “food, travel, music, weather, etc.” and “vocabulary names, landmarks, store names, facility names, etc.” for identifying places of interest. In estimating the utterance intention, it is preferable to consider not only the text but also the emotion of the speaker. For example, if the content of the utterance (text) indicates consent to the proposal, a more detailed utterance intention can be estimated by considering the emotions of the speaker and whether the content is happily agreed or awkwardly agreed. can do.

  As a result of the processing in step S405, for each utterance, it is possible to estimate the speaker's intention such as “what to do” and the genre that is the topic. For example, with respect to the text of the utterance ID 2 in FIG. 5, from the word “Italian”, the genre is “food” and “Kitakamakura” by matching with the dictionary. It can be inferred that the topical location is “Kamakura” and that the intention of the utterance is “suggestion” from the word “doi”.

FIG. 6 shows the extraction result of the genre that is the topic, the topic location, and the intention of the utterance for each utterance shown in FIG. In the present embodiment, “speech n (S)” in which the intention or the like is estimated is expressed by the following equation, for example.

Utterance n (S) = (G _n , P _n , I _n )

Here, n is an utterance ID (1 to k), and it is assumed that the utterance IDs are assigned in the order of the utterances. S is the speaker (A, B, C ...) , G n, P n, I n are, respectively, the genre of the estimated speech, location has become a hot topic, the intention of the speech.

For example, when the utterance 1 of the speaker A is checked against the vocabulary intention understanding corpus / dictionary 206 and matched with “G ₁ : Food and drink”, “P ₁ : Kamakura”, and “I ₁ : Extraction of topic” As shown.

Utterance 1 (A) = ("Food &Drink","Kamakura","Cut out topic")

  In addition, it is also preferable to obtain information such as the genre that is the topic, the topic location, and the intention of the utterance in consideration of information other than the content (text) of the utterance. In particular, the intention of the utterance is preferably obtained in consideration of the emotion of the speaker obtained from the utterance feature amount. Even if the utterance content indicates approval for the proposal, it can be determined from the utterance feature quantity whether the user agrees happily or not. Further, depending on the utterance, the above information may not be extracted from the utterance content (text). In such a case, the conversation state analysis unit 204 may estimate the intention of the utterance in consideration of the extraction result of the utterance intention before and after the chronological occurrence or the utterance content (text).

  In step S406, the conversation state analysis unit 204 extracts the utterances estimated to be the same theme in consideration of the genre of each utterance obtained in step S405 and the time-series results of the utterances, and the result is obtained. The utterance group is identified as an utterance group included in a series of conversations. By this process, it is possible to specify an utterance included from the start to the end of one conversation.

  In the identity determination of the conversation theme, similarity of utterance genre and topic location is taken into consideration. For example, it is determined that the genre is “food” from the extracted word “fish” and the topic location is “sea” from the extracted word “sea”, but the utterance ID 5 is “food”. It can be determined that they have the same conversation theme. Further, when the utterance includes a word for determining “topic extraction” (“determine”) like utterance ID 1 and a word for determining “aggregation” (“decision”) like utterance ID 9 Each utterance can be estimated as the utterance at the start or end of the conversation of the same theme. Also, considering the temporal relationship of utterances, even if the genre of utterances and the topic location are the same, if the time interval between utterances is too long, it may be determined as a different conversation theme. Also, some utterances do not include vocabulary from which intent, genre, etc. can be extracted. In such a case, considering the flow of chronological utterances, it is preferable to consider that the utterances of the same speaker occurring between the start and end of the same conversation are included in the same conversation.

  FIG. 7 is a diagram illustrating a result of specifying a series of utterance groups from the genre, topic location, and utterance intention of each utterance shown in FIG. Here, three conversations are extracted. The conversation 1 is a conversation related to “food (lunch)”, “food (dish)”, and “Kamakura”, and includes utterance IDs 1, 2, 3, 5, 7, and 9. Conversation 2 is a conversation related to “weather” and “sports (athletic meet)”, and includes utterance IDs 4, 6, and 8. Note that “weather” and “sports (athletic meet)” are different genres, but if utterances related to “sports (athletic meet)” occur immediately after utterances related to “weather”, these utterances are “weather” Determined to be included in the conversation. The conversation 3 is a conversation related to “music” and includes utterance IDs 10 and 11.

  The utterance shown in FIG. 5 is performed by a total of five speakers A to E, but not all of them are participating in the same conversation. Here, three of the speakers A to C have a conversation 1 regarding food and drink, and speakers D and E have a conversation 2 regarding weather. The conversation state analysis unit 204 in the present embodiment focuses on the genre of each utterance, the topic (thing) that is the topic, and the intention of the utterance. However, it is possible to appropriately identify the utterance group included in the series of conversations.

In the present embodiment, the series of “conversations m” specified in this way is represented by the following expression, for example.

Conversation _{m (S A, S B,} S C ...)
= {Speak 1 (S _A ), Speak 2 (S _B ), Speak 3 (S _C ) ...}
_{= T m {(G A,} P A, I A), (G B, P B, I B), (G C, P C, I C) ...}

Here, m is a conversation ID (1 to k), and it is assumed that conversation IDs are assigned in the order in which conversations occur. S _{A, B, C ...} is the speaker (A, B, C ...) , T m, G n, P n, I n , respectively, the estimated conversation theme, genre speech, speech Indicate the location of the topic and the intention of the utterance.

For example, when the utterance group of the theme “Food & Drink” of speakers A, B, and C is specified as conversation 1, it is shown as follows.

Conversation 1 (A, B, C)
= T _“meal” {(“food (lunch)”, “Kamakura”, “cutting out topics”),
("Eating and drinking (cooking)", "Kamakura", "Proposal"),
("Food & Drink (Cooking)", "na", "Negation / Proposal") ...}

  In step S407, the conversation state analysis unit 204 generates and outputs conversation state data in which the above analysis results are integrated. For example, the conversation status data includes information as shown in FIG. 8 for utterances in the same conversation during the most recent predetermined period (for example, 3 minutes). A speaker with many utterances is a speaker whose both the number of utterances and the utterance time within a period are equal to or greater than a predetermined value (for example, once and 10 seconds). A speaker with few utterances is a speaker whose number of utterances within the period is both less than a predetermined value. The average utterance interval or overlap between speakers is the time of a silent period between utterance intervals or the overlap of utterance intervals for each speaker pair. The utterance volume, tone, pitch, and utterance speed are required for each speaker and for all speakers. Each is represented by one or more of an average value, a maximum value, a minimum value, a variation range, and a standard deviation within a period, and particularly when variation is measured remarkably, it is shown in association with information such as the corresponding utterance content. In addition, the conversation status data includes, for each utterance within the period, the text of the utterance content, the conversation theme, the estimated speaker name, the intention of the utterance, the topic of the utterance (genre, place, thing, etc.), and the emotion of the speaker. The conversation status data also includes correspondences between utterances and relationships between speakers.

  FIG. 9A shows an example in which the correspondence between utterances, the conversation theme of each utterance, the intention of the utterance, and the emotion of the speaker are displayed. In FIG. 9A, the utterance sections are shown in time series for the speakers A to E, and the correspondence between the utterances is shown by arrows. For each utterance, the intention of the utterance and the emotion of the speaker are shown (when available). For example, speaker B makes a proposal (utterance ID 2) for topic extraction by speaker A (utterance ID 1), and speaker C responds to the proposal and re-suggests (utterance ID 3) after receiving both of these utterances. You can see that you are doing. Note that the correspondence between utterances does not necessarily need to be determined based only on utterances (voice data). For example, it is determined whether or not a certain utterance is directed to a specific member from the line of sight of the speaker acquired from the camera 113 and the face or body direction, and the correspondence between the utterances is obtained based on this determination result. May be.

  FIG. 9B shows what kind of utterance is occurring in the conversations of the speakers A to E and how the relationship and intimacy between the speakers is estimated. Yes. In the utterance between any two speakers, the intimacy between the two speakers can be determined from the intention of the utterance, utterance features (number of utterances, utterance time, overlap of utterances, tension level), wording (politeness) Relationships (whether flat or vertical) can be determined. Although not shown in FIG. 9B, when there is a vertical relationship between speakers, it can be determined which is the superior and which is the inferior.

The conversation state analysis unit 204 outputs the conversation state data as described above to the group state determination unit 207. By using the conversation status data, it becomes possible to link the flow of conversation and the feature amount change of each utterance, and the state of the group in conversation can be accurately estimated.

<Group status determination processing>
Next, details of the group state determination processing in step S304 in FIG. 3 will be described. FIG. 10 is a flowchart showing the flow of the group state determination process.

  In step S <b> 1001, the group state determination unit 207 acquires the conversation state data output from the conversation state analysis unit 204. The group state determination unit 207 analyzes the group state including the group type, the role (relationship) of each member, the group state change, and the like by the following processing based on the conversation state data.

  In step S1002, the group state determination unit 207 determines a connection between speakers in a conversation. The conversation status data includes the speaker of each utterance, the connection between utterances, and the intention of the utterance (suggestion, approval, disagreement, etc.). Therefore, based on the conversation status data, the frequency of conversation between the speaker pair (for example, “speaker A and speaker B frequently talk directly”, “speaker A and speaker B directly ) And the extent to which the pair of speakers is proposing, approving, or disagreeing (“Speaker A makes suggestions to speaker B X times, and agrees with Y”) , Times, disagreements Z times "). The group state determination unit 207 obtains such information for each speaker pair in the group.

  In step S1003, the group state determination unit 207 determines the opinion exchange status between members. Opinion exchange status includes information such as the active exchange of opinions within the group, the ratio of approval and disapproval of proposals, and the presence or absence of leaders in decision making. The activity of exchanging opinions can be evaluated by, for example, the number of utterances between proposals and final decision making, or the number of pros and cons. In addition, the presence or absence of a leader in decision-making depends on whether or not there is little disagreement with respect to the proposal of a specific speaker and whether only consent or approval is generated. It can be evaluated by whether or not it is adopted as a general opinion. Since the conversation status data includes the speaker of each utterance, the connection between the utterances, the intention of the utterance, the content of the utterance, etc., the group state determination unit 207 determines the opinion exchange status based on the conversation status data. it can.

  In step S1004, the group state determination unit 207 determines the wording of the utterance feature amount and utterance content included in the conversation state data, the connection between the speakers obtained in step S1002, and the opinion exchange situation between the speakers obtained in step S1003. Based on the above, the group type (group model) is estimated. The group type is predefined, for example, as shown in FIG. 11A, for example, group type A: “a group that is flat and has a high degree of intimacy, and members can freely express their opinions”, group type B: “A group that has a hierarchical relationship but has a high degree of intimacy and a specific member leading the group's decision-making”, Group type C: “A hierarchical relationship that has a low degree of closeness and a specific member An example of such a group is “leading group decision making”. The group type A assumes a group in which everyone, such as best friends, is connected in a flat manner. The group type A may or may not include a leader (particularly a member who has an influence on decision making). The group type B is assumed to be a group such as a family that has a strong connection between members and a vertical relationship. In the group type B, there is a leader (for example, a parent). Group type C assumes a group that has a relatively dry connection and a vertical relationship, such as a boss and a subordinate in the workplace. In the group type C, there is a leader (top person). Here, only three are given as an example, but the number of group types may be any number.

  The group model definition storage unit 208 stores determination criteria for each group type. The group model definition storage unit 208 stores a plurality of determination criteria based on utterance feature amounts, wording of utterance contents, connection between speakers, opinion exchange information, and the like. FIG. 11B shows an example of a determination criterion based on the utterance feature amount. Group type A is “a group that is flat and has a high degree of intimacy, and members can speak to each other without hesitation.” For example, “all speakers are actively speaking”, “speaking tends to overlap There are many features such as “Yes”, “Fluctuation in tone and pitch of each utterance”, “Fluctuation in utterance volume is large”, and “Some oppositions to the proposal occur to some extent”. Therefore, in the group model definition storage unit 208, as a criterion for the group type A based on the utterance feature amount, for example, “the number of utterances is 3 or more in 3 minutes or a total of 20 seconds or more Criteria such as “over 60%”, “overlap of utterances over 3 minutes in 3 minutes or a total of over 5 seconds”, “variation of each speaker's tone, pitch or sound pressure level over a predetermined threshold” including. The group state determination unit 207 evaluates how much the current group satisfies these determination criteria, and obtains an evaluation value indicating the probability that the current group is the group type A. Evaluation values are similarly obtained for the other group types B and C.

  The group state determination unit 207 may determine the type of the group using only the evaluation value obtained here, that is, based only on the utterance feature amount, but other factors are considered in order to further improve the determination accuracy. To determine the group type.

  The group state determination unit 207, for example, analyzes the utterance content (text) in the conversation, and includes command words, honorific words, polite words, humility words included in each speaker's utterances, words (words used in close relations), Get the frequency of words used by children and words used for children. As a result, the language of each speaker in the conversation can be understood. The group state determination unit 207 estimates the group type in consideration of wording. For example, if there is a person who speaks in a command tone in the group and there is a person who responds with honorific, polite or humble words, it can be determined that there is a high possibility of being group type C. Further, when “a person who speaks in a command tone in the group, but there is a person who responds with a simple word”, it can be determined that the possibility of the group type A is high. Further, when “uses many words spoken by almost the speakers in the group”, it can be determined that the possibility of the group type A or B is high. Further, if there are “a person who speaks in the language used by the parent (adult) for the child and a person who speaks in the language used by the child” in the group, it can be determined that the possibility of the group type B is high. The example given here is an example, and if the relationship between the group type and the wording is defined in advance, the group state determination unit 207 determines which group type the current group corresponds to is highly likely. it can.

  The group state determination unit 207 can also determine the group type from the opinion exchange status in the conversation. For example, when opinions are actively exchanged within the group, or when a relatively large number of refusals or disagreements occur with respect to the proposal, it can be determined that the possibility of group type A or B is high. In addition, when the exchange of opinions is not active in the group or when the leader exists in the group, it can be determined that the possibility of the group type C is high. The example given here is an example. If the relationship between the group type and the opinion exchange status is defined in advance, the group status determination unit 207 is likely to be the group type to which the current group corresponds. Can be determined.

  As described above, the group state determination unit 207 integrates the group types estimated based on the utterance feature amount, the wording, the opinion exchange situation, and the connection between speakers, and determines the best matching group type as the current Determine as group type.

In step S1005, the group state determination unit 207 estimates the role of each member in the group using the analysis results in steps S1002 and S1003 and other conversation status data. Roles in the group include leaders in decision making and followers to the leaders. Moreover, you may estimate a superior, a subordinate, a parent, a child, etc. as a role. It is also preferable to consider the group type determined in step S1004 in the member role estimation.

  In step S1006, the group state determination unit 207 estimates a group state change. The group status includes the frequency of utterances, participants in the conversation, who is the leader in the conversation, and the like. Examples of the state change estimated in step S1006 include a decrease in the utterance frequency of a specific speaker, a decrease in the overall utterance frequency, separation of conversation groups, and a change in leader.

  In step S1007, the group state determination unit 207 intervenes and adjusts the group type estimated in step S1004, the role of each member estimated in step S1005, and the group state change estimated in step S1006 as group state data. Output to the unit 209. The intervention / arbitration unit 209 can grasp the state of the group in conversation by referring to the group state data, and can perform appropriate intervention according to the state.

<Intervention / mediation>
Next, details of the intervention content determination process in step S305 in FIG. 3 will be described. FIG. 12 is a flowchart showing the flow of the intervention content determination process.

  In step S <b> 1201, the intervention / arbitration unit 209 acquires the conversation state data output from the conversation state analysis unit 204 and the group state data output from the group state determination unit 207. The intervention / arbitration unit 209 determines the content of information to be presented when performing intervention or arbitration by the following processing based on these data.

  In step S <b> 1202, the intervention / arbitration unit 209 acquires from the intervention policy definition storage unit 210 an intervention policy corresponding to the group type and group state change included in the group state data. The intervention policy is information indicating which member in the group is preferentially supported and how it is supported according to the group status. Examples of intervention policies defined in the intervention policy definition storage unit 210 are shown in FIGS.

FIG. 13A shows an example of an intervention policy corresponding to the group type. For example, as an example of an intervention policy for Group Type A that has a close relationship and high degree of intimacy, and members can express their opinions with each other, in order to encourage members to discuss and decide, On the other hand, a policy of “presenting information on a selection element (for example, a candidate store when a meal place is determined)” is defined. In addition, members who are in a weak position that cannot express an opinion as an example of an intervention policy for group type B that has a hierarchical relationship but is highly intimate and a specific member is leading the group's decision making. In order to encourage the members to draw out their opinions and be hired, “provide the facilitator member information on which member's opinion is preferable and information on the selection factors. A policy is defined that “leads out opinions and helps them be adopted”. In addition, as an example of an intervention policy for group type C, which has a hierarchical relationship, has low intimacy, and a specific member leads the group's decision-making, supports that only the opinions of a specific member are not adopted For this reason, “In the first decision-making matter, the opinion of the upper member is treated preferentially, but in the second and later, information on which member's opinion is preferable to the facilitator member, A policy is defined that presents information on the selected element, sequentially extracts opinions from the relevant members, and supports the adoption of the opinions. In addition, the facilitator members in these policies are those who are particularly close to weak members who cannot express their opinions, and are able to draw out their opinions and support their adoption. Intended for people. In FIG. 13A, one intervention policy is defined for each group type, but a plurality of intervention policies may be defined for each group type.

  FIG. 13B is an example of an intervention policy corresponding to a group state change. For example, when an utterance stagnation (decrease in utterance frequency) of a specific speaker occurs, information related to the topic before the stagnation is presented if it occurs with a change in topic. In addition, when the stagnation of the entire utterance has occurred, information related to the topic before the stagnation is presented. In addition, when a group is divided into two subgroups and each has a different conversation, information related to the topic in one of the subgroups may be of interest to other groups. To present. Also, when the leader changes, information is provided so that the new leader can lead the topic. In FIG. 13B, one intervention policy is defined for each state change of the group, but a plurality of intervention policies may be defined for each state change.

  The intervention policy as described above can be said to be information that defines the priority of intervention for each member in the group and what kind of intervention is to be performed in accordance with the type of group and the change in the state of the group. Here, the priority of intervention is set for members who have roles in the group (such as leaders) rather than being set for individual members, or specific conditions (decreased utterance frequency) are set. It may be set for members who meet. However, not all intervention policies need to include intervention priority.

  In step S1203, the intervention / mediation unit 209 determines an intervention target member and an intervention method based on the intervention policy acquired in step S1202. For example, the intervention / mediation unit 209 determines to provide the leader with information that suits the tastes of other members, or provides information related to a topic preferred by a speaker whose utterance is stagnant. To decide. In step S1203, it may be determined that no intervention is currently performed. The determination in step S1203 does not have to be performed based only on the intervention policy, but is preferably performed based on other information such as conversation status data. For example, if it is determined that opinions are exchanged for decision making within the group based on the intention of the utterance included in the conversation status data, the intervention target and intervention method are determined based on the intervention policy that supports decision making. Good.

  In step S1204, the intervention / mediation unit 209 generates / acquires presentation information according to the intervention target member and the intervention method. For example, when providing information that matches the preferences of other members to the leader, first, the preferences of other members are determined based on the previous conversation theme and the emotions (excitement level, etc.) of the members. It is determined by acquisition or acquisition from the user information DB 123. If the member prefers Italian cuisine during a conversation about a lunch place, information about the Italian restaurant is acquired from the related information WEB site 130 or the like. At this time, it is preferable to narrow down the stores to be presented in consideration of the position information obtained from the GPS device 112 of the vehicle 110.

  In step S1205, the intervention / arbitration unit 209 generates and outputs intervention instruction data including the presentation information generated and acquired in step S1204. In the present embodiment, intervention instruction data is transmitted from the server device 120 to the navigation device 111 of the vehicle 110. The output control unit 212 of the navigation device 111 generates synthesized speech and display text based on the intervention instruction data, and presents information from the speaker 213 and the display 214 (S306).

  The series of conversation intervention support processes (FIG. 3) described above are repeatedly executed. It is preferable that the repetition interval is short so that intervention can be performed at an appropriate timing for the utterance. However, it is not necessary to perform every process every time in the repetitive process. For example, the conversation state analysis S303 and the group state determination S304 may be performed with a certain interval (for example, 3 minutes). In the group state determination, the group type determination and the group state change determination may be performed at different execution intervals.

<Advantageous effects of this embodiment>
In the present embodiment, the conversation state analysis unit 204 specifies an utterance group having the same conversation theme in a conversation performed by a plurality of speakers, and whether or not there is a relationship between the utterances. You can understand what kind of relationship there is. Furthermore, the conversation situation can be estimated from the interval between speeches and the degree of overlap between speakers in the same conversation. In the conversation state analysis method according to the present embodiment, even when a large number of speakers are divided into different groups and have conversations at the same time, the state of each conversation can be grasped.

  In the present embodiment, the group state determination unit 207 determines the type or state change of the group in which the conversation is performed, the role of each speaker in the group, and the relationship between them based on the conversation state data. I can grasp it. Such an understanding makes it possible to determine which speaker is to be supported more preferentially when the system intervenes in the conversation, and to perform appropriate intervention according to the group status.

<Modification>
In the above description, an example in which the conversation intervention support system is configured as a telematics service in which the vehicle and the server device cooperate with each other is shown, but the specific system form is not limited to this. For example, it can be configured as a system that acquires a conversation in a room such as a conference room and intervenes in the conversation.

201: microphone 202: noise removal unit 203: sound source separation unit 204: conversation state analysis unit 205: speech recognition corpus / dictionary 206: vocabulary intention understanding corpus / dictionary 207: group state determination unit 208 group model definition storage unit 209: Intervention / arbitration unit 210: Intervention policy definition storage unit 211: Related information database 212: Output control unit 213: Speaker 214: Display

Claims (11)

A conversation situation analysis device for analyzing the situation of conversations by a plurality of speakers,
An acquisition means for acquiring conversation voices from a plurality of speakers;
Separating means for separating the conversational sound into a plurality of utterances for each speaker and each utterance section;
For each of the plurality of utterances, recognition means for recognizing the utterance content using voice recognition processing;
An analysis means for analyzing the relationship between utterances based on utterance content, and an analysis means for identifying utterances estimated to have the same conversation theme as a series of utterance groups based on the content of each utterance;
Conversation situation analysis device. The recognizing means recognizes the utterance content by comparing a text of the utterance obtained by the voice recognition processing with a dictionary,
The analysis means obtains the intention and topic of the utterance by comparing the text of the utterance content recognized by the recognition means with the dictionary, and estimates the conversation theme of the utterance based on the intention and topic of the utterance.
The conversation state analysis apparatus according to claim 1. For each of the plurality of utterances, further comprising a feature quantity calculating means for calculating a voice feature quantity,
The analysis means estimates the emotion of the speaker at the time of each utterance based on the change in the voice feature amount for each speaker, and estimates the intention of the utterance in consideration of the emotion.
The conversation state analysis apparatus according to claim 2. The analysis means acquires a correspondence relationship between utterances in the series of utterance groups based on the intention of the utterance;
The conversation state analysis apparatus according to claim 2 or 3. It further has an imaging means for photographing the speaker,
The analysis unit obtains a correspondence relationship between utterances in the series of utterance groups in consideration of the direction of the speaker's body, face, or line of sight in the image captured by the imaging unit.
The conversation state analysis apparatus according to claim 4. It further has an imaging means for photographing the speaker,
The analysis unit estimates a speaker's emotion according to a change in a facial feature amount calculated from the speaker's face image in the image captured by the imaging unit, and considers the emotion to determine the relationship between utterances. analyse,
The conversation state analysis apparatus according to any one of claims 1 to 5. The analysis means obtains the relationship between the speakers based on the relationship between the utterances and at least one of the content of the utterance, the feature amount of the utterance, and the emotion of the speaker at the time of utterance.
The conversation state analysis apparatus according to any one of claims 1 to 6. An output means for outputting conversation status data which is data relating to the series of utterance groups;
The conversation situation analysis device according to any one of claims 1 to 7. The conversation status data includes the speaker of each utterance, the correspondence between utterances, the meaning and intention of each utterance, the emotion of the speaker at the time of each utterance, the utterance frequency of each speaker in the utterance group, and the voice feature amount in each utterance , Including at least one of the relationships between speakers,
The conversation state analysis apparatus according to claim 8. A support device for intervening and supporting a conversation by a plurality of speakers,
A conversation situation analysis device according to claim 8 or 9,
Group state determination means for determining the state of a group of a plurality of speakers participating in a series of utterance groups based on conversation state data output from the conversation state analysis device;
Intervention means for determining the content of the intervention in the conversation based on the state of the group and performing the intervention in the conversation;
A support device comprising: A conversation situation analysis method for analyzing the situation of conversations by a plurality of speakers, wherein a computer
An acquisition step of acquiring conversation voices by a plurality of speakers;
A separation step of separating the conversational sound into a plurality of utterances for each speaker and each utterance section;
For each of the plurality of utterances, a recognition step for recognizing the utterance content using a speech recognition process;
An analysis step for analyzing the relationship between utterances based on utterance content, and an analysis step for identifying utterances estimated as the same conversation theme as a series of utterance groups based on the content of each utterance;
Conversation situation analysis method to execute.

Images