JP2017123027A - Conversation support system, conversation support device, and conversation support program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a conversation support system, a conversation support device, and a conversation support program that encourage a participant of a conversation who has missed a chance to speak at an appropriate timing in the conversation to speak.SOLUTION: In a conversation support system 100, a next speaker probability estimation unit 108 estimates a probability with which a participant of a conversation will become the next speaker at an arbitrary time based on the result of measuring the non-verbal behavior of the participant in the conversation. A control unit 109 estimates an anticipated next speaker, who is a participant to speak next, based on the probability of each participant becoming the next speaker, and a timing when the anticipated next speaker starts to speak, and promotes the anticipated next speaker to speak if the control unit has determined that the anticipated next speaker did not speak at the estimated timing. Speech inducing units 111 to 119 receive an instruction from the control unit and promote a subject to speak.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a conversation support system, a conversation support apparatus, and a conversation support program.

  When a conversation participant speaks at an appropriate time, the conversation proceeds in a good atmosphere. The purpose of the conversation is various, but conversation with a good atmosphere is effective in achieving the purpose, and the satisfaction of participants is high. However, speaking at an appropriate time during a conversation requires advanced communication skills. Therefore, even in a scene where utterance is expected, some participants miss the opportunity to speak because they are not good at grasping the timing of utterance or because other participants have spoken first. In addition, when the participant who is expected to speak most from the atmosphere of the conversation does not speak, the other participants wait for the participant's speech and hesitate to speak, and as a result, the conversation stops. Sometimes.

  On the other hand, there is a technique for making the next speaker speak in a conference. In this technology, in a multi-person TV (television) conference, the utterance desire of each participant is estimated from physical motion and utterance information, and the next next speaker is determined based on the desire. Then, in order to make the next speaker surely speak, control is performed to let other participants hear the filler of the person. In addition, in a multi-person TV conference, there is a technique for detecting a person with many utterances and generating voice so as to suppress the utterance of the person so that everyone can speak without any difference (see, for example, Patent Document 2). .

JP 2012-146072 A JP 2007-158526 A

  The technique of Patent Document 1 described above prevents the start of utterances by a system other than the participant who tried to be the next speaker, and the technique of Patent Document 2 blocks (disturbs) the speech of a specific participant. This is to promote the speech of other participants. However, these conventional techniques do not prompt the participant or other participants to speak when the participant misses the timing of speaking.

  In view of the above circumstances, the present invention provides a conversation support system, a conversation support apparatus, and a conversation support program capable of prompting an utterance when a participant of the conversation misses an appropriate utterance timing during the conversation. The purpose is that.

  One aspect of the present invention is a next speaker that estimates a next speaker probability, which is a probability that each of the participants becomes a next utterance at an arbitrary time, based on a measurement result of non-verbal behavior of each participant in conversation. Based on the probability estimation unit and the next speaker probability of the participant, a predicted next speaker who is a participant to speak next and a timing at which the predicted next speaker starts speaking are estimated and estimated. When it is detected that the predicted next speaker does not speak at the timing, a control unit that instructs the predicted next speaker to be uttered as a target, and receives an instruction from the control unit, A conversation support system including an utterance guidance unit that performs processing for prompting a subject to speak.

  One aspect of the present invention is the above-described conversation support system, in which the control unit detects a speech other than the next speaker when the predicted next speaker does not speak at the estimated timing. The utterance guidance unit is instructed to urge utterance with the speaker as the target person.

  One aspect of the present invention is the conversation support system described above, in which the utterance guide unit selects a robot or a speaker displayed on a display device to perform an operation indicating transfer of the utterance right to the target person. Control.

  One aspect of the present invention is the conversation support system described above, in which the utterance guiding unit is a robot or a speaker's eye, head, or display displayed on a display device so as to direct a line of sight toward the subject. Control one or more of the body parts.

  One aspect of the present invention is the above-described conversation support system, in which the utterance guide unit controls the robot or the speaker's upper limb displayed on the display device to be presented to the subject.

  One aspect of the present invention is the above-described conversation support system, in which the utterance guiding unit outputs a voice that urges the subject to speak.

  One aspect of the present invention is a next speaker that estimates a next speaker probability, which is a probability that each of the participants becomes a next utterance at an arbitrary time, based on a measurement result of non-verbal behavior of each participant in conversation. Based on the probability estimation unit and the next speaker probability of the participant, a predicted next speaker who is a participant to speak next and a timing at which the predicted next speaker starts speaking are estimated and estimated. A control unit that instructs the utterance guiding unit that performs the process of prompting the utterance to prompt the utterance with the predicted next speaker as the target person when it is detected that the predicted next speaker has not uttered at the timing; , A conversation support device.

  According to one aspect of the present invention, a computer estimates a next speaker probability, which is a probability that each participant will make a next utterance at an arbitrary time, based on a measurement result of non-verbal behavior of each participant in conversation. A next speaker probability estimating step, and estimating a timing at which the predicted next speaker and the predicted next speaker who are to be uttered next based on the next speaker probability of the participant start utterance, When it is detected that the predicted next speaker does not speak at the estimated timing, an instruction is given to the speech guidance unit that performs processing for prompting speech to promote the speech with the predicted next speaker as the target person. And a control step for executing a control step.

  According to the present invention, when a participant of a conversation misses an appropriate utterance timing during the conversation, the utterance can be prompted.

It is a figure which shows the outline of a function structure with which the robot 100 in 1st Embodiment is provided. It is a figure which shows the specific structural example of the sensor 103 in 1st Embodiment. It is a diagram illustrating an example of the next speaker probability P ^{ns i} output by the next speaker probability estimation unit 108 _^(t) in the first embodiment. It is a figure which shows the specific example of the detail of a structure of the sound control part 110 in 1st Embodiment. It is a figure which shows the specific example of the external appearance and structure of the robot 100 in 1st Embodiment. It is a flowchart which shows operation | movement of the robot 100 in 1st Embodiment. It is a figure which shows the outline of a function structure with which the robot 100A in 2nd Embodiment is provided. It is a flowchart which shows operation | movement of 100 A of robots in 2nd Embodiment. It is a figure which shows the example of a breath inhalation area. It is the figure which illustrated the gaze object transition pattern. It is the figure which illustrated time structure information.

Embodiments of the present invention will be described below with reference to the drawings.
(First embodiment)
FIG. 1 is a diagram illustrating an outline of a functional configuration included in the robot 100 according to the first embodiment. The robot 100 is an example of a conversation support system. The robot 100 according to the first embodiment is a robot that has a conversation with a plurality of participants. As shown in FIG. 1, the robot 100 includes a microphone 101, a camera 102, a sensor 103, a voice input unit 104, a video input unit 105, a sensor input unit 106, a speech segment detection unit 107, and a next story. Person probability estimation unit 108, control unit 109, sound control unit 110, mouth control unit 111, gaze control unit 112, head control unit 113, torso control unit 114, speaker 115, mouth A head drive unit 116, an eye drive unit 117, a head drive unit 118, and a torso drive unit 119.

  The microphone 101 collects sounds around the robot 100 including voices of participants who are talking, and outputs a sound signal including a voice signal (hereinafter simply referred to as a voice signal). As long as the microphone 101 can collect at least the voices of the participants, the installation position and number of the microphones 101 can be set arbitrarily. For example, the microphone 101 is composed of a plurality of microphones attached to each of a plurality of participants. In this manner, the microphone 101 is close to the participant's mouth and can be collected with high accuracy by wearing the participant individually. For example, the microphone 101 may be mounted on the robot 100 or may be installed in the outside world other than the participants and the robot 100. In the robot 100, the plurality of microphones 101 and the voice input unit 104 are connected so as to be able to transmit and receive voice signals by wire or wirelessly.

  The camera 102 captures images of participants who are talking and outputs a video signal. If the camera 102 can photograph all the participants, the installation position and the number of the cameras 102 can be arbitrary. For example, the camera 102 is an imaging device having a wide angle of view so that all participants are in view. Further, for example, the camera 102 may be a plurality of cameras for the number of participants who respectively photograph the appearance of all participants. In this case, in the robot 100, the video input unit 105 and the plurality of cameras are connected so as to be able to transmit and receive video signals by wire or wirelessly.

  The sensor 103 is a first sensor that measures the position of a participant who has a conversation with respect to the position of the robot 100, a second sensor that measures the breathing motion of the participant, a third sensor that detects a gaze target of the participant, and A plurality of sensors such as a fourth sensor for detecting the participant's head movement are provided, and sensor signals from these sensors are output to the sensor input unit 106.

FIG. 2 is a diagram illustrating a specific configuration example of the sensor 103 according to the first embodiment.
As shown in FIG. 2, the sensor 103 measures a position measuring device (first sensor) 201 that measures the position (particularly the face position) of the participant who talks with respect to the position of the robot 100 and the breathing motion of the participant. A breathing motion measuring device (second sensor) 202, a gaze target detecting device (third sensor) 203 for detecting a participant's gaze target, and a head motion detecting device for detecting a participant's head motion ( 4th sensor) 204. The position measuring device 201 is installed in the robot 100, for example. The breathing motion measurement device 202 is attached to the trunk of the participant, and the gaze target detection device 203 and the head motion detection device 204 are attached to the participant's head. The position measuring device 201 is connected to the sensor input unit 106. The respiratory motion measurement device 202, the gaze target detection device 203, and the head motion detection device 204 are connected to the sensor input unit 106 so as to be able to transmit and receive sensor signals in a wired or wireless manner.

  The voice input unit 104 in FIG. 1 receives the voice signal from the microphone 101 and outputs the voice signal to the utterance section detection unit 107, the next speaker probability estimation unit 108, and the sound control unit 110. The voice input unit 104 performs processing such as converting the voice signal from the microphone 101 into a voice signal in a signal format that can be processed in the robot 100. The video input unit 105 receives the video signal from the camera 102 and outputs the video signal to the next speaker probability estimation unit 108. The video input unit 105 performs processing such as converting the video signal from the camera 102 into a video signal in a signal format that can be processed in the robot 100. The sensor input unit 106 receives the sensor signal from the sensor 103 and outputs the sensor signal to the next speaker probability estimation unit 108. The sensor input unit 106 performs processing such as converting the sensor signal from the sensor 103 into a sensor signal in a signal format that can be processed in the robot 100.

The utterance section detection unit 107 detects a section in which each participant uttered based on the voice feature amount obtained from the voice signal from the voice input unit 104 by an existing arbitrary technique. For example, the utterance section detection unit 107 provides an arbitrary window width based on the voice signal from the voice input unit 104 and indicates the voice characteristics such as the power, the number of zero crossings, and the frequency of the voice signal in the section. It is calculated as a voice feature value that is a value. The utterance section detection unit 107 detects the utterance section by comparing the calculated voice feature quantity with a predetermined threshold. The utterance section detection unit 107 outputs utterance section information, which is information related to the detected utterance section, to the next speaker probability estimation unit 108, the control unit 109, and the sound control unit 110. The utterance section information includes the start and end times of the utterance and the information of the speaker. Note that the VAD (Voice Activity Detection) technique for automatically detecting a section where speech is present (speech section) and a section where speech is not present (non-speech section) in a speech signal acquired from the microphone 101 is as follows. As shown in Document 1, it is a known technique. The utterance interval detection unit 107 detects an utterance interval using a known VAD technique.
Reference 1: Hiroshi Sawada and four others, "Detection of utterance section with multi-microphones with multi-persons -Example with pin microphones", Acoustical Society of Japan Spring Research Presentation, pp. 679-680, March 2007

The next speaker probability estimation unit 108 includes an audio signal from the audio input unit 104, a video signal from the video input unit 105, a sensor signal from the sensor input unit 106, and speech segment information from the speech segment detection unit 107. And the next speaker probability, which is the probability that each participant will be the next speaker at time t, is output. The next speaker probability estimation unit 108 acquires speaker information indicating a speaker in the speech section specified by the speech section information based on the audio signal, the video signal, the sensor signal, and the speech section information. The next speaker probability estimation unit 108 is based on the audio signal, the video signal, the sensor signal, and the acquired speaker information, and the next speaker probability P ^ns _i that is the probability that each participant i will be the next speaker at time t. (T) is calculated and output to the control unit 109. The next speaker probability estimation unit 108 calculates the next speaker probability P ^ns _i (t) based on the non-language behavior of the participant. That is, the next-speaker probability estimating unit 108 does not need to obtain information on the user's language behavior by calculating the next-speaker probability P ^ns _i (t) by analyzing the utterance contents of the participants. The next speaker probability estimation unit 108 outputs the speaker information and the participant position information to the control unit 109 in addition to the next speaker probability P ^ns _i (t).

  Note that the next speaker probability estimation unit 108 may acquire the position information of the participant based on, for example, a sensor signal obtained by measuring the position of the participant of the sensor 103 or based on a video signal. Alternatively, it may be acquired based on a sensor signal and a video signal obtained by measuring the positions of the participants of the sensor 103.

FIG. 3 is a diagram illustrating an example of the next speaker probability P ^ns _i (t) output by the next speaker probability estimation unit 108 according to the first embodiment. FIG. 3 shows an example of change in the next speaker probability P ^ns _i (t) after time t _{bu when the} participants A to D break the utterance of the participant A. The rectangle to which reference numeral 31 is assigned indicates the utterance section of participant A. The utterance section 31 ends at the utterance end time t _bu . The dotted line indicated by the next speaker probability P ^ns _A (t) 32 indicates the change in the next speaker probability of the participant A at time t after the utterance end time t _bu . The dotted line indicated by the next speaker probability P ^ns _B (t) 33 indicates the change in the next speaker probability of the participant B at the time t after the utterance end time t _bu . The dotted line indicated by the next speaker probability P ^ns _C (t) 34 indicates the change in the next speaker probability of the participant C at time t after the utterance end time t _bu . A dotted line indicated by a next speaker probability P ^ns _D (t) 35 indicates a change in the next speaker probability of the participant D at time t after the utterance end time t _bu . Thus, the next speaker probability estimation unit 108 determines the next speaker probability P ^ns _i (t) at time t after the utterance end time t _bu of the participant i (iε {A, B, C, D}). Calculate the change in. Details of the next speaker estimation processing in the next speaker probability estimation unit 108 will be described later.

  The control unit 109 in FIG. 1 receives the next speaker probability from the next speaker probability estimation unit 108 as an input, and a predicted next episode that is a participant predicted to speak next based on the input next speaker probability. And the predicted next speaker start timing (utterance start timing). The control unit 109 includes an operation pattern information storage unit 1091. The motion pattern information storage unit 1091 stores motion pattern information indicating a motion that the robot 100 prompts to speak.

The control unit 109 selects a predicted next speaker using any one of first to fifth next speaker selection methods described below. Note that, in the following description, a case will be described in which four participants A, B, C, and D have a conversation with the robot 100. The control unit 109 acquires the next speaker probabilities P ^ns _i (t), (i∈ {A, B, C, D}) of A to D from the next speaker probability estimation unit 108.

(First speaker selection method)
The control unit 109 compares the next speaker probabilities P ^ns _i (t), (iε {A, B, C, D}) of the participants A to D, respectively. The control unit 109 determines any of the participants A to D having the highest maximum value of the next speaker probability P ^ns _i (t) as the predicted next speaker. The control unit 109 sets time t when the next speaker probability P ^ns _i (t) of the predicted next speaker takes the maximum value as the speech start timing of the predicted next speaker. Note that the control unit 109 may determine that the predicted next speaker is the robot 100 when any of the next speaker probabilities P ^ns _i (t) of the participants A to D does not exceed the first threshold. .

(Second next speaker selection method)
The control unit 109 has a maximum value greater than or equal to the second threshold value at the earliest time among the participants A to D when the next speaker probability P ^ns _i (t), (iε {A, B, C, D}). Participants who take are determined to be predicted next speakers. The control unit 109 sets time t when the next speaker probability P ^ns _i (t) of the predicted next speaker takes the maximum value as the speech start timing of the predicted next speaker. Note that the control unit 109 may determine that the predicted next speaker is the robot 100 when any of the next speaker probabilities P ^ns _i (t) of the participants A to D does not exceed the second threshold. .

(Third next speaker selection method)
The control unit 109 sets each of the next speaker probabilities P ^ns _i (t), (iε {A, B, C, D}) of the participants A to D for a predetermined time (for example, from the utterance end time). Integrate for 3 to 4 seconds or more) to obtain an integral value P ^ns _i . The integration interval may be an infinite time from the utterance end time, or may be a time until the next speaker probability P ^ns _i (t) of all the participants A to D becomes less than a predetermined value and becomes no significant value. The control unit 109 determines any of the participants A to D having the largest integral value P ^ns _i as a predicted next speaker. The control unit 109 sets time t when the next speaker probability P ^ns _i (t) of the predicted next speaker takes the maximum value as the speech start timing of the predicted next speaker. Note that the control unit 109 may determine that the predicted next speaker is the robot 100 when the integral value P ^ns _i does not exceed the third threshold value for all the participants A to D.

(Fourth speaker selection method)
The control unit 109 adds the next speaker probability P ^ns _i (t), (i∈ {A, B, C, D}) of the participants A to D (P ^ns _A (t) + P ^ns _B (T) + P ^ns _C (t) + P ^ns _D (t)) is acquired and compared with an arbitrary probability P _γ that is the fourth threshold value. The control unit 109 determines that the added value of the next speaker probabilities of all the participants A to D is equal to or greater than the probability P _γ ((P ^ns _A (t) + P ^ns _B (t) + P ^ns _C (t) + P ^ns _D ( When t)) ≧ _Pγ ), the predicted next speaker and the utterance start timing are obtained by any one of the first to third next speaker selection methods described above. However, in the first to third next speaker selection methods, the comparison with the first to third threshold values may not be performed. The control unit 109 adds the next speaker probabilities of all the participants A to D to be less than the probability P _γ ((P ^ns _A (t) + P ^ns _B (t) + P ^ns _C (t) + P ^ns _D ( If t)) <P _γ ), it is determined that the predicted next speaker is the robot 100.

(Fifth speaker selection method)
The control unit 109 sets each of the next speaker probabilities P ^ns _i (t), (iε {A, B, C, D}) of the participants A to D for a predetermined time (for example, 3 to 4). sec or longer) is integrated to obtain the integrated value ^{P ns} _i. The integration interval may be an infinite time from the end of the utterance, or may be a time until the next speaker probability P ^ns _i (t) of all participants is less than a predetermined value. The control unit 109 acquires an added value (P ^ns _A + P ^ns _B + P ^ns _C + P ^ns _D ) obtained by adding the integral values P ^ns _i of all the participants A to D, and an arbitrary probability that is the fifth threshold value Compare with _Pθ . When the added value of the integral values of the participants A to D is equal to or higher than the probability P _θ (the control unit 109) ((P ^ns _A + P ^ns _B + P ^ns _C + P ^ns _D ) ≧ P _θ ), The predicted next speaker and the utterance start timing are obtained by any of the third next speaker selection methods. However, in the first to third next speaker selection methods, the comparison with the first to third threshold values may not be performed. Control unit 109, if the sum of integral values of all participants A~D is less than the probability _{^{P θ ((P ns A +}} P ns B + P ns C + P ns D) <P θ) is predicted next story It is determined that the person is the robot 100.

The next speaker probability P ^ns _i (t), (iε {A, B, C, D}) often has a peak after a predetermined time from the end of the utterance, as shown in FIG. Therefore, in the first to fifth next speaker selection methods, the control unit 109 provides a window width including the time t for obtaining the next speaker probability P ^ns _i (t), and the next story within the window width. The maximum speaker probability may be used as the next speaker probability P ^ns _i (t) at time t. In addition, in the first to fifth next speaker selection methods, the control unit 109 provides a window width including the time t for obtaining the next speaker probability P ^ns _i (t), and the next talk within the window width is provided. When the speaker probability has a plurality of peaks, the next speaker probability of the nth peak (n is an integer of 1 or more) may be used as the next speaker probability P ^ns _i (t) at time t. .

  When the control unit 109 determines that the predicted next speaker is the robot 100 by the first to fifth next speaker selection methods, the control unit 109 outputs an utterance control signal that instructs the sound control unit 110 to utter. . When the control unit 109 determines that the predicted next speaker is one of the participants A to D, the control unit 109 outputs an utterance control signal that instructs the sound control unit 110 to suppress the utterance and is estimated. It is determined whether or not the predicted next speaker has spoken at the utterance start timing. When the control unit 109 detects that the predicted next speaker does not utter at the estimated utterance start timing, the control unit 109 reads the operation pattern information from the operation pattern information storage unit 1091. The control unit 109 transmits an utterance guidance operation control signal instructing to perform the operation indicated by the read operation pattern information to the sound control unit 110, the mouth control unit 111, the line-of-sight control unit 112, the head control unit 113, and Output to one or more of the body controller 114. The utterance guidance operation control signal is a signal for instructing the utterance guidance target person to perform an operation for prompting utterance. The action pattern information includes, for example, outputting utterance sound with a content urging the utterance guidance target person, directing the line of sight toward the utterance guidance target person, and presenting the upper limb in the direction of the utterance guidance target person. The operation is shown. The utterance guidance operation control signal includes information for specifying the utterance guidance target person. The control unit 109 sets the utterance induction target person as a participant different from the predicted next speaker or the predicted next speaker. The speech guidance operation control signal output to the line-of-sight control unit 112, the head control unit 113, or the torso control unit 114 further includes information on the position of the speech guidance target person.

  When the control unit 109 does not detect the start of the utterance section by a predetermined timing after outputting the utterance induction operation control signal, the control unit 109 selects a new utterance induction target person. The control unit 109 generates an utterance guidance operation control signal for instructing a new utterance guidance target person to perform an utterance urging operation, and the utterance guidance operation control signal is transmitted to the sound control unit 110, the mouth control unit 111, and the line of sight. Output to one or more of the control unit 112, the head control unit 113, and the torso control unit 114.

  Mouth control unit 111, line-of-sight control unit 112, head control unit 113, torso control unit 114, speaker 115, mouth drive unit 116, eye drive unit 117, and head drive unit 118 Then, the body drive unit 119 operates as an utterance guidance unit that receives an instruction from the control unit 109 and performs processing for prompting the utterance guidance target person to speak.

  The sound control unit 110 outputs a sound signal to the speaker 115 based on the utterance control signal or the utterance guidance operation control signal from the control unit 109. The sound control unit 110 determines whether or not to cause the robot 100 to speak based on the speech control signal. When the sound control unit 110 determines that the robot 100 is to speak based on the speech control signal, the sound control unit 110 generates conversation information including conversation contents (words) to be uttered by the robot 100, and the generated conversation information is included in the generated conversation information. Based sound signal is output. For example, the sound control unit 110 analyzes the conversation content of the participant based on the voice signal and the utterance section information, and generates conversation information for causing the robot 100 to utter based on the analysis result. In addition, when receiving the utterance guidance operation control signal, the sound control unit 110 generates conversation information that urges the utterance guidance target person set in the utterance guidance operation control signal to speak, and based on the generated conversation information. Output a sound signal.

Here, the details of the configuration of the sound control unit 110 in the first embodiment will be described with reference to an example.
FIG. 4 is a diagram illustrating a specific example of details of the configuration of the sound control unit 110 according to the first embodiment. The sound control unit 110 includes a voice analysis unit 401, a conversation information generation unit 402, a conversation information DB (database) 403, an utterance information generation unit 404, and a sound signal generation unit 405.

  The conversation information DB 403 stores conversation sample information for allowing the robot 100 to speak. The conversation sample information, noun often used in everyday conversation, "Hello" greeting and "Thank you" such as, it is the information that contains the phrase of the speech signal that frequently used in everyday conversation, such as "Are you okay?" . Furthermore, the conversation information DB 403 stores a voice signal of each speaker's name and a voice signal of a phrase that prompts utterances such as “What do you think?” And “Do you have something?” .

  The voice analysis unit 401 analyzes the voice signal based on the voice signal from the voice input unit 104 and the utterance section information from the utterance section detection unit 107, specifies the contents (words), and determines the analysis result. Output.

When the conversation information generation unit 402 receives the utterance control signal, the conversation information generation unit 402 generates conversation information that is the utterance content of the robot 100 based on the analysis result of the voice analysis unit 401. The conversation information generation unit 402 acquires conversation sample information corresponding to the content of conversation from the conversation information DB 403 based on the analysis result of the voice analysis unit 401. The conversation information generation unit 402 generates conversation information based on the acquired conversation sample information. The conversation information generation unit 402 generates conversation information in response to a request for conversation information from the utterance information generation unit 404 and outputs the conversation information to the utterance information generation unit 404.
In addition, when the speech information generation unit 402 receives the speech guidance operation control signal, the conversation information generation unit 402 generates a speech signal of the name of the speech guidance target person set in the speech guidance operation control signal and a speech signal of the phrase that prompts speech. Obtained from the conversation information DB 403. The conversation information generation unit 402 generates conversation information for continuously outputting these audio signals, and outputs the conversation information to the utterance information generation unit 404.

  The utterance information generation unit 404 receives the conversation information from the conversation information generation unit 402 and the utterance control signal or utterance guidance operation control signal from the control unit 109 and outputs an utterance signal. The utterance information generation unit 404 requests the conversation information generation unit 402 for conversation information based on the utterance control signal or the utterance guidance operation control signal from the control unit 109. The utterance information generation unit 404 generates an utterance for the robot 100 to utter based on the conversation information acquired from the conversation information generation unit 402 upon request and the utterance control signal or utterance guidance operation control signal from the control unit 109. Generate a signal. The utterance information generation unit 404 outputs the generated utterance signal to the sound signal generation unit 405.

  The sound signal generation unit 405 receives the utterance signal from the utterance information generation unit 404 and outputs a sound signal. The sound signal generation unit 405 generates a sound signal for uttering from the speaker 115 based on the utterance signal from the utterance information generation unit 404 and outputs the sound signal to the speaker 115.

  The mouth control unit 111 shown in FIG. 1 outputs a mouth drive signal to the mouth drive unit 116 based on the speech guidance operation control signal from the control unit 109. The line-of-sight control unit 112 outputs an eye part drive signal to the eye part drive unit 117 based on the speech guidance operation control signal from the control unit 109. The head control unit 113 outputs a head drive signal to the head drive unit 118 based on the speech guidance operation control signal from the control unit 109. The torso control unit 114 outputs a torso drive signal to the torso drive unit 119 based on the speech guidance operation control signal from the control unit 109.

  FIG. 5 is a diagram illustrating a specific example of the appearance and configuration of the robot 100 according to the first embodiment. The robot 100 in the first embodiment has, for example, the appearance shown in FIG. 5 and the functional configuration shown in FIG.

  As shown in FIG. 5, the robot 100 is, for example, a humanoid robot (humanoid robot) having a shape modeled on a human upper body. The robot 100 includes at least a speech function for speaking, a voice recognition function for recognizing a human voice, and a camera function for photographing a participant. The robot 100 includes a head 53 having a face on which a right eye 51a and a left eye 51b and a mouth portion 52 are arranged.

  The robot 100 includes a neck 54 that supports the head 53 and a body 55 that supports the neck 54. The torso 55 has a right arm 55a and a left arm 55b, which are upper limbs, provided on the upper side. A camera 102 is installed between the right eye 51 a and the left eye 51 b of the head 53. In the following description, the right eye 51a and the left eye 51b are collectively referred to as the eye part 51.

  Since only the camera 102 is shown in FIG. 5 in the configuration shown in FIG. 1, an example of the installation position of the configuration shown in FIG. The microphone 101 and the sensor 103 are installed at an arbitrary position in the body 55 of the robot 100 or a position away from the body 55 (for example, the position of the participant). The configuration other than the microphone 101, the camera 102, and the sensor 103 shown in FIG. 1 is installed inside the robot 100. For example, the speaker 115 is installed inside the mouth portion 52 shown in FIG.

  Here, the arrangement of the mouth drive unit 116, the eye drive unit 117, the head drive unit 118, and the torso drive unit 119 included in the robot 100 and the objects to be driven will be described. The head 53 includes an eye drive unit 117 that moves the black eyes (line of sight) of the right eye 51 a and the left eye 51 b, and a mouth drive unit 116 that opens and closes the mouth 52.

  The neck portion 54 includes a head drive unit 118 that causes the head 53 to perform a predetermined movement (for example, a movement that changes the direction of the face or the face), and supports the head 53. The torso 55 includes a torso drive unit 119 that moves the shoulder and inflates the chest part as if breathing. The mouth drive unit 116 opens and closes the mouth 52 of the robot 100 based on the mouth drive signal from the mouth control unit 111. The eye drive unit 117 controls the direction of black eyes (= the direction of the line of sight of the robot 100) in the eye 51 of the robot 100 based on the eye drive signal from the line of sight control unit 112.

  The head drive unit 118 controls the movement of the head 53 of the robot 100 based on the head drive signal from the head control unit 113. The torso drive unit 119 controls the shape of the torso 55 of the robot 100 based on the torso drive signal from the torso controller 114. The torso driving unit 119 also controls the movement of the right arm 55a and the left arm 55b of the robot 100 based on the torso driving signal from the torso control unit 114.

Next, the operation of the robot 100 in the first embodiment will be described.
FIG. 6 is a flowchart showing the operation of the robot 100 according to the first embodiment. The process shown in FIG. 6 is a process that is performed when the robot 100 starts an operation of having a conversation with a plurality of participants. Hereinafter, a case where the participants A to D and the robot 100 are participating in the conversation will be described as an example.

  The audio input unit 104 receives the audio signal from the microphone 101, the video input unit 105 receives the video signal from the camera 102, and the sensor input unit 106 receives the sensor signal from the sensor 103 (step). S101). The utterance section detection unit 107 calculates a speech feature amount based on the speech signal from the speech input unit 104, and compares the calculated speech feature amount with a predetermined threshold value to detect a speech section (step S102).

The next speaker probability estimation unit 108 determines that each participant i (iε {A, B, C, D}) at the time t based on the audio signal, the video signal, the sensor signal, and the acquired speaker information. Next speaker probability P ^ns _i (t), which is the probability of becoming a speaker, is calculated (step S103). Based on the next-speaker probability of each participant calculated by the next-speaker probability estimating unit 108, the control unit 109 uses one of the first to fifth next-speaker selection methods described above to predict a predicted next talk. The utterance start timing of the speaker and the predicted next speaker is obtained (step S104).

  The control unit 109 determines whether the predicted next speaker is one of the participants A to D (step S105). When the control unit 109 determines that the predicted next speaker is one of the participants A to D (NO in step S105), the control unit 109 outputs an utterance control signal that instructs the sound control unit 110 not to utter. . The control unit 109 determines whether any of the participants A to D has uttered before the utterance induction timing elapses (step S106). This utterance induction timing is a timing after the utterance start timing, may be immediately after the utterance start timing, and is a timing determined based on a time when it feels unnatural when silence continues during a conversation. Also good. In the latter timing, for example, a predetermined time (for example, 2 to 3 seconds) may elapse from the utterance end time, or a predetermined time may elapse from the estimated utterance start timing. Further, the utterance induction timing may be a time at which the next speaker probability of the predicted next speaker becomes a predetermined value or less.

  When the utterance section detection unit 107 detects the start of the utterance section by the utterance guidance timing, the control unit 109 determines that any of the participants A to D has uttered (YES in step S106), and performs the process in step S107. Execute.

  On the other hand, when the utterance section detection unit 107 does not detect the start of the utterance section by the utterance guidance timing (NO in step S106), the control unit 109 performs utterance guidance processing (step S108). In the utterance guidance process, the control unit 109 sets the utterance guidance target person as the predicted next speaker or the speaker having the next next speaker probability next to the predicted next speaker. Whether the utterance induction target person is a predicted next speaker or a speaker having the next speaker probability that is next to the predicted next speaker may be determined in advance or may be determined dynamically. In the case of dynamic determination, for example, the probability Px that the participant x uttered when the participant x (x is any one of A to D) who is the predicted next speaker is urged in the past. Participant y uttered when utterance was urged in the past to participant y (y ≠ x, y is any one of A to D) with the next speaker probability next to the predicted next speaker It can be determined based on the probability Py. Specifically, if Px is greater than or equal to a predetermined threshold, or if Px> Py, participant x is the predicted next speaker, and if Px is lower than the predetermined threshold or if Px <Py Let y be the predicted next speaker.

  The control unit 109 uses the sound control unit 110, the mouth control unit 111, the line-of-sight control unit 112, the head control unit 113, and the torso control unit to generate an utterance guide operation control signal in which information for specifying the utterance guide target is set. Output to one or more of 114. The control unit 109 further sets information on the position of the speech guidance target person in the speech guidance operation control signal output to the line-of-sight control unit 112, the head control unit 113, or the torso control unit 114. Thereby, the robot 100 performs any one or a plurality of operations (Operation 1) to (Operation 5) below, and signals the transfer of the utterance right to the utterance induction target person.

(Operation 1) The sound control unit 110 outputs, from the speaker 115, speech sound whose content prompts the speech guidance target person to speak. For example, an utterance of contents for making a question or request to the utterance guidance target person is output. Specifically, utterances such as "What do you think about Mr. XX?" At the same time, the mouth control unit 111 controls the mouth 52 to open and close while the mouth drive signal is output to the mouth drive unit 116 and the sound is output from the speaker 115.

(Operation 2) The line-of-sight control unit 112 outputs an eye part drive signal to the eye part drive unit 117, and controls the direction of the black eye in the eye part 21 to be the direction of the speech guidance target person. In addition, it is known that turning the line of sight will promote speech (Reference 2).
Reference 2: Ryo Ishii and two others, “Gaze control for conversation promotion in avatar voice chat system”, Journal of Human Interface Society, Vol. 10, no. 1, p. 87-94, 2008

(Operation 3) The head control unit 113 outputs a head driving signal to the head driving unit 118 and moves the neck 54 to control the head 53 toward the utterance guidance target person. Thus, the head 53 and the line of sight are controlled to be in the direction of the predicted next speaker.

(Operation 4) The torso control unit 114 outputs a torso drive signal to the torso drive unit 119, and controls the torso 55 to rotate in the direction of the speech guidance target person. Thereby, it controls so that a trunk | drum, a head, and a eyes | visual_axis may become the direction of a speech guidance object person.

(Operation 5) The torso control unit 114 outputs a torso drive signal to the torso drive unit 119, and controls so that one or both of the right arm 55a and the left arm 55b are directed toward the utterance guidance target person.

  The control unit 109 determines whether any of the participants A to D has uttered before the next utterance guidance timing has elapsed after performing the utterance guidance process in step S108 (step S109). When the utterance section detection unit 107 does not detect the start of the utterance section until the next utterance guidance timing elapses (NO in step S109), the control unit 109 performs the utterance guidance process again (step S108). .

  When performing the utterance guidance process after determining NO in step S109, the control unit 109 may set the utterance guidance target person as the utterance guidance target person in the immediately preceding utterance guidance process, or in the immediately preceding utterance guidance process. It is good also as a participant with the next speaker probability next to the said participant. For example, when the same participant becomes an utterance induction target consecutively m times (m is an integer of 1 or more), the control unit 109 determines that the next speaker has the next highest probability of the speaker. Also good. Further, the control unit 109 may set the utterance induction target person as the next participant of the utterance induction target person in the utterance induction process immediately before the time when the next speaker probability has the maximum value. Alternatively, the control unit 109 may set the utterance guidance target person as the predicted next speaker when the predicted next speaker has not yet become the utterance guidance target person.

Specifically, in the first or fourth next speaker selection method, when the next speaker probability P ^ns _i (t) of the participant x is the highest and the utterance start timing is the time t1, the time t1 Let t2 (P ^ns _x (t2) <o) be a time when the next speaker probability P ^ns _x (t) falls below a certain probability o under the condition that the participant x does not start speaking. When there is another participant y exceeding the next speaker probability P ^ns _x (t2) at time t2 (P ^ns _x (t2) <P ^ns _y (t2)), the robot 100 speaks to the participant y at time t2. (T2 ≧ t1).

Further, in the third or fifth next speaker selection method, at the time t1 (speech start timing) when the integral value P ^ns _i of the participant x is the highest and the next speaker probability P ^ns _x (t) is the maximum. Let t2 (P ^ns _x (t2) <o) be a time when the next speaker probability P ^ns _x (t) falls below a certain probability o under the condition that the participant x does not start speaking. When there is another participant y exceeding the next speaker probability P ^ns _x (t2) at time t2 (P ^ns _x (t2) <P ^ns _y (t2)), the robot 100 speaks to the participant y at time t2. (T2 ≧ t1).

In the second next speaker selection method, after the time t1 at which the next speaker probability P ^ns _x (t) of the participant x is maximized, another participant y whose next speaker probability has the maximum value is obtained. When there is, the robot 100 prompts the participant y to speak at time t2 (t2 ≧ t1).

  When the utterance section detection unit 107 detects the start of the utterance section before the next utterance guidance timing elapses (YES in step S109), one of the participants A to D speaks. And the process of step S107 is executed.

  In step S105, when the control unit 109 determines that the predicted next speaker is the robot 100 (step S105: YES), the control unit 109 outputs an utterance control signal for controlling the robot 100 to utter. The sound control unit 110 determines that speech is to be performed based on the speech control signal from the control unit 109, generates conversation information for causing the robot 100 to speak, and outputs a sound signal based on the generated conversation information to the speaker 115. (Step S110). As a result, the robot 100 generates an utterance corresponding to the sound signal from the speaker 115.

  The sound control unit 110 determines whether or not to end the utterance of the robot 100 based on the utterance control signal from the control unit 109 (step S111). If the utterance of the robot 100 is not terminated (NO in step S111), the sound control unit 110 returns to the process in step S110. When the utterance of the robot 100 is ended (YES in step S111), the sound control unit 110 stops outputting the sound signal in response to stopping the generation of the conversation information.

  After YES is determined in step S106, step S109, or step S111, the robot 100 determines whether or not to end the conversation operation for performing conversation with a plurality of participants (step S107). If it is determined that the conversation operation is not terminated (NO in step S107), the process returns to step S101. If it is determined that the conversation operation is to be ended (YES in step S107), the robot 100 ends the conversation operation. For example, when the participant turns on a power switch (not shown) or turns on a conversation mode switch (not shown), the robot 100 starts a conversation operation, and the participant turns off the power switch. The robot 100 ends the conversation operation at the timing when the switch of the conversation mode or the switch of the conversation mode is turned off.

  As described above, the robot 100 according to the first embodiment estimates the next speaker based on the next speaker probability of each participant when conversing with a plurality of participants, and the estimated next speaker. If the timing of utterance is missed, the next speaker is urged to speak. Thereby, it can guide so that the participant who missed the timing of speech may speak easily. Further, when the estimated next speaker misses the utterance timing, it is possible to urge other speakers to speak. For example, the participant may intentionally refrain from speaking. Therefore, by prompting other participants to speak, silence is generated during the conversation, and the participants are less likely to feel awkward.

  In step S109, if the utterance section detection unit 107 does not detect the start of the utterance section until the next utterance guidance timing elapses, the robot 100 performs the process from step S103 and performs each participation. The next speaker probabilities of the speakers A to D may be recalculated.

  In step S106, when the control unit 109 determines that the speech of any participant has been detected (YES in step S106), the control unit 109 further determines whether or not the speaker is the predicted next speaker. You may make it do. When the control unit 109 determines that the speaker is the predicted next speaker, the control unit 109 performs the process of step S107. On the other hand, when the control unit 109 determines that the speaker is not the predicted next speaker, the other participant y interrupts and speaks even though the participant x who is the predicted next speaker is scheduled to speak. The robot 100 is controlled to urge the participant x to speak. The timing of prompting can be arbitrary. For example, an utterance break of the participant y is detected, and immediately after the break is detected or after a predetermined time from the break, the utterance guidance process is performed with the predicted next speaker as the utterance guidance target person. The break can be defined as, for example, when a ending such as “to” is uttered or when a silent period exceeds an arbitrary time Ds. Further, the control unit 109 instructs to output a voice with a content for stopping the utterance of the participant y immediately after detecting the utterance of the participant y or a predetermined time after the utterance start time of the participant y. May be output to the sound control unit 110. As a result, the sound control unit 110 outputs the voice of the utterance with the content “Please wait a moment, Mr. YY” from the speaker 115. Thereafter, the robot 100 may execute the processing from step S108 with the predicted next speaker as a speech guidance target person. In this way, the speech of the predicted next speaker may be urged by the voice whose content is to stop the speech of the participant y.

  In step S109, when it is determined that the utterance of any participant is detected (YES in step S109), the control unit 109 determines whether or not the utterer is the utterance guidance target person. It may be. When the control unit 109 determines that the utterer is the utterance guidance target person, the control unit 109 executes the process of step S107. On the other hand, when the control unit 109 determines that the speaker is not the speech guidance target person, the other participant y interrupts and speaks even though the participant x who is the speech guidance target person is scheduled to speak. The robot 100 is controlled to urge the participant x to speak. For example, as described above, the control unit 109 performs the utterance guidance process for the same utterance guidance target person immediately after detecting the utterance break of the participant y or after a predetermined time from the break. Alternatively, the control unit 109 instructs to output a voice whose content is to stop the utterance of the participant y immediately after detecting the utterance of the participant y or a predetermined time after the utterance start time of the participant y. Is output to the sound control unit 110.

  In this embodiment, the case where the robot 100 participates in the conversation has been described as an example. However, the robot 100 may perform only the operation of prompting the participant to speak without participating in the conversation.

(Second Embodiment)
In the second embodiment, the next speaker probability P ^ns _R (t) of the robot itself is obtained from the movement of the robot itself (breathing motion, line-of-sight motion, head motion). The robot estimates the predicted next speaker and the utterance start timing based on the obtained next speaker probability P ^ns _R (t) and the next speaker probabilities of other participants. Therefore, the robot participates in the conversation and performs the same movement as the person in the conversation during the conversation. That is, during the conversation, the robot performs a breathing action that emits a breathing sound or changes the swelling of the chest, a gaze action such as directing the line of sight toward the speaker, and a head action that crawls according to the conversation. Below, it demonstrates centering on the difference with 1st Embodiment.

  FIG. 7 is a diagram illustrating an outline of a functional configuration provided in the robot 100A according to the second embodiment. A robot 100A in the second embodiment shown in FIG. 7 includes the same components as the robot 100 in the first embodiment. Therefore, in the description of the robot 100A, the same components as those of the robot 100 according to the first embodiment are denoted by the same reference numerals and description thereof is omitted.

  As shown in FIG. 7, the robot 100A includes a microphone 101, a camera 102, a sensor 103, an audio input unit 104, a video input unit 105, a sensor input unit 106, an utterance section detection unit 107, and a next story. Person probability estimation unit 108A, control unit 109A, sound control unit 110, mouth control unit 111, line of sight control unit 112, head control unit 113, torso control unit 114, speaker 115, mouth Unit driving unit 116, eye unit driving unit 117, head driving unit 118, torso driving unit 119, and sensor signal conversion unit 120.

  The next speaker probability estimation unit 108A includes an audio signal from the audio input unit 104, a video signal from the video input unit 105, a sensor signal from the sensor input unit 106, and speech segment information from the speech segment detection unit 107. Then, the pseudo sensor signal from the control unit 109A is input, and the next speaker probability, which is the probability that each participant and the robot 100A become the next speaker at time t, is output. The pseudo sensor signal is a sensor signal output by the sensor 103 when it is assumed that the robot 100 is operated based on the operation control signal generated by the control unit 109A and the operation of the robot 100A is detected by the sensor 103. is there.

The next speaker probability estimation unit 108A acquires speaker information indicating the speaker in the speech section specified by the speech section information based on the audio signal, the video signal, the sensor signal, and the speech section information. The next speaker probability estimation unit 108A, based on the audio signal, the video signal, the sensor signal, the pseudo sensor signal, and the acquired speaker information, is the probability that the robot 100A will be the next speaker at time t, P ^ns _R (t ) And the next speaker probability P ^ns _i (t), which is the probability that each participant i will be the next speaker at time t, is output to the control unit 109A. The next speaker probability estimation unit 108A outputs the speaker information and the participant position information to the control unit 109A in addition to the next speaker probabilities P ^ns _R (t) and P ^ns _i (t).

  The next speaker probability estimation unit 108A may acquire the position information of the participant based on, for example, a sensor signal obtained by measuring the position of the participant of the sensor 103 or based on a video signal. Alternatively, the position of the participant of the sensor 103 may be acquired based on the sensor signal and the video signal.

The control unit 109A receives the next speaker probability P ^ns _i (t) from the next speaker probability estimation unit 108A, the speaker information, and the position information of the participant, and outputs a speech control signal or a speech guidance operation control signal. . The control unit 109A estimates the predicted next speaker and the utterance start timing based on each participant and the next speaker probability P ^ns _i (t) of the robot 100A. Specifically, the control unit 109A selects the next speaker using any of the sixth to tenth next speaker selection methods described below. In the following description, a case will be described in which four participants A, B, C, and D have a conversation with the robot 100A. The control unit 109A acquires the next speaker probability P ^ns _i (t), (iε {A, B, C, D, R}) from the next speaker probability estimation unit 108A.

(Sixth speaker selection method)
The control unit 109A compares the participants A to D and the next speaker probability P ^ns _i (t), (iε {A, B, C, D, R}) of the robot 100A. When the control unit 109A determines that P ^ns _R (t) is the maximum, the control unit 109A sets the robot 100A as a predicted next speaker. When the control unit 109A determines that P ^ns _R (t) is not the maximum, the control unit 109A predicts one of the participants A to D having the highest maximum value of the next speaker probability P ^ns _i (t). Judge. The control unit 109A sets the time t when the next speaker probability P ^ns _i (t) of the predicted next speaker takes the maximum value as the speech start timing of the predicted next speaker.

(Seventh speaker selection method)
The control unit 109A predicts either the participant or the robot 100A whose next speaker probability P ^ns _i (t), (iε {A, B, C, D, R}) takes the maximum value at the earliest time. Judge as the next speaker. The control unit 109A sets the time t when the next speaker probability P ^ns _i (t) of the predicted next speaker takes the maximum value as the speech start timing of the predicted next speaker.

(Eighth next speaker selection method)
The control unit 109A sets the next speaker probabilities P ^ns _i (t), (i∈ {A, B, C, D, R}) of the participants A to D and the robot 100A for a predetermined time (for example, Then, the integration value P ^ns _i is obtained. The integration interval may be infinite time from the end of the utterance, or may be the time until the next speaker probability P ^ns _i (t) of all the participants becomes less than a predetermined value and is not significant. The control unit 109A determines that one of the participants A to D or the robot 100A having the largest integral value P ^ns _i is the predicted next speaker. The control unit 109A sets the time t when the next speaker probability P ^ns _i (t) of the predicted next speaker takes the maximum value as the speech start timing of the predicted next speaker.

(9th next speaker selection method)
The control unit 109A adds the next speaker probability P ^ns _i (t), (i∈ {A, B, C, D}) of the participants A to D (P ^ns _A (t) + P ^ns _B (T) + P ^ns _C (t) + P ^ns _D (t)) is acquired. The control unit 109A compares this added value with P ^ns _R (t) · ι obtained by multiplying the next speaker probability P ^ns _R (t) of the robot 100A by a constant ι (ι is an arbitrary value having a positive value). constant). When the control unit 109A determines that the addition value (P ^ns _A (t) + P ^ns _B (t) + P ^ns _C (t) + P ^ns _D (t)) <P ^ns _R (t) · ι, Is the predicted next speaker. When the control unit 109A determines that the added value (P ^ns _A (t) + P ^ns _B (t) + P ^ns _C (t) + P ^ns _D (t)) ≧ P ^ns _R (t) · ι, The predicted next speaker and the utterance start timing are obtained by any one of the first to third next speaker selection methods of the embodiment. However, in the first to third next speaker selection methods, the comparison with the first to third threshold values may not be performed. The predicted next speaker at this time is one of the participants A to D.

(10th next speaker selection method)
The control unit 109A sets the next speaker probabilities P ^ns _i (t), (i∈ {A, B, C, D, R}) of the participants A to D and the robot 100A for a predetermined time (for example, , Integration time P ^ns _i is obtained. The control unit 109A adds a constant ζ to an addition value (P ^ns _A + P ^ns _B + P ^ns _C + P ^ns _D ) obtained by adding the integration values P ^ns _i of all the participants A to _D, and the integration value P ^ns _R of the robot 100A. Is compared with P ^ns _R · ζ multiplied by (ζ is an arbitrary constant having a positive value). Control unit ^109A, if it is determined that _{^{<P ns R · ζ (P}} ns A + P ns B + P ns C + P ns D), the robot 100A and predicted next talker. When the control unit 109A determines that (P ^ns _A + P ^ns _B + P ^ns _C + P ^ns _D ) ≧ P ^ns _R · ζ, the controller selects one of the first to third speakers in the first embodiment. According to the method, the predicted next speaker and the utterance start timing are obtained. However, in the first to third next speaker selection methods, the comparison with the first to third threshold values may not be performed. The predicted next speaker at this time is one of the participants A to D.

The next speaker probability P ^ns _i (t), (iε {A, B, C, D, R}) often has a peak after a predetermined time from the end of the utterance, as shown in FIG. Therefore, in the sixth to tenth next speaker selection methods, control unit 109A provides a window width including time t for determining next speaker probability P ^ns _i (t), and the next episode within the window width. The maximum speaker probability may be used as the next speaker probability P ^ns _i (t) at time t. In addition, in the sixth to tenth next speaker selection methods, the control unit 109A provides a window width including the time t for obtaining the next speaker probability P ^ns _i (t), and the next story within the window width. When the speaker probability has a plurality of peaks, the next speaker probability of the nth peak (n is an integer of 1 or more) may be used as the next speaker probability P ^ns _i (t) at time t. .

The operation pattern information storage unit 1091A included in the control unit 109A stores operation pattern information of operations performed by the robot 100A during conversation in addition to the operation patterns stored in the operation pattern information storage unit 1091 of the first embodiment. The operation performed by the robot 100A during the conversation is, for example, the same operation as the operation performed by a person so as to let other people know that an utterance is to be performed before the utterance is started. For example, as a result of analyzing behaviors performed immediately before a non-speaker speaks as the next speaker when multiple people are talking, the following behaviors (1) to (3) are It is thought that this is an action that indicates to the surroundings.
(1) Speaking inspiratory sound or filler (2) Directing gaze toward the current speaker (3) Speaking into the current speaker's conversation

With reference to the analysis result described above, the control unit 109A controls the robot 100A to perform the operations (1) to (3) described above before the robot 100A speaks, so that the robot 100A speaks soon. Let participants foresee to start. When the robot 100A performs the operations (1) to (3) described above, the next speaker probability P ^ns _R (t) of the robot 100A estimated by the next speaker probability estimation unit 108A increases. That is, the operation of making the surrounding people sense that the utterance is performed includes, for example, an operation of moving the line of sight to the current speaker, an operation of raising the head, an operation of inhaling with the intake sound, and the like.

The control unit 109A may control the robot 100A to perform the above-described operations (1) to (3) using a technique described in the following publicly known document.
There is a known technique described in Reference Document 3 below as a technique for causing the robot 100A to perform the action of uttering the intake sound of (1).
Reference 3: Naoto Yoshida, 3 others, “Design of biological body emotion interaction based on factor analysis on breathing expression with breathing and abdominal movement”, HAI Symposium 2014, 2014 (2) gaze at current speaker As a technique for causing the robot 100 </ b> A to perform the operation of directing the above, there is a known technique described in Reference Document 2 above.
There is a known technique described in Reference Document 4 below as a technique for causing the robot 100 </ b> A to perform the action of speaking the current speaker in (3).
Reference 4: Tomio Watanabe and 3 others, “Physical interaction system based on speech using InterActor”, Journal of Human Interface Society, Vol. 2, No. 2, pp. 21-29, 2000

  When the predicted next speaker is any participant, the control unit 109A performs the same operation as the control unit 109 of the first embodiment. When the predicted next speaker is the robot 100A, the control unit 109A outputs an utterance control signal for controlling the utterance of the robot 100A to the sound control unit 110. Further, the control unit 109A outputs a sound generation instruction signal for instructing to sound a breathing sound or a filler to the sound control unit 110. Here, the filler is an utterance for joining the scenes that appears at the time of complaining, for example, “Ao”, “That”, “Et”, and the like. Further, the control unit 109A acquires the operation pattern information from the operation pattern information storage unit 1091A based on the speaker information and the participant position information from the next speaker probability estimation unit 108A, and generates an operation control signal. The generated motion control signal is output to the mouth control unit 111, the line-of-sight control unit 112, the head control unit 113, and the torso control unit 114.

  The sensor signal conversion unit 120 converts the motion control signal generated by the control unit 109A into a pseudo sensor signal and outputs the pseudo sensor signal to the next speaker probability estimation unit 108A.

  The appearance of the robot 100A in the second embodiment is the same as that of the robot 100 shown in FIG.

  With the above configuration, the robot 100A can turn the line of sight toward the participant based on the operation control signal, or can generate a breathing sound or a filler before speaking, when it is desired to speak. The participant can foresee the robot 100A speaking soon before the robot 100A starts speaking. By this prediction, it is possible to prevent a speech collision between the participant and the robot 100A and realize a smooth conversation.

Next, the operation of the robot 100A in the second embodiment will be described.
FIG. 8 is a flowchart showing the operation of the robot 100A in the second embodiment. The process illustrated in FIG. 8 is a process performed when the robot 100A starts an operation of having conversations with a plurality of participants, similarly to the process illustrated in FIG.

  The audio input unit 104 receives the audio signal from the microphone 101, the video input unit 105 receives the video signal from the camera 102, and the sensor input unit 106 receives the sensor signal from the sensor 103. Further, the conversation operation of the robot 100A is performed under the control of the control unit 109A (step S201). The conversation operation of the robot 100A includes the operations (1) to (3) described above. In response to the conversation operation of the robot 100A, the sensor signal conversion unit 120 outputs a pseudo sensor signal to the next speaker probability estimation unit 108A.

The utterance section detection unit 107 calculates a speech feature amount based on the speech signal from the speech input unit 104, compares the calculated speech feature amount with a predetermined threshold value, and detects a speech section (step S202). The next-speaker probability estimating unit 108A is a next-speak that is the probability that the robot 100A and each participant i will be the next speaker at time t based on the audio signal, video signal, sensor signal, pseudo sensor signal, and speaker information. A person probability P ^ns _i (t) is calculated (step S203).

Based on the robot 100A from the next-speaker probability estimating unit 108A and the next-speaker probability of each participant, the control unit 109A performs prediction using any of the sixth to tenth next-speaker selection methods described above. The utterance start timing of the next speaker and the predicted next speaker is obtained (step S204).
The processing of step S205 to step S211 of the robot 100A is the same as the processing of step S105 to step S111 of the first embodiment. However, the robot 100A directs the line of sight to the participant based on the operation control signal or sounds the breathing sound or filler based on the sound generation instruction signal before the process of step S210.

  As described above, the robot 100 </ b> A according to the second embodiment reduces the occurrence of utterance collisions in which the timing of utterances overlaps with other participants, and the utterance timing of the participants while speaking at an appropriate timing. If you miss, you can encourage utterance.

(Specific example of processing for estimating next speaker common to the first and second embodiments)
Next, a specific example of the process for estimating the next speaker common to the robot 100 described above and the robot 100A in the second embodiment will be described. For example, the techniques of the following references 5 and 6 can be applied to the estimation of the next speaker in the robot 100 and the robot 100A, but any existing technique may be used. When the techniques described in References 5 and 6 are used, the next speaker probability estimation unit 108 is used by using the transition pattern of the gaze behavior of the speaker and the non-speaker based on the gaze target information output by the gaze target detection device 203. Alternatively, the next speaker probability estimation unit 108A predicts the next speaker and the timing of the utterance.

Reference 5: Japanese Patent Application Laid-Open No. 2014-238525 Reference 6: Ryo Ishii and 4 others, “Prediction of next speaker and utterance timing based on gaze transition pattern in multi-person dialogue”, Japanese Society for Artificial Intelligence, SIG -SLUD-B301-06, pp.27-34, 2013

Below, the example of the next speaker estimation technique other than the references 5 and 6 applicable to this embodiment is shown.
The breathing behavior of conversation participants is closely related to the next speaker and the timing of the speech. Using this, the breathing motion of the participant in the conversation is measured in real time, the characteristic breathing motion performed immediately before the start of the utterance is detected from the measured breathing motion, and the next utterance is based on this breathing motion And the utterance timing are calculated with high accuracy. Specifically, as a feature of breathing movement performed immediately before the start of utterance, when a speaker who is speaking continuously speaks (when the speaker continues), he immediately breathes immediately after the end of the utterance. Inhale. Conversely, when the speaker does not speak next (speaker change), inhale slowly after the end of the speech, compared to when the speaker continues. Further, at the time of changing the speaker, the next speaker who speaks next inhales more greatly than the non-speaker who does not speak. Breathing performed before such utterance is performed at a timing roughly determined with respect to the start of the utterance. As described above, since the next speaker performs a characteristic breath inhalation immediately before the utterance, such breath inhalation information is useful for predicting the next speaker and the timing of the utterance. In this next speaker estimation technique, attention is paid to a person's breath inhalation, and information such as the amount of breath inhalation, the length of the breathing section, and timing is used to predict the next speaker and the speech timing.

In the following, the participants P ₁ of the _{A's, ...,} P _A is assumed a situation to perform a face-to-face communication. Participants P _a (where a = 1,..., A, A ≧ 2) are equipped with the respiratory motion measuring device 202 and the microphone 101. Respiration measuring device 202 measures the respiration of the participant P _a, respiration information B _a representative of the measurement results for each discrete time _t, to obtain _t, next speaker probability estimation unit 108 or the next speaker probability It outputs to the estimation part 108A. A configuration in which the respiratory motion measuring device 202 includes a band-type respiratory device will be described. The band-type breathing apparatus outputs a value indicating the degree of breathing depth according to the strength of expansion and contraction of the band. The greater the inhalation of the breath, the greater the stretch of the band, and the greater the exhalation of the breath, the greater the contraction of the band (the less the stretch of the band). Hereinafter, this value is referred to as an RSP value. It should be noted, RSP value, take a different size each participant P _a according to the strength of the expansion and contraction of the band. Therefore, in order to eliminate the difference of RSP values for each P _a resulting therefrom, using the average value mu _a and the standard deviation value [delta] _a of RSP values for each participant P _{a, μ a + δ a} is 1 , μ _a -δ _a normalizes RSP values for each participant _{P a} to be -1. This makes it possible to analyze the same respiratory motion data for all participants P _a. Each breathing motion measuring apparatus 202 sends the normalized RSP value to the next speaker probability estimating unit 108 or the next speaker probability estimating unit 108A as the breathing information Ba _{, t} .

Further, the microphone 101 acquires the voice of the participant P _a, the audio signals V _a representative of the speech of the participant P _a at each discrete time _t, to obtain _t, next speaker probability estimation unit 108 or Tsugihanashi To the person probability estimation unit 108A. The next speaker probability estimator 108 or the next speaker probability estimator 108A removes noise from the input speech signal V _{a, t} (where a = 1,..., A), and further utters the speech interval U _k (where , K is an identifier of the utterance section U _k ) and its speaker P _uk . However, the subscript “P _uk ” represents u _k = 1,. One utterance section U _k is defined as a section surrounded by Td [ms] continuous silence sections, and this utterance section U _k is defined as one unit of utterance. Thus, the following speaker probability estimation unit 108 or the next speaker probability estimation unit 108A, the speech period information representing each speech segment U _k, and speaker information (participant P ₁ representing the speaker P _uk, ..., any get speaker information) indicating whether the speaker P _uk in the speech segment U _k of P _a.

Next speaker probability estimation unit 108 or the next speaker probability estimation unit 108A, the breathing information _{B a} of each participant _{P a,} with _t, suction section _{I a,} the _k extracted breath of each participant _{P a} Further, parameters λ _{a, k} relating to breath inhalation are acquired. The breath inhaling section indicates a section between a start position where the breath is inhaled and an end position where the breath is finished after the breath is being exhaled.

FIG. 9 is a diagram illustrating an example of a breath inhaling section. Using FIG. 9, _a method for calculating the breath inhalation interval I _{a, k} will be exemplified. Here referred to the RSP value in the discrete time t of the participant _{P a R a,} and _t. The RSP value R _{a, t} corresponds to the respiration information B _{a, t} . As illustrated in FIG. 9, for example, when the following (Equation 1) holds,

RSP value _{R a} in the previous two frames discrete time t _{= t s (k), t} continuously decreases, RSP value _{R a} in the subsequent two _frames, since _t is increasing continuously, discrete time t _{Let s (k)} be the inhalation start position. Furthermore, when the following (Equation 2) holds,

Since the RSP values R _{a, t} of the previous two frames at the discrete time t = te _(k) continuously increase and then the RSP values _{Ra, t of the} two frames decrease continuously, the discrete time t _{Let e (k) be} the end position of breath inhalation. In this case, the suction section _{I a} breath of participants _{P a, k} becomes the interval from _{t s (k)} to _{t e (k),} the length of the suction section of breath _{t e (k) -t s} ( _k) .

Next speaker probability estimation unit 108 or the next speaker probability estimation unit 108A, the suction section _{I a breath,} when _k is extracted, the suction section _{I a breath, k,} respiration information _{B a, t,} and speech section Using at least part of U _k , parameters λ ′ _{a, k} relating to breath inhalation are extracted. Parameter lambda _{'a, k} is suction section _{I a} participant _{P a,} the amount of suction breath at _k, the suction section _{I a,} the length of _k, the suction section _{I a,} the suction amount of the breath at the _k It represents at least part of the temporal change and the time relationship between the utterance section U _k and the suction section I _{a, k} . The parameters λ ′ _{a, k} may represent only one of them, or a plurality of them, or all of them. The parameters λ ′ _{a, k} include, for example, at least a part of the following parameters MIN _{a, k} , MAX _{a, k} , AMP _{a, k} , DUR _{a, k} , SLO _{a, k} , INT1 _{a, k} . The parameter λ ′ _{a, k} may include only one of these, may include a plurality of these, or may include all of them.
· _{MIN a, k:} RSP value _{R a} at the start of the suction of the breath of the participants _{P a, t,} that is, the suction section _{I a breath, k} of the RSP value _{R a,} minimum value of _t.
· _{MAX a, k:} RSP value _{R a} of at the end of the suction of the breath of the participants _{P a, t,} that is, the suction section _{I a breath, k} of the RSP value _{R a,} the maximum value of _t.
· _{AMP a, k:} Participants _{P a} suction section _{I a breath, k} of RSP values _{R a,} the amplitude of _{t, i.e., MAX a,} k -MIN _a, value calculated by _k. This represents the amount of breath inhaled in the inhalation section _{Ia, k} .
· _{DUR a, k:} the suction section _{I a} breath of participants _{P a,} length of _k, that is, the suction section _{I a breath,} the discrete time of the start position from the discrete time _{t e} of the end position of _{k (k)} the value obtained by subtracting _{t s (k) t e (} k) -t s (k).
· _{SLO a, k:} Participants _{P a} suction section _{I a breath,} RSP value _{R a,} the average value of the slope per unit time _t in _{k, i.e., AMP a, k / DUR a} , calculated in _k Value. It represents the time change of the amount of breath inhaled in the inhalation section _{Ia, k} .
· INT1 _{a, k:} distance to the front of the suction from the end time _{t ue} of the speech segment _{U k (k) (the} end of the speech segment) of the breath of the participants _{P a} is started, ie, the suction of breath interval _{I a ,} discrete time _{t s (k)} from the speech segment _{U k} of the end time _{t ue} value obtained by subtracting the _{(k) t} s of the start position of _{k (k) -t ue (k} ). This represents the time relationship between the utterance section U _k and the suction section I _{a, k} .

The next speaker probability estimation unit 108 or the next speaker probability estimation unit 108A may further generate the following parameters INT2a _{, k} .
· INT2 _{a, k:} interval up to the speech segment _{U k + 1} of the next speaker is started from the time of the end intake of breath of the participants _{P a,} ie, the next speaker of the speech segment _{U k + 1} of the start time _{t us (k + 1 ) (} T _{) (k + 1)} −te _(k) obtained by subtracting the discrete time te _(k) at the end position of the breath inhalation interval I _{a, k} . The time relationship between the utterance section U _{k + 1} and the suction section I _{a, k} is represented. Parameters λ _'a, INT2 _a, a plus _k is denoted as parameter lambda _{a, k} to _k.

The next speaker probability estimator 108 or the next speaker probability estimator 108A obtains, for example, information representing the utterance interval U _{k + 1} , and after the parameters λ _{a, k} are obtained (the utterance interval U _{k + 1} is started). After), it is recorded in the database together with information indicating the utterance section U _k and its utterer P _uk , the utterance section U _{k + 1,} its utterer P _{uk + 1} and its utterance start timing T _{uk + 1} . The utterance timing of the next speaker P _{uk + 1} may be any time point in the utterance section U _{k + 1} or a time point corresponding thereto. The utterance timing T _{uk + 1} may be the start time t _{us (k + 1)} of the utterance interval U _{k + 1} , or the time t _{us (k + 1)} + γ (where γ is a positive or negative constant), It may be the end time t _{ue (k + 1)} of the utterance interval U _{k + 1} , may be the time t _{ue (k + 1)} + γ, or may be the central time t _{us (k + 1)} + (t _{ue ( ) of} the utterance interval U _{k + 1. k + 1)} -tus _{(k + 1)} ) / 2. Part or all of the information representing λ _{a, k} , U _k , P _uk , P _{uk + 1} , T _{uk + 1} is held in the database, and the next speaker probability estimation unit 108 or the next speaker probability estimation unit 108A performs the utterance interval U _{k + 1.} It is used to predict the next utterer later and the utterance timing.

The next-speaker probability estimating unit 108 or the next-speaker probability estimating unit 108 _{</ b} > A includes the speaker information P _uk , the utterance interval U _k , the breath intake amount of the participant Pa in the intake interval I _{a, k} , and the intake interval I _{a. ,} the length of _k, the suction section I _a, suction amount of time variation of the breath at _k, and speech periods U _k and the suction section I _a, based on at least part of the time relationship between _k, participants P _1, ..., obtain estimation information either is or is the next speaker P _{uk + 1,} and representing at least one of the following speaker P _{uk + 1} of the utterance timings of the P _a. However, subscript "uk + 1" of the _{"P uk + 1"} represents a _{u k + 1.} (If speech continues) if speaker _{P uk} speech period _{U k} performs speech even speech section _{U k + 1,} the next speaker is the same as the speaker _{P uk} speech period _{U k.} On the other hand, (if That utterance replacement) when uttered P _uk other participants in the speech period U _k performs speech even speech section U _{k + 1,} the following speaker is other than speaker P _uk speech period U _k participants It is.

The next-speaker probability estimating unit 108 or the next-speaker probability estimating unit 108 _{</ b} > A includes the speaker information P _uk , the utterance interval U _k , the breath intake amount of the participant Pa in the intake interval I _{a, k} , and the intake interval I _{a. ,} the length of _k, the suction section I _a, suction amount of time variation of the breath at _k, and section suction and speech period U _k I _a, the feature amount corresponding to at least part of the time relationship between _k f _a, A model for obtaining estimation information for _k is machine-learned, and estimation information for feature quantities is obtained using this model. Feature value _{f a, k} is the speaker information _{P uk,} speech segment _{U k,} the suction section _{I a} participant _{P a,} suction amount of breath at _k, the suction section _{I a,} the length of _k, the suction section I _It may correspond to only one of the temporal changes in the amount of inhalation of breath at _{a, k} and the time relationship between the utterance interval U _k and the inhalation interval I _{a, k} , or may correspond to a plurality of these. It may be good or all. The machine learning model, for example, past suction section _{I a, i} (although, i <k) suction of breath, the suction section _{I a,} the length of the _i, suction section _{I a,} breath in _i , The feature quantity f _{a, k} corresponding to at least a part of the temporal change in the amount of ingestion and the time relationship between the utterance section U _i and the ingestion section I _{a, i} , and the utterance sections U _i , U _{i + 1} and their speakers Information of P _uk and P _{uk + 1} is used as learning data.

The next speaker / speech timing estimation processing by the next speaker probability estimation unit 108 or the next speaker probability estimation unit 108A will be exemplified. In this example, the next speaker estimation model is a model that estimates the next speaker P _{uk + 1,} and the response timing estimation model is a model for estimating the response timing of the next speaker P _{uk + 1} is generated, using each model Thus, the next speaker P _{uk + 1} and its speech timing are estimated.

When learning the next speaker estimation model, the next speaker probability estimation unit 108 or the next speaker probability estimation unit 108A uses the past parameters λ _{a, i} (where a = 1,. And i <k), and information representing the utterance sections U _i and U _{i + 1} and the speakers P _ui and P _{ui + 1} are read out. The next speaker probability estimator 108 or the next speaker probability estimator 108A learns feature data F1 _{a, i} and U _i , U _{i + 1} , P _ui , P _{ui + 1} corresponding to at least a part of the parameters λ _{a, i.} Then, the next speaker estimation model is machine-learned. As the next speaker estimation model, for example, SVM (Support Vector Machine), GMM (Gaussian Mixture Model), HMM (Hidden Markov Model), or the like can be used.

Next speaker probability estimation unit 108 or the next speaker probability estimation unit 108A, the parameter lambda _'a, the feature amount corresponding to at least a portion of the _k F1 _a, the _k is applied to the next speaker estimation models are estimated thereby Information representing the next utterance P _{uk + 1} is a part of the “estimated information”. Note that the information indicating the next utterance P _{uk + 1} may be _a definite representation of any participant Pa or may be a probability representation. The probability that participant _{P a} becomes the next speaker, and P1 _a.

When learning the utterance timing estimation model, the next speaker probability estimation unit 108 or the next speaker probability estimation unit 108A uses the past parameters λ _{a, i} (where a = 1,. Yes, i <k), and information indicating the utterance sections U _i and U _{i + 1} and the utterers P _ui and P _{ui + 1} and the utterance start timing T _{ui + 1} of the utterance section U _{i + 1} is read. The next speaker probability estimator 108 or the next speaker probability estimator 108A includes the feature amounts F2 _{a, i} and U _i , U _{i + 1} , P _ui , P _{ui + 1} , T _{ui + 1} corresponding to at least a part of the parameters λ _{a, i.} Is used as learning data to machine-learn an utterance timing estimation model. As the next speaker estimation model, for example, SVM, GMM, HMM or the like can be used.

The next speaker probability estimation unit 108 or the next speaker probability estimation unit 108A obtains the speaker P _uk , at least a part of the parameters λ ′ _{a, k} , and the next speaker P _{uk + 1} estimated by the next speaker estimation model. Then, the feature amount F2 _{a, k} corresponding to at least a part of the parameter λ ′ _{a, k} is applied to the utterance timing estimation model. The next speaker probability estimator 108 or the next speaker probability estimator 108A applies the feature amount F2 _{a, k} to the utterance timing estimation model and utterance timing T _{uk + 1 of} the next utterance section U _{k + 1} (for example, utterance Information indicating the start time of the section U _{k + 1} ) is output as part of the “estimated information”. Note that the information representing the utterance timing may be deterministically representing any utterance timing or may be represented probabilistically. The probability that the participant _{P a} to start a speech to the time t (the probability time t is the utterance timing of the participant _{P a),} and _P2 a (t).
The next speaker probability P ^ns _i (t) at time t of the participant i estimated by the next speaker probability estimating unit 108 or the next speaker probability estimating unit 108A of the above-described embodiment is determined by the participant i being the primary speaker. If a participant _{P a} in the estimation technique, is calculated by the probability P1 _a × probability _P2 a (t).

The next-speaker probability estimating unit 108 or the next-speaker probability estimating unit 108A estimates the participant and timing to start the next utterance based on the observation value of the breathing motion. May be used.
When the gaze behavior is further used, a gaze target detection device 203 is further attached to each participant P _a (where a = 1,..., A). Gaze object detection device 203, participant P _a detects someone or gazing (gaze target), the participant P _a and gaze target G _a, next speaker information representing a _t at each discrete time t This is sent to the probability estimator 108 or the next speaker probability estimator 108A. Next speaker probability estimation unit 108 or next speaker probability estimation unit 108A receives gaze target information G _{1, t} ,..., G _{A, t} , utterance interval U _k , and speaker information P _uk , and ends the utterance interval. Before and after gaze target label information θ _{v, k} (where v = 1,..., V, V are the total number of gaze target labels) is generated. Gaze target label information is information indicating the gaze target participants in time interval corresponding to the end time T _se speech period U _k. Here, an example is shown of the gaze target label information theta _{v, k} was labeled gaze target participants P _a in the finite time interval including end time T _se. In this case, for example, deals with watching action that appeared in the interval from the speech interval _U time before the end time _{T se} of _{k T} se -T _b to the time point _T se + _{T a} subsequent to the end point _{T se.} T _{b, T a} is may be any value from 0 or more, as a guide, _{T b} is 0 seconds to 2.0 seconds, _{T a} is appropriate to about 0 seconds to 3.0 seconds.

The next speaker probability estimator 108 or the next speaker probability estimator 108A classifies the participants to be watched into the following types, and performs labeling on the eyes to be watched. Note that the symbol of the label has no meaning, and any notation may be used as long as it can be identified.
Label S: speaker (ie, representing participant P _uk who is a speaker)
Label L _ξ : Non-speaker (where ξ identifies participants who are non-speakers different from each other, and ξ = 1,..., A−1. For example, a participant is a non-speaker P _2. , non-speaker P _{3 when,} had a gaze sequentially labeled L ₁ to the non-speaker P _2, labeled L ₂ to the non-speaker P ₃ is assigned.)
・ Label X: No one is watching

When the label is S or _Lξ , information indicating whether or not mutual gaze (gaze crossing) has occurred is given. In this embodiment, when mutual gaze occurs, an _M label is _{added to} the end of the labels S and L _ξ as in S _M , L _ξM (subscript “ _ξM ” represents ξ _M ). .

FIG. 10 is a diagram illustrating a specific example of a gaze target label. FIG. 10 is an example of A = 4, and the speech sections U _k and U _{k + 1} and the gaze targets of each participant are shown in time series. In the example of FIG. 10, after the participant P ₁ speaks, an utterance change occurs and the participant P ₂ newly speaks. Here, participants P ₁ is a speaker after watching the participant P _4, gazing at the participant P _2. In the period from the time of T _se -T _b up to the point of _T se + _{T a,} when a participant _{P 1} had seen the participants _{P 2,} participants _{P 2} has seen participants _{P 1.} This indicates that what is happening is mutual gaze between the participants P ₁ and participants P _2. In this case, there are two gaze target labels L ₁ and L _2M generated from the gaze target information G _{1, t of} the participant P ₁ . In the above-mentioned period, the participants P ₂ is gazing after watching the participant P _4, the participants P ₁ is a speaker. In this case, you gaze target label participants _{P 2} is two and the _{L 1} and _{S M.} In addition, in the above-mentioned period, the participants P ₃ is gazing at the participant P ₁ is a speaker. In this case, the gaze target label of participants P ₃ is a S. In addition, in the above-mentioned period, the participants P ₄ is not anyone seen. In this case, the gaze target label of participants P ₄ is the X. Therefore, in the example of FIG. 10, V = 6.

The next speaker probability estimation unit 108 or the next speaker probability estimation unit 108A also acquires a start time and an end time for each gaze target label. Here, as a symbol indicating which gaze target label (GLε {S, S _M , L ₁ , L _1M , L ₂ , L _2M ,...) Of which (Rε {S, L}) is R _GL, the start time ST_R _GL, the end time is defined as ET_R _GL. Here, R represents the utterance state (speaker or non-speaker) of the participant, S is a speaker, and L is a non-speaker. For example, in the example of FIG. 10, the first fixation target label participants _{P 1} is _{S L1,} the start time ST_S _L1, the end time is ET_S _L1. The gaze target label information θv _{, k} is information including a gaze target label R _GL , a start time ST_R _GL , and an end time ET_R _GL .

Next speaker probability estimation unit 108 or the next speaker probability estimation unit 108A, by using the gaze target label information theta _{v, k,} gaze target transition pattern E _a of each participant P _a, generates a _k. The gaze target transition pattern is generated by generating a transition n-gram considering the temporal order using the gaze target label _RGL as a constituent element. Here, n is a positive integer. For example, considering the example of FIG. 10, the gaze target transition pattern E _{1, k} generated from the gaze target label of the participant P1 is L ₁ -L _2M . Similarly, gaze target transition pattern _{E 2} participants _{P 2, k} is _L 1 -S _M, gaze target transition patterns _{E 3, k} participants _{P 3} is S, gaze target transition patterns E participants _{P 4 4, k} becomes X.

The gaze target transition pattern E _{a, k} is, for example, after the utterance section U _{k + 1} is started, the utterance section U _k and its utterer P _uk , the next utterer P _{uk + 1} and the next utterance who perform the utterance corresponding to the utterance section U _{k + 1.} It is sent to the database together with information representing the start timing T _{uk + 1} . In the database, the gaze target transition pattern E _{a, k} is merged with the parameters λa, k, and a part or all of the information representing E _{a, k} , λ _{a, k} , U _k , P _uk , P _{uk + 1} , T _{uk + 1.} Is retained in the database.

Next speaker probability estimation unit 108 or the next speaker probability estimation unit 108A inputs the gaze target label information theta _{v, k,} and generates a time structure information theta _{v, k} for each gaze target label. The time structure information is information representing the temporal relationship of the gaze behavior of the participant, and (1) the time length of the gaze target label, (2) the interval between the gaze target label and the start time or end time of the utterance section, ( 3) An interval between the start time or end time of the gaze target label and the start time or end time of another gaze target label is used as a parameter.

Specific parameters of the time structure information are shown below. Hereinafter, the start time of the utterance section is defined as ST_U, and the end time of the utterance section is defined as ET_U.
_{· INT1 (= ET_R GL -ST_R GL} ): gazing target label _{R GL} of the start time ST_R _GL and end time ET_R interval of _{GL · INT2 (= ST_U-ST_R} GL): start time ST_R _GL of the gaze target label _{R GL} utterance How long before the start time ST_U of the section INT3 (= ET_U-ST_R _GL ): How long before the start time ST_R _GL of the gaze target label R _GL is before the end time ET_U of the speech section INT4 (= ET_R _GL -ST_U): gazing target label _{R GL} of the end time ET_R _GL Do · INT5 was after much than the start time ST_U of the speech segment (= ET_U-ET_R _GL): end time ET_R _GL is the utterance section of the gaze target label _{R GL} Than the end time ET_U of Have either · _INT6 had been before _{(= ST_R GL -ST_R GL ')} : the gaze target label _{R GL} of the start time ST_R _GL other of the gaze target label _{R GL'} of the start time ST_R _GL or was after much than _'· INT7 ( = ET_R _{GL '-ST_R GL):} gazing target label _{R GL} of the start time ST_R _GL other of the gaze target label _{R GL'} of the end time ET_R _{GL 'or} was before much than _{· INT8 (= ET_R GL -ST_R GL} ' ): gaze target label _{R GL} of the end time ET_R _GL is gazing target label _{R GL 'of} the start time ST_R _GL' or was after much than _{· INT9 (= ET_R GL -ET_R GL} '): the end of the gazing target label _{R GL} time ET_R _GL is none than the _'end time ET_R _{GL of'} gaze target label _{R GL} Did even after leprosy

Note that INT6 to INT9 are acquired for combinations with the gaze target labels of all participants. In the example of FIG. 10, since there are a total of six gaze target label information (L ₁ , L _2M , L ₁ , S _M , S, X), INT6 to INT9 each have 6 × 5 = 30 data. Generated.

The time structure information Θ _{v, k} is information including parameters INT1 to INT9 for the gaze target label information θ _{v, k} . Each of the above parameters constituting the time structure information Θ _{v, k} will be specifically described with reference to FIG. FIG. 11 is a diagram showing time structure information about the gaze target label L1 of the participant P1 (R = S) who is a speaker. That is, time structure information in R _GL = S _L1 . Note that for INT6 to INT9, only the relationship with the gaze target label L1 of the participant P2, that is, R _GL = L _L1 is shown in order to simplify the illustration. In the example of FIG. 11, it can be seen that INT1 to INT9 are obtained as follows.
INT1 = ET_S _L1 −ST_S _L1
-INT2 = ST_U-ST_S _L1
・ INT3 = ET_U-ST_S _L1
・ INT4 = ET_S _L1 −ST_U
・ INT5 = ET_U-ET_S _L1
INT6 = ST_S _L1 -ST_L _L1
INT7 = ET_L _L1 -ST_S _L1
INT8 = ET_S _L1 −ST_L _L1
INT9 = ET_S _L1 -ET_L _L1

The time structure information Θ _{v, k} is, for example, after the utterance section U _{k + 1} is started, the utterance section U _k and its utterer P _uk , the next utterer P _{uk + 1} who performs the utterance corresponding to the utterance section U _{k + 1} and the next utterance start. It is sent to the database together with information representing the timing T _{uk + 1} . In the database, the time structure information Θ _{v, k} is merged with the parameters λ _{a, k} and one piece of information representing Θ _{v, k} , λ _{a, k} , U _k , P _uk , U _{k + 1} , P _{uk + 1} , T _{uk + 1.} Parts or all are kept in the database.

The next-speaker probability estimating unit 108 or the next-speaker probability estimating unit 108A includes gaze target transition patterns E _{a, k} , time structure information Θ _{v, k} , speaker information P _uk , utterance interval U _k , and participant P _a . suction section _{I a,} suction amount of breath at _k, the suction section _{I a,} the length of _k, the suction section _{I a,} suction amount of time variation of the breath at _k, and speech periods _{U k} and the suction section _{I a,} Machine learning is performed on a model for obtaining estimation information for the feature quantity f _{a, k} corresponding to at least part of the temporal relationship with _k, and the next speaker probability P ^ns _i (estimation information for the feature quantity is used using the model. t) is obtained and output.

The next-speaker probability estimating unit 108 or the next-speaker probability estimating unit 108A described above estimates the participant and timing to start the next utterance based on the observation value of the breathing motion and the observation value of the line of sight. Information regarding the movement of the participant's head may be used. This takes advantage of the fact that people tend to crawl right before utterance. The next speaker probability estimation unit 108 or the next speaker probability estimation unit 108A analyzes the image data of each participant from the video input unit 105, and whether or not the participant has struck depending on whether or not the head has moved up and down. Determine whether. If the next speaker probability estimation unit 108 or the next speaker probability estimation unit 108A determines that the participant i has reached several seconds before the time t, the next speaker probability P ^ns _i ( A process of adding a predetermined value to t) is performed. Further, the next speaker probability estimation unit 108 or the next speaker probability estimation unit 108 </ b> A is based on at least one of the observation value of the breathing movement, the observation value of the line of sight, and the information on the movement of the participant's head. The probability P ^ns _i (t) may be calculated.

  Further, when the next speaker probability estimation unit 108 or the next speaker probability estimation unit 108A uses at least one of the observation value of the breathing movement, the observation value of the line of sight, and the movement of the participant's head, In accordance with information used by the next speaker probability estimation unit 108 or the next speaker probability estimation unit 108A, the sensor 103 is used for the position measurement device 201, the respiratory motion measurement device 202, the gaze target detection device 203, and the head motion detection device 204. Any one or more may be provided.

  Note that the robot 100 in the first embodiment and the robot 100A in the second embodiment include a microphone 101, a camera 102, a sensor 103, a voice input unit 104, a video input unit 105, a sensor input unit 106, and an utterance section detection. Although the unit 107, the next speaker probability estimation unit 108 or the next speaker probability estimation unit 108A, and the control unit 109 or the control unit 109A are built in, the present invention is not limited to this configuration. Microphone 101, camera 102, sensor 103, voice input unit 104, video input unit 105, sensor input unit 106, utterance section detection unit 107, next speaker probability estimation unit 108 (or next speaker probability estimation unit 108A) and control unit The conversation support device including at least a part of 109 (or control unit 109A) may be provided as a separate device from robot 100 (or robot 100A). The conversation support apparatus is configured to be able to communicate with the robot 100 (or the robot 100A). By transmitting a control signal from the control unit 109 (or the control unit 109A) to the robot 100 (or the robot 100A), the robot 100 (or the robot 100A) is configured. Alternatively, the utterance of the robot 100A) is controlled.

  The robot 100 and the robot 100A may be configured such that a part of the body is displayed on a display unit such as a display, and is displayed on the display unit as an agent whose whole body is a virtual person. It may be. The expression of a part of the body of the robot 100 and the robot 100A on the display unit may be, for example, a configuration in which the entire face is a display unit and a face image is displayed on the display unit. Various expressions can be performed by changing the face image displayed on the display unit. Note that the robot 100 and the robot 100A may be configured not to include the plurality of microphones 101 and the sensors 103. For example, the robot 100 and the robots 100A installed outside the robot 100 and the robot 100A may be wired or wirelessly connected. The signal transmission / reception may be possible.

  In the robot 100 according to the embodiment and the robot 100A according to the second embodiment, as long as the above-described speech control processing is not hindered, functions or the like provided by ordinary robots other than the functions shown in FIGS. May be provided. Further, the robot 100 according to the first embodiment may be configured to be able to perform an operation similar to that of a human during conversation such as a breathing operation like the robot 100A according to the second embodiment.

According to the embodiment described above, the conversation support system is, for example, a robot, and each participant is based on measurement results of non-verbal behavior such as gaze, breathing, and head movement of each participant during the conversation. The next speaker probability, which is the probability of the next utterance at an arbitrary time, is estimated. The conversation support system estimates the estimated next speaker who is the next participant to speak based on each participant's next speaker probability and the timing when the predicted next speaker starts speaking. When it is detected that the predicted next speaker does not speak at the determined timing, speech is urged with the predicted next speaker or a participant different from the predicted next speaker as the target person. The conversation support system controls the robot or the speaker displayed on the display device (agent whose body is a virtual person) so as to perform the operation indicating the transfer of the right to speak to the target person in order to promote the speech. To do. For example, a speaker displayed on a robot or a display device targets an upper limb that outputs a voice prompting the subject's utterance, or moves his eyes, head, or torso to direct his gaze or face toward the subject. Take nonverbal behaviors such as presenting to the person.
According to the above-described embodiment, the robot or the speaker displayed on the display device prompts the participant to speak by the participant who has missed the timing of the speech, thereby prompting the participant to speak. it can. In addition, it is possible to encourage the participants to speak so that the silence during the conversation is prolonged and the conversation atmosphere is not deteriorated.

  Each function part with which robot 100 or robot 100A in this embodiment mentioned above is provided is realizable with a computer, for example. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be a program for realizing a part of the above-described functions, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system. You may implement | achieve using programmable logic devices, such as FPGA (Field Programmable Gate Array).

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  The present invention can be applied to control of a robot that has a conversation with a participant, or can be applied to control of the movement of an agent (virtual person) displayed on a display device that has a conversation with a participant.

51a ... right eye, 51b ... left eye, 52 ... mouth, 53 ... head, 54 ... neck, 55 ... trunk, 55a ... right arm, 55b ... left arm, 100, 100A ... robot, 101 ... microphone, 102 ... camera, DESCRIPTION OF SYMBOLS 103 ... Sensor, 104 ... Audio | voice input part, 105 ... Image | video input part, 106 ... Sensor input part, 107 ... Speech area detection part, 108, 108A ... Next speaker probability estimation part, 109, 109A ... Control part, 110 ... Sound Control unit, 111 ... Mouth control unit, 112 ... Gaze control unit, 113 ... Head control unit, 114 ... Body control unit, 115 ... Speaker, 116 ... Mouth drive unit, 117 ... Eye drive unit, 118 ... Head drive unit, 119 ... trunk drive unit, 120 ... sensor signal conversion unit, 201 ... position measurement device, 202 ... breathing motion measurement device, 203 ... gaze target detection device, 204 ... head Operation detecting apparatus, 401 ... voice analysis unit, 402 ... conversation information generation unit, 403 ... conversation information DB, 404 ... voicing information generation unit, 405 ... sound signal generating unit, 1091,1091A ... operation pattern information storage unit

Claims (8)

Based on the measurement result of the non-verbal behavior of each participant in the conversation, a next speaker probability estimation unit that estimates a next speaker probability that is a probability that each of the participants will be the next utterance at an arbitrary time;
Based on the probability of the next speaker of the participant, a predicted next speaker who is a participant to speak next and a timing at which the predicted next speaker starts speaking are estimated, and the prediction is performed at the estimated timing. A control unit for instructing utterance with the predicted next speaker as a target when it is detected that the next speaker does not speak;
An utterance guidance unit that receives an instruction from the control unit and performs processing for prompting the subject to speak;
A conversation support system characterized by comprising: When the control unit detects that the predicted next speaker has not spoken at the estimated timing, the control unit causes the utterance guide unit to prompt the utterance with a speaker other than the next speaker as a target person. Instruct,
The conversation support system according to claim 1. The utterance guiding unit controls the robot or the speaker displayed on the display device to perform an operation indicating transfer of the utterance right to the target person,
The conversation support system according to claim 1 or 2, characterized in that The utterance guide unit controls one or more of a robot's eyes, a head, or a torso displayed on a display device to direct a line of sight toward the subject.
The conversation support system according to claim 3. The utterance guide unit controls the robot or the speaker to display the upper limb of the speaker displayed on the display device.
The conversation support system according to claim 3 or 4, characterized by the above. The utterance guiding unit outputs a voice prompting the subject to speak;
The conversation support system according to any one of claims 1 to 5, wherein Based on the measurement result of the non-verbal behavior of each participant in the conversation, a next speaker probability estimation unit that estimates a next speaker probability that is a probability that each of the participants will be the next utterance at an arbitrary time;
Based on the probability of the next speaker of the participant, a predicted next speaker who is a participant to speak next and a timing at which the predicted next speaker starts speaking are estimated, and the prediction is performed at the estimated timing. A control unit that instructs the utterance guiding unit that performs the process of prompting the utterance to detect the next speaker as the target person when detecting that the next speaker has not made the utterance;
A conversation support device comprising: On the computer,
Based on the measurement result of the non-verbal behavior of each participant in the conversation, the next speaker probability estimating step for estimating the next speaker probability, which is the probability that each of the participants will be the next utterance at an arbitrary time,
Based on the probability of the next speaker of the participant, a predicted next speaker who is a participant to speak next and a timing at which the predicted next speaker starts speaking are estimated, and the prediction is performed at the estimated timing. A control step for instructing an utterance guiding unit that performs processing for prompting an utterance to prompt an utterance with the predicted next speaker as a target when it is detected that the next speaker has not made an utterance;
Conversation support program for running.

Images