JP6480351B2 - Speech control system, speech control device and speech control program - Google Patents

Description

  The present invention relates to a speech control system, a speech control device, and a speech control program.

  BACKGROUND Conventionally, an agent system that displays an agent (virtual person) drawn by a humanoid robot and a computer graphic may have a conversation function of making conversations with a plurality of surrounding users.

  As a dialog system that interacts with a plurality of users, for example, a technique is disclosed that estimates whether or not the dialog system should speak next from the content of the user's speech, and performs a speech from the dialog system. (See Patent Document 1).

JP, 2013-6232, A

  However, the dialog system described in Patent Document 1 determines the utterance based only on the content of the user's utterance, and does not consider who the user's utterance is for. For this reason, the dialog system described in Patent Document 1 may speak by false judgment. For example, in a conversation of a plurality of users including the user A and the user B, it is assumed that the user A asks the user B "What do you think?" In this case, since the dialogue system described in Patent Document 1 can not grasp to whom the question "What do you think?" Thus, it has been difficult for conventional humanoid robots and agents to speak at appropriate timings to a plurality of surrounding users.

  In view of the above circumstances, the present invention has an object to provide a speech control system, a speech control device and a speech control program that can cause a robot or an agent talking to a plurality of users to make speech at appropriate timing. .

  According to one aspect of the present invention, there is provided an utterance control unit for controlling an utterance of a robot talking with a plurality of users or an utterance of a speaker displayed on a display device performing a conversation with a plurality of users; A second speaker estimation unit for acquiring a first next speaker probability that is a probability that the speaker will be the next speaker at any time, and the speech control unit is configured to acquire the first speaker obtained by the next speaker estimation unit. It is a speech control system which controls a speech of said robot or said speaker based on said 1st next speaker probability of a user.

  One aspect of the present invention is the speech control system, wherein the speech control unit is configured to set the robot or the speaker when the first next speaker probability of a plurality of the users is equal to or less than a predetermined threshold. Control to make them speak.

  One embodiment of the present invention is the speech control system, wherein the speech control unit is configured such that an integral value obtained by integrating the first next speaker probability of a plurality of the users for a predetermined time is equal to or less than a predetermined threshold. Control to cause the robot or the speaker to speak.

  One embodiment of the present invention is the speech control system, wherein the next speaker estimation unit acquires the first next speaker probability based on the non-verbal behavior of the user.

  One embodiment of the present invention is the speech control system described above, wherein the robot or the speaker can be controlled to perform an operation corresponding to the non-verbal behavior, and the next speaker estimation unit is configured to A second next speaker probability, which is a probability that the robot or the speaker will be the next speaker, is obtained at the time based on the information on the motion corresponding to the non-verbal behavior of the robot, and the utterance control unit Controlling the speech of the robot or the speaker based on a comparison of the first next speaker probability of the plurality of users and the second next speaker probability of the robot or the speaker .

  One aspect of the present invention is the speech control system described above, wherein the speech control unit acquires a speech necessity degree which is an index indicating the necessity of speech of the robot or the speaker, and acquires the speech The value of the second next speaker probability acquired by the next speaker estimation unit is changed according to the degree of necessity.

  According to one aspect of the present invention, there is provided an utterance control unit for controlling an utterance of a robot talking with a plurality of users or an utterance of a speaker displayed on a display device performing a conversation with a plurality of users; A second speaker estimation unit for acquiring a first next speaker probability that is a probability that the speaker will be the next speaker at any time, and the speech control unit is configured to acquire the first speaker obtained by the next speaker estimation unit. It is an utterance control device which controls an utterance of the robot or the speaker based on the first next speaker probability of the user.

  One embodiment of the present invention is a speech control program for controlling a speech of a robot talking with a plurality of users, or a speech of a speaker displayed on a display device talking with a plurality of users, A next speaker estimation step of acquiring a first next speaker probability that is a probability that the user will be the next speaker at any time; and the first of the user acquired in the next speaker estimation step A speech control program for causing a computer to execute a speech control step of controlling speech of the robot or the speaker based on the next speaker probability.

  According to the present invention, a robot or an agent who talks with a plurality of users can make speech occur at an appropriate timing.

It is a figure showing an outline of functional composition with which robot 100 in a 1st embodiment is provided. It is a figure which shows the specific structural example of the sensor 103 in 1st Embodiment. It is a diagram illustrating an example of the next speaker probability P ^{ns i} output by the next speaker estimation unit 108 _^(t) in the first embodiment. It is a figure which shows the specific example of the detail of a structure of the sound control part 110 in 1st Embodiment. It is a figure which shows the specific example of the external appearance and structure of the robot 100 in 1st Embodiment. FIG. 5 is a flow chart showing the operation of the robot 100 in the first embodiment. It is a figure showing an outline of functional composition with which robot 100A in a 2nd embodiment is provided. It is a figure which shows the specific example of the detail of a structure of control part 109A in 2nd Embodiment. It is a figure which shows the specific example of the detail of a structure of the sound control part 110A in 2nd Embodiment. It is a figure which shows the specific example of the pre-talk operation | movement of the robot 100A in 2nd Embodiment. It is a flow figure showing conversation operation of robot 100A in a 2nd embodiment. It is a figure which shows the example of the breathing area of breath. It is a figure which shows the specific example of a gaze object label. It is a figure which shows the time structure information about gaze object label L1 of user P1 (R = S) who is a speaker.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
First Embodiment
FIG. 1 is a diagram showing an outline of a functional configuration provided in a robot (utterance control system) 100 in the first embodiment. The robot 100 in the first embodiment is a robot that talks with a plurality of users. As shown in FIG. 1, the robot 100 includes a microphone 101, a camera 102, a sensor 103, an audio input unit 104, an image input unit 105, a sensor input unit 106, an utterance period detection unit 107, and a next talk. A person estimation unit 108, an utterance control unit 109, a sound control unit 110, and a speaker 115 are provided.

  The microphone 101 collects sounds around the robot 100 including the voice of the user who is conversing and the like, and outputs a sound signal including an audio signal (hereinafter referred to simply as an audio signal). The microphone 101 is configured of a plurality of microphones attached to each of a plurality of users. The microphone 101 may be mounted on the robot 100 itself. The microphone 101 is not limited to the configuration mounted on the user or the robot 100, and may be disposed around the user or the robot 100. The microphone 101 is preferably disposed at a position closer to the user's mouth, but any position and any number may be disposed as long as at least the user's voice can be collected. In the robot 100, the plurality of microphones 101 and the voice input unit 104 are configured to be able to transmit and receive voice signals by wire or wirelessly.

  The camera 102 captures an image of a user who has a conversation, and outputs an image signal. The camera 102 is preferably an imaging device having a wide angle of view so that all the users are in the angle of view. Further, the cameras 102 may be a plurality of cameras corresponding to the number of users who respectively capture the images of all the users. The camera 102 may be configured at any position and any number as long as at least all the users can be photographed. In this case, in the robot 100, the video input unit 105 and the plurality of cameras are connected to be able to transmit and receive video signals by wire or wirelessly.

  The sensor 103 includes a first sensor that measures the position of the user who talks, a second sensor that measures the breathing motion of the user, a third sensor that detects the gaze target of the user, and a head motion of the user And a plurality of sensors such as a fourth sensor that detects the sensor signal, and outputs a sensor signal from each of the sensors to the sensor input unit 106.

FIG. 2 is a view showing a specific configuration example of the sensor 103 in the first embodiment.
As shown in FIG. 2, the sensor 103 measures the position of the user who talks (the first sensor) 201, and the respiratory movement measurement device (the second sensor) which measures the breathing movement of the user. 202, a gaze target detection device (third sensor) 203 for detecting a gaze object of a user, and a head movement detection device (fourth sensor) 204 for detecting a head movement of the user. The position measurement device 201 is installed, for example, in the robot 100, and the respiratory movement measurement device 202, the gaze target detection device 203, and the head movement detection device 204 are mounted on the head of the user or the like. The position measurement device 201 is connected to the sensor input unit 106. The respiratory movement measurement device 202, the gaze target detection device 203, and the head movement detection device 204 are connected to the sensor input unit 106 so as to be able to transmit and receive sensor signals in a wired or wireless manner.

  The voice input unit 104 receives a voice signal from the microphone 101, and outputs a voice signal to the speech zone detection unit 107, the next speaker estimation unit 108, and the sound control unit 110. The audio input unit 104 performs processing such as converting an audio signal from the microphone 101 into an audio signal in a signal format that can be processed in the robot 100. The video input unit 105 receives the video signal from the camera 102 and outputs the video signal to the next speaker estimation unit 108. The video input unit 105 performs processing such as converting a video signal from the camera 102 into a video signal in a signal format that can be processed in the robot 100. The sensor input unit 106 receives the sensor signal from the sensor 103 and outputs the sensor signal to the next speaker estimation unit 108. The sensor input unit 106 performs processing such as converting a sensor signal from the sensor 103 into a sensor signal in a signal format that can be processed in the robot 100.

The speech zone detection unit 107 sets an arbitrary window width based on the speech signal from the speech input unit 104 and sets the power, the number of zero crossings, the frequency, etc. of the speech signal in the zone to a value indicating the feature of speech. It is calculated as an audio feature quantity. The speech zone detection unit 107 detects a speech zone by comparing the calculated speech feature amount with a predetermined threshold. The speech zone detection unit 107 outputs speech zone information that is information related to the detected speech zone to the next speaker estimation unit 108 and the sound control unit 110. In the voice signal acquired from the microphone 101, a voice activity detection (VAD) technique for automatically detecting a section where speech is present (a speech section) and a section where speech is not present (a non-speech section) is described below. As shown in Document 1, this is a known technique. The speech zone detection unit 107 detects a speech zone using a known VAD technique.
Reference 1: Hiroshi Sawada, 4 others, "Speech segment detection with a large number of microphones-a case with pin microphones", Spring Meeting of the Acoustical Society of Japan, pp. 679-680, March 2007

The next speaker estimation unit 108 includes the audio signal from the audio input unit 104, the video signal from the video input unit 105, the sensor signal from the sensor input unit 106, and the speech segment information from the speech segment detection unit 107. The next speaker probability is output, which is the probability that each user will be the next speaker at time t. The next speaker estimation unit 108 acquires, based on the audio signal, the video signal, the sensor signal, and the utterance period information, the speaker information indicating the utterer of the utterance period specified by the utterance period information. The next speaker estimation unit 108 determines a next speaker probability P ^ns _i which is a probability that each user i will be the next speaker at time t based on the audio signal, the video signal, the sensor signal, and the acquired speaker information. t) is calculated and output to the speech control unit 109. The next speaker estimation unit 108 calculates the next speaker probability P ^ns _i (t) based on the non-verbal behavior of the user. That is, in the calculation of the next speaker probability P ^ns _i (t) in the next speaker estimation unit 108, it is not necessary to analyze the content of the user's speech and obtain information on the user's speech behavior.

FIG. 3 is a diagram showing an example of the next speaker probability P ^ns _i (t) output by the next speaker estimation unit 108 in the first embodiment. FIG. 3 shows a change example of the next speaker probability P ^ns _i (t) after time t _{bue at} which the user A's utterance _{ends at} four users A to _D. The rectangle given the reference numeral 31 indicates the utterance section of the user A. The speech section 31 ends at the speech end time t _bue . _A dotted line indicated by the next speaker probability P ^ns _A (t) 32 indicates a change in the next speaker probability at time t after the speech end time t _bue of the user A. A dotted line indicated by the next speaker probability P ^ns _B (t) 33 indicates a change in the next speaker probability at time t after the speech end time t _bue of the user B. A dotted line indicated by the next speaker probability P ^ns _C (t) 34 indicates a change in the next speaker probability at time t after the speech end time t _bue of the user C. The dotted line indicated by the next speaker probability P ^ns _D (t) 35 indicates a change in the next speaker probability at time t after the speech end time t _bue of the user D. As described above, the next speaker estimation unit 108 calculates a change in the next speaker probability P ^ns _i (t) at time t after the speech end time t _bue of the user i. The details of the estimation process of the next speaker in the next speaker estimation unit 108 will be described later.

  The speech control unit 109 receives the next speaker probability from the next speaker estimation unit 108, and outputs a speech control signal to the sound control unit 110. The speech control unit 109 outputs a speech control signal for controlling whether to cause the robot 100 to make a speech based on the next speaker probability of each user from the next speaker estimation unit 108. For example, if the next speaker probability of any one of the users is equal to or higher than the threshold, the speech control unit 109 does not cause the robot 100 to utter, and if the next speaker probability of all users is equal to or lower than the threshold, A speech control signal is output to control to cause speech. Thus, the robot 100 can start speaking at a timing when the probability of the start of speaking by another user is low. Thus, the robot 100 can speak at more appropriate timing when talking with a plurality of users.

  Specifically, the speech control unit 109 controls the speech of the robot 100 using any of the four control methods of the first control method to the fourth control method described below. In the following description, a case in which four users A, B, C, and D and the robot 100 have a conversation will be described.

(First control method)
The speech control unit 109 determines the next speaker probability P ^ns _i (t), ( _i ∈ {A, B, C, D}) of the users A to D at time t acquired from the next speaker estimation unit 108, The first threshold value is compared with an arbitrary probability P _α (hereinafter referred to as a first comparison). According to the first comparison, the speech control unit 109 determines that the next speaker probability P ^ns _i (t) of any one of the users A to D is equal to or higher than the probability P _α (P ^ns _i (t) P P _α If it is determined that the robot 100 is determined to be), an utterance control signal for controlling the robot 100 not to make an utterance is output. According to the first comparison, the utterance control unit 109 determines that the next speaker probability P ^ns _i (t) of all the users A to D is less than the probability P _α (P ^ns _i (t) <P _α ) When it is determined, an utterance control signal for controlling the robot 100 to make an utterance at time t is output.

(Second control method)
The speech control unit 109 calculates the next speaker probability P ^ns _i (t), ( _i A {A, B, C, D}) of the users A to D at time t acquired from the next speaker estimation unit 108, Integration is performed for a predetermined time (for example, a time of 3 to 4 seconds or more) at time t to obtain an integral value P ^ns _i . The speech control unit 109 compares the integral value P ^ns _i with an arbitrary probability P _β which is a second threshold (hereinafter, referred to as a second comparison). If the speech control unit 109 determines that the integral value P ^ns _i of any one of the users A to D is greater than or equal to the probability P _β (P ^ns _i P P _β ) according to the second comparison, the robot A speech control signal for controlling not to make 100 speak is output. If the speech control unit 109 determines that the integral value P ^ns _i of all the users A to D is less than the probability P _β (P ^ns _i <P _β ) according to the second comparison, the robot 100 A speech control signal is output to control to cause speech at time t.

(Third control method)
The speech control unit 109 uses the next speaker probability P ^ns _i (t), ( _i ∈ {A, B, C, D}) of the users A to D at time t acquired from the next speaker estimation unit 108 Te, to get an added value obtained by adding the following speaker probability of use everyone _{^{(P ns a (t) +}} P ns B (t) + P ns C (t) + P ns D (t)). Speech control unit 109, compares the sum value _{^{(P ns A (t) +}} P ns B (t) + P ns C (t) + P ns D (t)), and any probability P _gamma is a third threshold value (Hereinafter referred to as the third comparison). According to the third comparison, the speech control unit 109 adds the value (P ^ns _A (t) + P ^ns _B (t) + P ^ns _C (t) + P ^ns _D (t)) of one of the users A to _D. ) is the case where it is determined to be equal to or greater than the probability _{^{P γ ((P ns a (}} t) + P ns B (t) + P ns C (t) + P ns D (t)) ≧ P γ), not speaking the robot 100 Output an utterance control signal to control the Speech control unit 109, by the third comparison, the sum of all of the user _{^{A~D (P ns A (t)}} + P ns B (t) + P ns C (t) + P ns D (t)) is When it is determined that the probability is less than P _γ ((P ^ns _A (t) + P ^ns _B (t) + P ^ns _C (t) + P ^ns _D (t)) <P _γ ), the robot 100 speaks at time t It outputs a speech control signal to control to make it

(4th control method)
The speech control unit 109 calculates the next speaker probability P ^ns _i (t), ( _i A {A, B, C, D}) of the users A to D at time t acquired from the next speaker estimation unit 108, Integration is performed for a predetermined time (for example, a time of 3 to 4 seconds or more) at time t to obtain an integral value P ^ns _i . Speech control unit 109 acquires the integrated value sum value obtained by adding the ^{P ns} _i of the user all the _{^{(P ns A + P ns B}} + P ns C + P ns D). Speech control unit 109 compares the sum value _{^{(P ns A + P ns B}} + P ns C + P ns D), and any probability P _theta is the fourth threshold value (hereinafter,. Fourth of comparison). According to the fourth comparison, the speech control unit 109 determines that the sum (P ^ns _A + P ^ns _B + P ^ns _C + P ^ns _D ) of any one of the users A to _D has a probability P _θ or more ((P ^ns If it is determined that the _{^{a + P ns B + P ns}} C + P ns D) ≧ P θ), and outputs a speech control signal for controlling so as not to utterance robot 100. Speech control unit 109, by the fourth comparison, the sum of all of the user _{^{A~D (P ns A + P ns}} B + P ns C + P ns D) is less than the probability _{^{P θ ((P ns A +}} P If it is determined that ^ns _B + P ^ns _C + P ^ns _D ) <P _θ ), a speech control signal is output to control the robot 100 to make speech at time t.

The next speaker probability P ^ns _i (t), ( _i A {A, B, C, D}) often has a peak after a predetermined time from the end of the speech, as shown in FIG. Therefore, in the first control method to the fourth control method, the speech control unit 109 provides a window width including the time t for obtaining the next speaker probability P ^ns _i (t), and the next speaker within the window width The maximum value of the probability may be output as the next speaker probability P ^ns _i (t) at time t. Further, in the first control method to the fourth control method, the speech control unit 109 provides a window width including the time t for obtaining the next speaker probability P ^ns _i (t), and the next speaker in the window width When there are a plurality of peaks in the probability, the next speaker probability of the n-th (n is an integer of 1 or more) peak may be output as the next speaker probability P ^ns _i (t) at time t. .

  The sound control unit 110 transmits a sound signal to the speaker 115 based on the voice signal from the voice input unit 104, the speech segment information from the speech segment detection unit 107, and the speech control signal from the speech control unit 109. Output. The sound control unit 110 determines whether to cause the robot 100 to make a speech based on the speech control signal. When the sound control unit 110 determines that the robot 100 is made to speak based on the speech control signal, the sound control unit 110 generates conversation information including conversation contents (words) to be uttered by the robot 100, and generates the conversation information generated. Output a sound signal based on it. The sound control unit 110 analyzes the contents of the user's conversation based on, for example, the voice signal and the utterance interval information, and generates conversation information for causing the robot 100 to speak based on the analysis result.

Here, the details of the configuration of the sound control unit 110 in the first embodiment will be described by way of an example.
FIG. 4 is a diagram showing a specific example of the details of the configuration of the sound control unit 110 in the first embodiment. The sound control unit 110 includes a voice analysis unit 401, a conversation information generation unit 402, a conversation information DB (database) 403, an utterance information generation unit 404, and a sound signal generation unit 405.

  The conversation information DB 403 stores conversation sample information for causing the robot 100 to make a conversation. The conversation sample information, noun often used in everyday conversation, "Hello" greeting and "Thank you" such as, it is the information that contains the phrase of the speech signal that frequently used in everyday conversation, such as "Are you okay?" .

  The voice analysis unit 401 analyzes the voice signal based on the voice signal from the voice input unit 104 and the speech segment information from the speech segment detection unit 107, identifies the content (word), and analyzes the analysis result. Output.

  The conversation information generation unit 402 generates conversation information to be the speech content of the robot 100 based on the analysis result of the speech analysis unit 401. The conversation information generation unit 402 acquires, from the conversation information DB 403, conversation sample information according to the content of conversation based on the analysis result of the speech analysis unit 401. The conversation information generation unit 402 generates conversation information based on the acquired conversation sample information. In response to the request for speech information from the speech information generation unit 404, the speech information generation unit 402 generates speech information and outputs the speech information to the speech information generation unit 404.

  The utterance information generation unit 404 receives the conversation information from the conversation information generation unit 402 and the utterance control information from the utterance control unit 109, and outputs an utterance signal. The utterance information generation unit 404 requests conversation information to the conversation information generation unit 402 based on the utterance control information from the utterance control unit 109. The utterance information generation unit 404 generates an utterance signal to be uttered by the robot 100 based on the conversation information acquired from the conversation information generation unit 402 in response to the request and the speech control information from the speech control unit 109. The utterance information generation unit 404 outputs the generated utterance signal to the sound signal generation unit 405.

  The sound signal generation unit 405 receives the speech signal from the utterance information generation unit 404, and outputs a sound signal. The sound signal generation unit 405 generates a sound signal to be uttered from the speaker 115 based on the utterance signal from the utterance information generation unit 404, and outputs the sound signal to the speaker 115.

  FIG. 5 is a view showing a specific example of the appearance and configuration of the robot 100 in the first embodiment. The robot 100 in the first embodiment has, for example, an appearance shown in FIG. 5 and a functional configuration shown in FIG.

  As shown in FIG. 5, the robot 100 is, for example, a humanoid robot (humanoid robot) having a shape that models a human upper body. The robot 100 has at least a speech function for speech, a speech recognition function for recognizing human speech, and a camera function for photographing a user. The robot 100 includes a head 53 having a face on which the right eye 51 a and the left eye 51 b and the mouth 52 are disposed.

  The robot 100 includes a neck 54 supporting the head 53 and a body 55 supporting the neck 54. The trunk 55 is provided with a right arm 55a and a left arm 55b at the upper side. In addition, a camera 102 is installed between the right eye 51 a and the left eye 51 b of the head 53. In the following description, when the right eye 51a and the left eye 51b are described collectively, they are referred to as the eye 51.

  Of the configuration shown in FIG. 1, only the camera 102 is shown in FIG. 5, so an example of the installation position of the configuration shown in FIG. 1 other than the camera 102 will be described. The microphone 101 and the sensor 103 are installed at an arbitrary position in the trunk 55 of the robot 100 or at a position distant from the trunk 55 (for example, the position of the user). The configuration other than the microphone 101, the camera 102, and the sensor 103 shown in FIG. 1 is installed inside the robot 100, and for example, the speaker 115 is installed inside the mouth 52 shown in FIG.

Next, the operation of the robot 100 in the first embodiment will be described.
FIG. 6 is a flow chart showing the operation of the robot 100 in the first embodiment. The process shown in FIG. 6 is a process performed when the robot 100 starts an operation of talking with a plurality of users.

  The audio input unit 104 receives an audio signal from the microphone 101, the video input unit 105 receives a video signal from the camera 102, and the sensor input unit 106 receives a sensor signal from the sensor 103 (step S101). The speech zone detection unit 107 calculates a speech feature based on the speech signal from the speech input unit 104, and compares the calculated speech feature with a predetermined threshold to detect a speech zone (step S102).

The next speaker estimation unit 108 determines a next speaker probability P ^ns _i which is a probability that each user i will be the next speaker at time t based on the audio signal, the video signal, the sensor signal, and the acquired speaker information. t) is calculated (step S103). The speech control unit 109 outputs a speech control signal for controlling whether to cause the robot 100 to make a speech based on the next speaker probability of each user from the next speaker estimation unit 108 (step S104).

  The sound control unit 110 determines, based on the speech control signal from the speech control unit 109, whether to cause the robot 100 to make a speech (step S105). Here, if it is determined that the robot 100 can not make a speech (NO in step S105), the process returns to step S101. When it is determined that the robot 100 is made to speak (YES in step S105), the sound control unit 110 generates conversation information for causing the robot 100 to utter, and a speaker based on the generated conversation information is used as a speaker It outputs to 115 (step S106). Thereby, the robot 100 produces an utterance corresponding to the sound signal from the speaker 115.

  The sound control unit 110 determines whether to end the speech of the robot 100 based on the speech control signal from the speech control unit 109 (step S107). Here, when the speech of the robot 100 is not ended (NO in step S107), the sound control unit 110 returns to the process of step S106. When ending the speech of the robot 100 (YES in step S107), the sound control unit 110 stops the output of the sound signal in response to the stop of the generation of the conversation information.

  Next, the robot 100 determines whether to end the conversation operation for talking with a plurality of users (step S108). Here, if it is determined that the conversation operation is not ended (NO in step S108), the process returns to step S101. When it is determined that the conversation operation is to be ended (YES in step S108), the robot 100 ends the conversation operation. For example, when the user turns on the power switch (not shown) or turns on the switch in conversation mode (not shown), the robot 100 starts the conversation operation and the user turns off the power switch. The robot 100 ends the conversation operation at the timing when the conversation mode switch is turned off.

  As described above, when the robot 100 according to the first embodiment talks with a plurality of users, it is less likely that each user will speak based on the next speaker probability of each user. Utterance can be made at appropriate timing. As a result, the robot 100 can perform an utterance with reduced occurrence of an utterance collision in which the timing of the utterance overlaps with that of another user. In addition, the robot 100 in speech can end the speech when it is likely that one of the users will speak based on the next speaker probability of each user.

Second Embodiment
The robot 100A in the second embodiment will be described. The robot 100A in the second embodiment differs from the robot 100 in the first embodiment in that the next speaker probability P ^{ns of the} robot 100A itself from the movement (breathing motion, sight line motion, head motion) of the robot 100A itself _The point at which _R (t) is determined, and the point at which the robot 100A's speech is controlled based on the determined next speaker probability P ^ns _R (t) and the next speaker probability of other users.

  In addition, the robot 100A in the second embodiment is different in that it performs the same movement as a human being in a conversation during a conversation. During a conversation, the robot 100A performs a breathing motion that emits a breathing sound or changes the swelling of the chest, a gaze motion such as directing a gaze to the current speaker, and a head motion that crawls according to the conversation.

  FIG. 7 is a diagram schematically showing the functional configuration of the robot 100A according to the second embodiment. The robot 100A in the second embodiment shown in FIG. 7 includes the same components as the robot 100 in the first embodiment. Therefore, in the description of the robot 100A, the same components as those of the robot 100 in the first embodiment will be assigned the same reference numerals and descriptions thereof will be omitted.

  As shown in FIG. 7, the robot 100A includes the microphone 101, the camera 102, the sensor 103, the voice input unit 104, the video input unit 105, the sensor input unit 106, the speech section detection unit 107, and the next talk. The person estimation unit 108A, the control unit 109A, the sound control unit 110A, the mouth control unit 111, the sight control unit 112, the head control unit 113, the trunk control unit 114, the speaker 115, the mouth unit A drive unit 116, an eye unit drive unit 117, a head drive unit 118, and a body drive unit 119 are provided.

  The next speaker estimation unit 108 A includes an audio signal from the audio input unit 101, a video signal from the video input unit 102, a sensor signal from the sensor input unit 106, and speech period information from the speech period detection unit 107. The pseudo sensor signal from the control unit 109A is input, and the next speaker probability that is the probability that each user and the robot 100A will be the next speaker at time t is output. The pseudo sensor signal is a sensor signal output from the sensor 103 when it is assumed that the robot 100A is operated based on the operation control signal generated by the control unit 109A and the operation of the robot 100A is detected by the sensor 103. is there.

The next speaker estimation unit 108A acquires, based on the audio signal, the video signal, the sensor signal, and the utterance period information, the speaker information indicating the utterer of the utterance period specified by the utterance period information. The next speaker estimation unit 108A is a probability that the robot 100A and each user i will be the next speaker at time t based on the audio signal, the video signal, the sensor signal, the pseudo sensor signal, and the acquired speaker information. The speaker probability P ^ns _i (t) is calculated and output to the control unit 109A. The next speaker estimation unit 108A outputs the speaker information and the position information of the user to the control unit 109A, in addition to the next speaker probability P ^ns _i (t).

  The next speaker estimation unit 108A may acquire the user's position information, for example, based on the sensor signal obtained by measuring the position of the user from the sensor 103, or may acquire it based on the video signal. Alternatively, the position of the user from the sensor 103 may be acquired based on the measured sensor signal and video signal.

The control unit 109A receives the next speaker probability P ^ns _i (t) from the next speaker estimation unit 108A, the speaker information and the user's position information, and outputs a speech control signal and an operation control signal. The control unit 109A generates and generates a speech control signal for controlling whether or not the robot 100A is made to speak based on the robot 100A from the next speaker estimation unit 108A and the next speaker probability of each user. The speech control signal is output to the sound control unit 110A. The control unit 109A generates an operation control signal based on the generated speech control signal, the speaker information, and the position information of the user, and the generated operation control signal is transmitted to the mouth control unit 111 and the gaze control unit 112. , And the head control unit 113 and the torso control unit 114.

Control unit 109A generates a pseudo sensor signal based on the operation control signal, and outputs the generated pseudo sensor signal to next speaker estimation section 108A. Thereby, the next speaker estimation unit 108A can estimate the next speaker probability P ^ns _R (t) that is the probability that the robot 100A will be the next speaker at time t. In addition, although the control part 109A in 2nd Embodiment is a structure which produces | generates a pseudo sensor signal, it is not restricted to this structure. For example, the control unit 109A may be configured to have a function of generating a pseudo sensor signal from the operation control signal of the robot 100A, without having a function of generating a pseudo sensor signal. Further, the next speaker estimation unit 108A may be configured to estimate the next speaker probability of the robot 100A based on the operation control signal of the robot 100A.

Control unit 109A outputs a speech control signal for controlling the speech of robot 100A according to the relationship between the next speaker probability P ^ns _R (t) of robot 100A and the next speaker probability of all users. For example, when the next speaker probability P ^ns _R (t) of the robot 100A is a value equal to or less than the next speaker probability of all users, the control unit 109A controls the speech control signal to prevent the robot 100A from making speech. Output. If the next speaker probability P ^ns _R (t) of the robot 100A is a value larger than the next speaker probability of all users, the control unit 109A outputs an utterance control signal that causes the robot 100A to utter. . As a result, the robot 100A can start speaking at a timing when its next speaker probability is high and the next speaker probability among other users is low. Thus, the robot 100 can speak at more appropriate timing when talking with a plurality of users.

  Specifically, control unit 109A controls the speech of robot 100A using any of the following four control methods of the fifth control method to the eighth control method. In the following description, a case where four users A, B, C, and D and R who is the robot 100A have a conversation will be described.

(Fifth control method)
From control unit 109A, control unit 109A calculates next speaker probability P ^ns _i (t), ( _i D {A, B, C, D) of users A to D and robot 100A R at time t. , R}). Control section 109A calculates next speaker probability P ^ns _R (t) of robot 100A, and next speaker probability P ^ns _A (t), P ^ns _B (t), P ^ns _C (t) of users A to D. , P ^ns _D (t), and when it is determined that P ^ns _R (t) is maximum, an utterance control signal is output to control the robot 100A to utter at time t. When it is determined that P ^ns _R (t) is not the maximum in the above comparison, the control unit 109A outputs an utterance control signal that controls the robot 100A not to utter.

(Sixth control method)
Control unit 109A determines next speaker probability P ^ns _i (t) of robot 100A and users A to D at time t acquired from next speaker estimation unit 108A, (i∈ {A, B, C, D, R). }) Is integrated with respect to time t for a predetermined time (for example, a time of 3 to 4 seconds or more) to obtain an integral value P ^ns _i . The control unit 109A compares the integral value P ^ns _R of the robot 100A with the integral values P ^ns _A , P ^ns _B , P ^ns _C and P ^ns _{D of} the users A to _D, and P ^ns _R is at maximum. If it is determined that there is an utterance control signal, the utterance control signal is output to control the robot 100A to utter at time t when P ^ns _R (t) is maximum. When it is determined in the above comparison that P ^ns _R is not the maximum, the control unit 109A outputs an utterance control signal that controls the robot 100A not to make an utterance.

(Seventh control method)
From control unit 109A, control unit 109A calculates next speaker probability P ^ns _i (t), ( _i D {A, B, C, D) of users A to D and robot 100A R at time t. , R}). Control unit 109A acquires an added value obtained by adding the following speaker probability All users _{^{(P ns A (t) +}} P ns B (t) + P ns C (t) + P ns D (t)). Control unit 109A includes an addition value _{^{(P ns A (t) +}} P ns B (t) + P ns C (t) + P ns D (t)), the constant to the next speaker probability ^P _ns R of the robot 100A (t) Compare with P ^ns _R (t) · ι multiplied by し た (ι is an arbitrary constant that becomes a positive value). If the control unit 109A determines that (P ^ns _A (t) + P ^ns _B (t) + P ^ns _C (t) + P ^ns _D (t)) P P ^ns _R (t) · 発 話, the robot 100A speaks to the robot 100A It outputs an utterance control signal that controls not to be caused. If the control unit 109A determines that (P ^ns _A (t) + P ^ns _B (t) + P ^ns _C (t) + P ^ns _D (t)) <P ^ns _R (t) · ι, the robot 100A determines the time A speech control signal is output to control to make the speech at t. Note that the constant ι may be, for example, the same value as the number of users, or may be a larger value if it is desired to increase the speech opportunity of the robot 100A, or may be a smaller value if the speech of the robot 100A is to be conservative. .

(Eighth control method)
Control unit 109A determines next speaker probability P ^ns _i (t) of robot 100A and users A to D at time t acquired from next speaker estimation unit 108A, (i∈ {A, B, C, D, R). }) Is integrated with respect to time t for a predetermined time (for example, a time of 3 to 4 seconds or more) to obtain an integral value P ^ns _i . Control unit 109A obtains the integral value sum value obtained by adding the ^{P ns} _i of the user all the _{^{(P ns A + P ns B}} + P ns C + P ns D). The utterance control unit 109 compares the addition value (P ^ns _A + P ^ns _B + P ^ns _C + P ^ns _D ) with P ^ns _R · ζ obtained by multiplying the integral value P ^ns _R of the robot 100A by a constant ζ (ζ is Any constant that is positive). When it is determined that (P ^ns _A + P ^ns _B + P ^ns _C + P ^ns _D ) ≧ P ^ns _R · ζ, the control unit 109A outputs an utterance control signal that controls the robot 100A not to speak. If the control unit 109A determines that (P ^ns _A + P ^ns _B + P ^ns _C + P ^ns _D ) <P ^ns _R · ζ, the control unit 109A causes the robot 100 to speak at time t when the P ^ns _R (t) becomes maximum. It outputs an utterance control signal to be controlled.

The next speaker probability P ^ns _i (t), ( _i ∈ {A, B, C, D, R}) often has a peak after a predetermined time from the end of the speech, as shown in FIG. Therefore, in the fifth control method to the eighth control method, control unit 109A provides a window width including time t for obtaining next speaker probability P ^ns _i (t), and the next speaker probability in the window width The maximum value of may be output as the next speaker probability P ^ns _i (t) at time t. Further, in the fifth control method to the eighth control method, control unit 109A provides a window width including time t for obtaining next speaker probability P ^ns _i (t), and the next speaker probability in the window width When there are a plurality of peaks, the next speaker probability of the nth (n is an integer of 1 or more) peak may be output as the next speaker probability P ^ns _i (t) at time t.

  The sound control unit 110A outputs a sound signal to the speaker 115 based on the voice signal from the voice input unit 104, the speech segment information from the speech segment detection unit 107, and the speech control signal from the control unit 109A. Do. The sound control unit 110A determines whether to cause the robot 100A to make a speech based on the speech control signal. When sound control unit 110A determines that robot 100A is to make a speech based on the speech control signal, sound control unit 110A generates conversation information including conversation contents (words) to be uttered by robot 100A, and generates the generated conversation information. Output a sound signal based on it. The sound control unit 110A analyzes the contents of the user's conversation based on, for example, the voice signal and the utterance interval information, and generates conversation information for making the robot 100A speak based on the analysis result.

  The mouth control unit 111 outputs a mouth driving signal to the mouth driving unit 116 based on the operation control signal from the control unit 109A. The gaze control unit 112 outputs an eye part drive signal to the eye part drive unit 117 based on the operation control signal from the control unit 109A. Head control unit 113 outputs a head drive signal to head drive unit 118 based on the operation control signal from control unit 109A. The body control unit 114 outputs a body driving signal to the body driving unit 119 based on the operation control signal from the control unit 109A.

  The appearance of the robot 100A in the second embodiment is the same as the robot 100 shown in FIG. Here, with reference to FIG. 2, the arrangement of the mouth drive unit 116, the eye drive unit 117, the head drive unit 118, and the trunk drive unit 119 included in the robot 100A will be described and objects to be driven. The head 23 includes an eye driving unit 117 that moves the black eyes (visual lines) of the right eye 21 a and the left eye 21 b, and a mouth driving unit 116 that opens and closes the mouth 22.

  The neck 24 includes a head drive unit 118 that performs a predetermined movement (for example, a motion to turn on or turn the face) relative to the head 23 and supports the head 23. The torso 25 includes a torso drive 119 that moves the shoulders and inflates the chest as if breathing. The mouth driving unit 116 opens and closes the mouth 22 of the robot 100A based on the mouth driving signal from the mouth control unit 111. The eye part driving unit 117 controls the direction of the black eye in the eye part 21 of the robot 100A (= the direction of the line of sight of the robot 100) based on the eye part driving signal from the line of sight control unit 112.

  The head drive unit 118 controls the movement of the head 23 of the robot 100A based on the head drive signal from the head control unit 113. The body drive unit 119 controls the shape of the body 25 of the robot 100 based on the body drive signal from the body control unit 114. The body driving unit 119 also controls the movement of the right arm 25 a and the left arm 25 b of the robot 100 based on the body driving signal from the body control unit 114.

The details of the configuration of the control unit 109A in the second embodiment will be described by way of an example.
FIG. 8 is a diagram showing a specific example of details of the configuration of the control unit 109A in the second embodiment. The control unit 109A includes an utterance control unit 301, an operation pattern information storage unit 302, an operation control unit 303, and a sensor signal conversion unit 304. The speech control unit 301 outputs a speech control signal for controlling the speech of the robot 100A.

  The speech control unit 301 receives the next speaker probability from the next speaker estimation unit 108A as an input, and outputs a speech control signal to the sound control unit 110A. The speech control unit 301 uses any of the four control methods of the fifth control method to the eighth control method described above based on the robot 100A from the next speaker estimation unit 108A and the next speaker probability of each user. A speech control signal for controlling the speech of the robot 100A is output. The speech control unit 301 also outputs a speech control signal to the operation control unit 303. The speech control unit 301 outputs the sound generation instruction signal to the sound control unit 110A in response to the sound generation instruction signal instructing to generate the breathing sound and the filler from the operation control unit 303. Here, the filler is a voice for connecting the place which appears at the time of saying and so on, and is, for example, a voice such as "Ano", "No", "Her".

  The motion pattern information storage unit 302 stores motion pattern information which is information on a motion pattern of a motion performed by the robot 100A during a conversation. The motion pattern information storage unit 302 causes, for example, motion pattern information to cause the robot 100A to perform the same action as the action that a person performs to cause the surrounding people to sense that the utterance is to be made before the start of the utterance. It is stored.

As a result of analyzing the action taken immediately before the non-speaker speaks as the next speaker while multiple people are talking, the following actions (1) to (3) It is thought that it is an action shown to the surrounding that "it begins".
(1) Speak inspiratory noise or filler (2) Look at the current speaker (3) Look at the conversation of the current speaker

Based on the analysis result described above, the control unit 109A controls the robot 100A to perform the above-described operations (1) to (3) before the robot 100A makes an utterance, so that the robot 100 makes an utterance immediately. The user can be foreseen to start. When the robot 100A performs the above-described operations (1) to (3), the next speaker probability P ^ns _R (t) of the robot 100A estimated by the next speaker estimation unit 108A is increased. That is, the action of causing the surrounding person to sense that the user speaks includes, for example, an action of moving the line of sight to the current speaker, an action of turning the head, an action of sucking with the intake sound, and the like.

The control unit 109A may control the robot 100A to perform the above-described operations (1) to (3) using the technology described in the following reference.
As a technique for causing the robot 100A to perform the operation of producing the suction sound of (1), there is a known technique described in Reference 2 below.
Reference 2: Naoto Yoshida, 3 others, "Design of biological body emotion interaction based on factor analysis on respiratory expression with exhalation and abdominal movement", Gaze to current speakers in HAI Symposium 2014, 2014 (2) There is a known technique described in the following reference 3 as a technique for causing the robot 100A to perform an operation of pointing the
Reference 3: Atsushi Ishii and 2 others, "Gaze control to promote conversation in avatar voice chat system", Human Interface Society Journal, 2008 (3) Motion to talk to the speaker of the current speaker to robot 100A There is a known technique described in the following reference 4 as a technique for performing.
Reference 4: Tomio Watanabe, 3 others, "A physical interaction system based on speech voice using InterActor", Journal of Human Interface Society, Vol. 2, No. 2, pp. 21-29, 2000

  The motion control unit 303 receives the motion pattern information from the motion pattern information storage unit 302 based on the speech control signal from the speech control unit 301, the speaker information from the next speaker estimation unit 108A, and the position information of the user. The motion control signal is acquired and generated, and the generated motion control signal is output to the mouth control unit 111, the sight control unit 112, the head control unit 113, and the trunk control unit 114. The operation control unit 303 outputs, to the speech control unit 301, a sound generation instruction signal instructing to sound the breathing sound and the filler according to the operation pattern information. The position measurement device 201 may have a function of acquiring position information for specifying the position of the user's face as the user's position information. Thereby, the robot 100 uses, for example, position information for specifying the position of the user's face as the position information of the user, for example, an operation of directing the face and the line of sight of the robot 100 to the face of the current speaker It can be performed.

The details of the configuration of the sound control unit 110A in the second embodiment will be described by way of an example.
FIG. 9 is a diagram showing a specific example of the details of the configuration of the sound control unit 110A in the second embodiment. The sound control unit 110A in the second embodiment shown in FIG. 9 includes the same components as the sound control unit 110 in the first embodiment. Therefore, in the description of the sound control unit 110A, the same components as those of the sound control unit 110 in the first embodiment will be assigned the same reference numerals and descriptions thereof will be omitted.

  The sound control unit 110A includes a voice analysis unit 401, a conversation information generation unit 402, a conversation information DB 403A, an utterance information generation unit 404A, and a sound signal generation unit 405.

  The conversation information DB 403A further stores respiratory sound information and filler information in addition to the conversation sample information stored in the conversation information DB 403 of the first embodiment described above. The breathing sound information is information on breathing sound to be generated by the robot 100A. The respiratory sound information is, for example, information including an audio signal of an inspiratory sound which is emitted when a person inhales as "soo" or "shoe". The filler information stores filler information which is information of a filler to be sounded by the robot 100. The filler information is information including sound signals of fillers such as “Ann”, “Hmm”, “Ett” and the like.

  In addition to the function of the utterance information generation unit 404 of the above-described first embodiment, the utterance information generation unit 404A responds to a speech instruction signal instructing to pronounce the breathing sound and the filler from the control unit 109A. It requests the generation unit 402 for conversation information for producing a breathing sound and a filler. As a result, the conversation information generation unit 402 refers to respiratory information and filler information including respiratory sound and a voice signal for producing a filler from the conversation information DB 403A, and speaks conversation information for producing a respiratory sound and a filler. It is output to the generation unit 404A. The utterance information generation unit 404A generates a speech signal for causing the robot 100 to utter the breathing sound or the filler based on the speech information for producing the breathing sound or the filler acquired from the conversation information generation unit 402 in response to the request. The utterance information generation unit 404 outputs the generated utterance signal to the sound signal generation unit 405.

With the above-described configuration, when the robot 100A wants to speak, the robot 100A can direct the line of sight to the user based on the operation control signal, and can sound the breathing sound and the filler before the speech. As a result, since the next speaker probability P ^ns _R (t) of the robot 100A is increased, the possibility that the control unit 109A outputs speech control information to be uttered by the robot 100A is increased. The user can foresee that the robot 100 will speak soon before the robot 100 starts speaking. By this prediction, a speech collision between the user and the robot 100 can be prevented, and a smooth conversation can be realized.

FIG. 10 is a diagram showing a specific example of the pre-conversation operation of the robot 100A in the second embodiment. As shown in FIG. 10, a specific example in the case where the robot 100A and the current speaker 60 who is the user who is speaking are present will be described. The left side of FIG. 10 shows a state in which the robot 100A is listening to the talk of the current speaker 60. The right side of FIG. 10 shows an operation immediately before the robot 100A starts speech. As shown in FIG. 10, the robot 100 turns its head to the current speaker 60 by rotating the head 53 in the direction indicated by the arrow 61 immediately before the start of speech. At the same time or before or after the rotation of the head 53, an inspiratory sound 62 "sud" is produced from the speaker 115 in the mouth 52. As a result, since the next speaker probability P ^ns _R (t) of the robot 100A is increased, the possibility of the speech of the robot 100A can be enhanced. The current speaker 60 can foresee that the robot 100A will speak soon. That is, the robot 100A can speak at more appropriate timing.

Next, the conversation operation of the robot 100A in the second embodiment will be described.
FIG. 11 is a flowchart showing the conversation operation of the robot 100A in the second embodiment. Similar to the process shown in FIG. 6, the process shown in FIG. 11 is a process performed when the robot 100A starts an operation to talk with a plurality of users.

  The audio input unit 104 receives an audio signal from the microphone 101, the video input unit 105 receives a video signal from the camera 102, and the sensor input unit 106 receives a sensor signal from the sensor 103. Further, the conversation operation of the robot 100A is performed under the control of the control unit 109A (step S201). The conversation operation of the robot 100A includes the operations (1) to (3) described above. In response to the conversational motion of the robot 100A, the control unit 109A outputs the pseudo sensor signal to the next speaker estimation unit 108A.

The speech zone detection unit 107 calculates a speech feature amount based on the speech signal from the speech input unit 104, and compares the calculated speech feature amount with a predetermined threshold to detect a speech zone (step S202). The next speaker estimation unit 108A is a next speaker who has a probability that the robot 100A and each user i will be the next speaker at time t based on the audio signal, the video signal, the sensor signal, the pseudo sensor signal, and the speaker information. The probability P ^ns _i (t) is calculated (step S203).

  The control unit 109A outputs a speech control signal based on the robot 100A from the next speaker estimation unit 108A and the next speaker probability of each user (step S204). The sound control unit 110A determines whether to cause the robot 100A to make a speech based on the speech control signal from the control unit 109A (step S205). Here, if it is determined that the robot 100A is not allowed to make a speech (NO in step S205), the process returns to step S201. When it is determined that the robot 100A is made to utter (YES in step S105), the sound control unit 110A generates conversation information for making the robot 100A utter, and the speaker makes a sound signal based on the generated conversation information It outputs to 115 (step S206). As a result, the robot 100A produces an utterance corresponding to the sound signal from the speaker 115.

  The sound control unit 110A determines whether to end the speech of the robot 100A based on the speech control signal from the control unit 109A (step S207). Here, when the speech of the robot 100A is not ended (NO in step S207), the sound control unit 110A returns to the process of step S206. When ending the speech of the robot 100A (YES in step S207), the sound control unit 110A stops the output of the sound signal in response to the stop of the generation of the conversation information.

  Next, the robot 100A determines whether to end the conversation operation for talking with a plurality of users (step S208). Here, when it is determined that the conversation operation is not ended (NO in step S208), the process returns to the process of step S201. When it is determined that the conversation operation is ended (YES in step S208), the robot 100A ends the conversation operation.

  As described above, when the robot 100A in the second embodiment talks with a plurality of users, the robot 100A compares the next speaker probability of the robot 100A with the next speaker probability of each user. Utterance can be made at more appropriate timing. As a result, the robot 100A can perform an utterance with reduced occurrence of an utterance collision in which the timing of the utterance overlaps with that of another user. In addition, the robot 100A can perform the utterance with the occurrence of the utterance collision reduced more surely by performing the operation of making the surrounding people detect that the utterance is to be made before the utterance.

(Modification of the second embodiment)
The modification of the second embodiment is an index that indicates the necessity of speech of the robot 100A, and the higher the necessity of speech is, the larger the value of speech required degree is calculated, and the calculated degree of speech necessity is used , The next speaker probability of the robot 100A is changed. The robot 100A acquires the content of the user's speech using a known speech recognition technology, and calculates the degree of speech necessity according to the acquired speech content.
As a specific calculation method of the degree of necessity of speech, a method of calculating the speech level U _L indicating the degree of necessity of speech on the basis of "speech contents", "relationship with the other party" and "personality of the speaker" It is described in reference 5.
Reference 5: Mai Kawagoe, Yasuhiko Kitamura "Multi-Agent Dialogue System Considering Utterance Timing", IEICE Technical Report, AI, Artificial Intelligence and Knowledge Processing, 106 (617), pp 53-56, March 2007 21st

Control unit 109A, for example by using a speech level _{U L,} instead of the ^P _ns R follows the speaker probability ^P _ns R (t) used in the fifth control method or seventh control method described above by the following equation (t) Ask for '.
P ^ns _R (t) '= P ^ns _R (t) · U _L · ζ (ζ is an arbitrary constant)
The following equation may be used as a modification of the above equation.
P ^ns _R (t) '= P ^ns _R (t) + U _L ζ (ζ is an arbitrary constant)

Control unit 109A, for example by using a speech level _{U L,} determine the ^{P ns} _R 'in place of the integrated value ^{P ns} _R used in the sixth control method or the eighth control method described above by the following equation.
P ^ns _R '= P ^ns _R · U _L · ζ (ζ is an arbitrary constant)
The following equation may be used as a modification of the above equation.
P ^ns _R '= P ^ns _R + U _L · ζ (ζ is an arbitrary constant)

  By controlling the speech on the basis of the next speaker probability which changes in accordance with the degree of necessity of speech in this manner, the robot 100A can speak according to the necessity of the speech, and when the speech is unnecessary, I will not speak more. That is, the robot 100A can speak at more appropriate timing.

(Specific example of processing for estimating the next speaker common to the first and second embodiments)
Next, a specific example of processing for estimating the next speaker common to the robot 100 in the first embodiment and the robot 100A in the second embodiment described above will be described. For example, the techniques of the following references 6 and 7 can be applied to the next speaker estimation in the robot 100 and the robot 100A, but any existing technique may be used. When the techniques described in References 6 and 7 are used, the next-speaker estimation unit 108 or the next-speaker estimation unit 108 using the transition patterns of the gaze actions of the speaker and the non-speaker based on the gaze target information output by the gaze target detection device 203. The next speaker estimation unit 108A predicts the timing of the next speaker and the utterance.

Reference 6: Japanese Patent Laid-Open No. 2014-238525 Reference 7: Jun Ishii, 4 others, "Prediction of the next speaker and the speech timing based on the gaze transition pattern in multi-person dialogue", Artificial Intelligence Society research meeting material, SIG -SLUD-B301-06, pp. 27-34, 2013

An example of the next speaker estimation technique other than the references 6 and 7 applicable to the present embodiment will be shown below.
The user's breathing behavior of the conversation is closely related to the timing of the next speaker and the speech. Using this fact, the user's respiratory activity of the conversation is measured in real time, and the characteristic respiratory activity performed just before the start of the speech is detected from the measured respiratory activity, and the next utterance is made based on this respiratory activity. People and their speech timing with high accuracy. Specifically, as a feature of the breathing movement performed immediately before the start of the utterance, when the utterer is uttering continuously (when the utterer continues), the breath immediately immediately after the end of the utterance immediately and sharply Inhale. On the other hand, when the utterer does not speak next (during utterer substitution), he / she breathes in slowly, leaving a gap from the end of the uttering as compared with the uttering continuation. In addition, at the time of speaker change, the next speaker who speaks next inhales a large amount of breath compared to the non-speaker who does not speak. The breathing performed prior to such an utterance is performed at a roughly determined timing relative to the start of the utterance. As described above, since the next speaker performs characteristic breath suction immediately before the speech, such breath suction information is useful for predicting the next speaker and the timing of his / her speech. In this next speaker estimation technology, attention is focused on the inhaling of a person's breath, and information on the amount of inhaling breath, the length of the inhaling section, the timing, and the like is used to predict the next utterer and the utterance timing.

In the following, the user P ₁ of the _{A's, ...,} P _A is assumed a situation to perform a face-to-face communication. The respiratory movement measurement device 202 and the microphone 101 are attached to the user P _a (where a = 1,..., A, A ≧ 2). Respiration measuring device 202 measures the respiration of the user P _a, respiration information B _a representative of the measurement results for each discrete time _t, to obtain _t, next speaker estimation unit 108 or the next speaker estimator Output to 108A. A configuration in which the respiratory motion measurement device 202 includes a band-type respiratory device will be described. The band type breathing apparatus outputs a value indicating the degree of breathing depth by the stretching strength of the band. The greater the inspiration of the breath, the greater the extension of the band, and the greater the exhalation of breath, the greater the contraction of the band (the smaller the extension of the band). Hereinafter, this value is called the RSP value. The RSP value takes a different size for each user P _a according to the strength of expansion and contraction of the band. Therefore, in order to eliminate the difference of RSP values for each P _a resulting therefrom, using the average value mu _a and the standard deviation value [delta] _a of RSP values for each user P _{a, μ a + δ a} is 1 , R ap values are normalized for each user P _a such that μ _a −δ _a is −1. This makes it possible to analyze respiratory motion data of all users P _a identically. Each respiratory motion measurement device 202 sends the normalized RSP value to the next speaker estimation unit 108 or the next speaker estimation unit 108A as respiration information Ba _{, t} .

Further, the microphone 101, the user P obtains the sound _a, the audio signals V _a representative of the voice of the user P _a at each discrete time _t, to obtain _t, next speaker estimation unit 108 or the next speaker It outputs to estimation section 108A. The next-speaker estimation unit 108 or the next-speaker estimation unit 108A removes noise from the input voice signal V _{a, t} (where a = 1,..., A), and further the speech interval U _k (where k is k). Extracts the identifier of the speech section U _k and its speaker P _uk . However, the subscript of “P _uk ” represents u _k = 1,. One speech segment U _k is defined as a segment surrounded by Td [ms] continuous silent segments, and this speech segment U _k is defined as one unit of speech. As a result, the next speaker estimation unit 108 or the next speaker estimation unit 108A outputs the utterance section information representing each utterance section U _k and the speaker information representing the speaker P _uk (users P ₁ ,..., P _A Speaker information representing which is the speaker P _uk in the utterance section U _k .

Next speaker estimation unit 108 or the next speaker estimation unit 108A, the breathing information _{B a} of each user _{P a,} with _t, extracts the suction section _{I a, k} breath of the user _{P a,} further The parameters λ _{a, k} for breathing in are obtained. The breath inhaling section indicates a section between the start position where the breath starts and the end position where the breath ends, from the state of exhaling.

FIG. 12 is a diagram illustrating an example of a breath suction section. A method of calculating the breathing interval _{Ia, k} will be illustrated using FIG. Here, the RSP value at the discrete time t of the user P _a is expressed as R _{a, t} . The RSP value _{Ra, t} corresponds to the respiration information _{Ba, t} . As exemplified in FIG. 12, for example, when the following (Expression 1) holds,

RSP value _{R a} in the previous two frames discrete time t _{= t s (k), t} continuously decreases, RSP value _{R a} in the subsequent two _frames, since _t is increasing continuously, discrete time t _{Let s (k) be} the starting point for breathing. Furthermore, when the following (formula 2) holds,

The RSP value _{Ra, t in} the previous two frames of discrete time t = t _{e (k)} rises continuously, and then the RSP value _{Ra, t in} two frames decreases continuously, so discrete time t _{Let e (k) be} the end position of the breath intake. In this case, the user _{P a} suction section _{I a breath, k} becomes the interval from _{t s (k)} to _{t e (k),} the length of the suction section of breath _{t e (k) -t s} ( _k) .

Next speaker estimation unit 108 or the next speaker estimation unit 108A, the suction section _{I a breath,} when _k is extracted, the suction section _{I a breath, k,} respiration information _{B a, t,} and speech segment _{U k} The parameter λ ′ _{a, k} for breathing in is extracted using at least a portion of Parameter lambda _{'a, k} is the suction of the user _{P a} suction section _{I a,} breath in _k amounts, suction section _{I a,} the length of _k, the suction section _{I a,} the suction amount of the breath at the _k The time change represents at least a part of the time relationship between the speech interval U _k and the suction interval I _{a, k} . The parameters λ ′ _{a and k} may represent only one of them, or may represent a plurality of them, or all of them. Parameter lambda _{'a, k} includes, for example, the following parameters _{MIN a, k, MAX a,} k, AMP a, k, DUR a, k, SLO a, k, INT1 a, at least a portion of _k. The parameter λ ′ _{a, k} may include only one of these, may include a plurality of these, or may include all of these.
· _{MIN a, k:} user _{P a} suction at the start of the RSP value _{R a breath, t,} that is, the suction section _{I a breath, k} of the RSP value _{R a,} minimum value of _t.
· _{MAX a, k:} user _{P a} suction at the end of the RSP value _{R a breath, t,} that is, the suction section _{I a breath, k} of the RSP value _{R a,} the maximum value of _t.
· _{AMP a, k:} User _{P a} suction section _{I a breath, k} of RSP values _{R a,} the amplitude of _{t, i.e., MAX a,} k -MIN _a, value calculated by _k. It represents the amount of inhaled breath in the inhaling section _{Ia, k} .
DUR _{a, k} : The length of the breathing interval I _{a, k} of the user P _a , that is, the discrete time of the starting position from the discrete time t _{e (k)} of the ending position of the breathing interval I _{a, k} the value obtained by subtracting _{t s (k) t e (} k) -t s (k).
SLO _{a, k} : The average value of the slope per unit time of the RSP value R _{a, t} in the breathing interval I _{a, k} of the user P _a , that is, calculated by AMP _{a, k} / DUR _{a, k} Value. It represents the time change of the amount of inhaled breath in the inhale section _{Ia, k} .
• INT1 _{a, k} : an interval from the end time t _{ue (k) of the preceding} utterance section U _k (end of the utterance section) to the start of breathing of the user P _a , that is, a breath inspiration section I _{a ,} discrete time _{t s (k)} from the speech segment _{U k} of the end time _{t ue} value obtained by subtracting the _{(k) t} s of the start position of _{k (k) -t ue (k} ). This represents the time relationship between the speech interval U _k and the suction interval I _{a, k} .

The next speaker estimation unit 108 or the next speaker estimation unit 108A may further generate the following parameter INT2a _{, k} .
· INT2 _{a, k:} interval up to the speech segment _{U k + 1} of the next speaker is started from the time of the end intake of breath of the user _{P a,} ie, the next speaker of the speech segment _{U k + 1} of the start time _{t us (k + 1 The} value _{tus (k + 1)} -te _(k) obtained by subtracting the discrete time te _(k) at the end position of the breathing interval _{Ia, k from} the It represents the time relationship between the speech interval U _{k + 1} and the suction interval I _{a, k} . Parameters λ _'a, INT2 _a, a plus _k is denoted as parameter lambda _{a, k} to _k.

The next speaker estimation unit 108 or the next speaker estimation unit 108A obtains information representing, for example, the speech interval U _{k + 1} and further obtains the parameter λ _{a, k} (after the speech interval U _{k + 1} is started) The utterance section U _k and its speaker P _uk , the utterance section U _{k + 1} and its speaker P _{uk + 1} and its speech start timing T _{uk + 1} are recorded in the database. The speech timing of the next speaker P _{uk + 1} may be any time point of the speech interval U _{k + 1} or a time point corresponding thereto. The speech timing T _{uk + 1} may be the start time _{tus (k + 1)} of the speech segment U _{k + 1} or may be the time _{tus (k + 1)} + γ (where γ is a positive or negative constant). The end time t _{ue (k + 1)} of the speech section U _{k + 1} may be used, or the time t _{ue (k + 1)} + γ may be used, or the center time t _{us (k + 1)} + (t _{ue ( k) of} the speech section U _{k + 1 It} may be _{k + 1)} -t _{us (k + 1} ) / 2. A part or all of the information representing λ _{a, k} , U _k , P _uk , P _{uk + 1} , T _{uk + 1} is held in the database, and the next speaker estimation unit 108 or the next speaker estimation unit 108A performs more than the speech interval U _{k + 1} It is used to predict the subsequent next speaker and its speech timing.

Next speaker estimation unit 108 or the next speaker estimation unit 108A, speaker information _{P uk,} speech segment _{U k,} the suction section _{I a} of the user _{P a,} suction amount of breath at _k, the suction section _{I a, k} The user P ₁ , ..., based on at least a part of the length of the breathing section I _{a, k} and the time change of the breathing amount in the breathing section I _{a, k} and the time relationship between the speech section U _k and the sucking section I _{a, k} which of the P _a is either the next speaker P _{uk + 1,} and obtain estimated information representative of at least one of the following speaker P _{uk + 1} of the utterance timing. However, the subscript “ _{uk + 1} ” of “P _{uk + 1} ” represents u _{k + 1} . (If speech continues) if speaker _{P uk} speech period _{U k} performs speech even speech section _{U k + 1,} the next speaker is the same as the speaker _{P uk} speech period _{U k.} On the other hand, (if That utterance replacement) when uttered P _uk other users of the speech segment U _k performs speech even speech section U _{k + 1,} the following speaker is other than speaker P _uk speech period U _k users It is.

Next speaker estimation unit 108 or the next speaker estimation unit 108A, speaker information _{P uk,} speech segment _{U k,} the suction section _{I a} of the user _{P a,} suction amount of breath at _k, the suction section _{I a, k} For a feature amount f _{a, k} corresponding to at least a part of the time length of the suction interval I _{a, k} and the time change of the breathing amount in the suction interval I _{a, k} and the time relationship between the speech interval U _k and the suction interval I _{a, k} A model for obtaining estimated information is machine-learned, and this model is used to obtain estimated information for feature quantities. Feature value _{f a, k} is the speaker information _{P uk,} speech segment _{U k,} the suction section _{I a} of the user _{P a,} suction amount of breath at _k, the suction section _{I a,} the length of _k, the suction section I _It may correspond to only one time change of the amount of breath suction at _{a, k} and the time relationship between the speech interval U _k and the suction interval I _{a, k} , or may correspond to a plurality of these. It is good and may correspond to all. The machine learning model, for example, past suction section _{I a, i} (although, i <k) suction of breath, the suction section _{I a,} the length of the _i, suction section _{I a,} breath in _i Changes in the amount of suction and feature quantities f _{a, k} corresponding to at least a part of the time relationship between the speech interval U _i and the suction interval I _{a, i} , and the speech intervals U _i , U _{i + 1} and their speakers Information of P _uk and P _{uk + 1} is used as learning data.

A next speaker / speech timing estimation process by the next speaker estimation unit 108 or the next speaker estimation unit 108A is illustrated. In this example, the next speaker estimation model is a model that estimates the next speaker P _{uk + 1,} and the response timing estimation model is a model for estimating the response timing of the next speaker P _{uk + 1} is generated, using each model Then, the next speaker P _{uk + 1} and its speech timing are estimated.

When learning the next-speaker estimation model, the next-speaker estimation unit 108 or the next-speaker estimation unit 108A uses the past parameters λ _{a, i} (where a = 1,..., A) as learning data. , I <k), and information representing utterance intervals U _i and U _{i + 1} and their speakers P _ui and P _{ui + 1} . The next-speaker estimation unit 108 or the next-speaker estimation unit 108A sets the feature quantities F1 _{a, i} and U _i , U _{i + 1} , P _ui , P _{ui + 1} corresponding to at least a part of the parameters λ _{a, i} as learning data. The next speaker estimation model is machine-learned. For the next speaker estimation model, for example, SVM (Support Vector Machine), GMM (Gaussian Mixture Model), HMM (Hidden Markov Model) or the like can be used.

Next speaker estimation unit 108 or the next speaker estimation unit 108A, the parameter lambda _'a, the feature amount corresponding to at least a portion of the _k F1 _a, the _k is applied to the next speaker estimation model was estimated thereby following Information representing the utterance P _{uk + 1} is made part of “estimated information”. Note that the information representing the next utterance P _{uk + 1} may represent either the user P _a definitively or probabilistically. The probability that the user P _a will be the next speaker is P 1 _a .

When learning a speech timing estimation model, the next speaker estimation unit 108 or the next speaker estimation unit 108A uses the database as parameters λ _{a, i} (where a = 1,..., A) as learning data. Information representing at least a part of i) <k>, speech sections U _i and U _{i + 1,} their speakers P _ui and P _{ui + 1} , and speech start timing T _{ui + 1} of the speech section U _{i + 1} is read out. Next speaker estimation unit 108 or the next speaker estimation unit 108A, the learning parameter lambda _a, feature amount corresponding to at least a portion of the _{i F2 a, i} and _{U i, U i + 1,} P ui, the _{P ui + 1, T ui +} 1 Machine learning of a speech timing estimation model is performed as data. For example, SVM, GMM, HMM, etc. can be used as the next speaker estimation model.

If the next speaker estimation unit 108 or the next speaker estimation unit 108A obtains the speaker P _uk , at least a part of the parameters λ ′ _{a, k} , and the next speaker P _{uk + 1} estimated by the next speaker estimation model applies the parameter lambda _'a, feature amount corresponding to at least a portion of the _k F2 _a, the _k utterance timing estimation model. The next-speaker estimation unit 108 or the next-speaker estimation unit 108A applies the feature amount F2 _{a, k} to the utterance timing estimation model to determine the utterance timing T _{uk + 1 of} the next utterance segment U _{k + 1} (for example, the utterance segment U _The information representing the start time of ( _{k + 1} ) is output as part of the "estimated information". Note that the information indicating the utterance timing may represent any one of the utterance timings in a definite manner, or may represent the utterance timings in a probabilistic manner. The probability that the user _{P a} to start a speech to the time t (the probability time t is the utterance timing of the user _{P a),} and _P2 a (t).
The next speaker probability P ^ns _i (t) at time t of the user i estimated by the next speaker estimation unit 108 or the next speaker estimation unit 108A in the above-described embodiment is the user i's next speaker estimation technique In the case of the user P _{a in,} it is calculated by probability P 1 _a × probability P 2 _a (t).

The above-mentioned next-speaker estimation unit 108 or next-speaker estimation unit 108A estimates the user and timing to start speeching next based on the observed value of the respiratory motion, but further uses the observed value of the line of sight. May be
When the eye-gaze action is further used, the gaze target detection device 203 is further attached to each user P _a (where a = 1,..., A). Gaze object detection device 203, the user P _a detects someone or gazing (gaze target), the user P _a and gaze target G _a, next speaker information representing a _t at each discrete time t It sends to the estimation unit 108 or the next speaker estimation unit 108A. The next-speaker estimation unit 108 or the next-speaker estimation unit 108A receives the gaze target information G _{1, t} ,..., G _{A, t} , the speech segment U _k and the speaker information P _uk as input, and before and after the speech segment end Gaze target label information θ _{v, k} (where, v = 1,..., V, V is the total number of gaze target labels). The gaze target label information is information indicating the gaze target of the user in the time interval corresponding to the end time point T _se of the speech interval U _k . Here, gaze target label information θ _{v, k in} which the gaze target of the user P _a in a finite time interval including the end time point T _se is labeled is illustrated. In this case, for example, a gaze action appearing in a section from a time point T _se −T _b before an end time point T _se of the speech interval U _{k to} a time point T _se + T _a after the end time point T _se is handled. T _{b, T a} is may be any value from 0 or more, as a guide, _{T b} is 0 seconds to 2.0 seconds, _{T a} is appropriate to about 0 seconds to 3.0 seconds.

The next speaker estimation unit 108 or the next speaker estimation unit 108A classifies the gaze target user into the following types, and labels the gaze target. Note that the symbols of the labels have no meaning, and any notation may be used as long as they can be determined.
Label S: current speaker (ie, represents the user P _uk who is the current speaker)
Label L _ξ : Non-speaker (However, 識別 identifies different non-speaker users, and, = 1,..., A−1. For example, a certain user is a non-speaker P ₂ , The non-speaker P ₃ , the non-speaker P ₂ is assigned the label L ₁ , and the non-speaker P ₃ the label L ₂ ).
・ Label X: No one is watching

Label when S or L _xi] are mutually gaze (line-of-sight intersections) to impart information that whether happened. In this embodiment, when the mutual gaze has occurred is, S _M, as in the L _{ξM (under "KushiM"} in superscript represents xi] _M), imparts M label at the end of the label S, L _xi] .

FIG. 13 is a diagram illustrating a specific example of the gaze target label. FIG. 13 shows an example of A = 4, where utterance segments U _k and U _{k + 1} and gaze targets of each user are shown in time series. In the example of FIG. 13, after the user P ₁ is uttered, occurs speech alternation shows how when a new user P ₂ has an utterance. In this case, after the user P ₁ is the current speaker was gazing at the user P _4, gazing at the user P _2. In the period from the time of T _se -T _b up to the point of _T se + _{T a,} when the user _{P 1} had seen the user _{P 2,} the user _{P 2} is a look at the user _{P 1.} This represents that the mutual fixation is happening at the user P ₁ and the user P _2. In this case, the gaze target labels generated from the gaze target information G _{1, t of the} user P ₁ are two, L ₁ and L _{2 M.} In the above-mentioned period, the user P ₂ After watching the user P _4, gazing at the user P ₁ is the current speaker. In this case, you gaze target label of the user _{P 2} is two and the _{L 1} and _{S M.} In addition, in the above-mentioned period, the user P ₃ is gazing at the user P ₁ is the current speaker. In this case, the gaze target label of the user P ₃ is a S. In addition, in the above-mentioned period, the user P ₄ is not anyone seen. In this case, the gaze target label of the user P ₄ is the X. Therefore, in the example of FIG. 13, V = 6.

The next speaker estimation unit 108 or the next speaker estimation unit 108A also acquires a start time and an end time for each gaze target label. Here, R as a symbol indicating which gaze target label (GL ∈ {S, S _M , L ₁ , L _{1 M} , L ₂ , L _{2 M} , ...}) of who (R ∈ {S, L}) Define _GL , its start time as ST_R _GL , and its end time as ET_R _GL . Here, R represents the user's speech state (current speaker or non-speaker), S is the current speaker, and L is the non-speaker. For example, in the example of FIG. 13, the first fixation target label of the user _{P 1} is _{S L1,} the start time ST_S _L1, the end time is ET_S _L1. The gaze target label information θ _{v, k} is information including a gaze target label R _GL , a start time ST_R _GL , and an end time ET_R _GL .

Next speaker estimation unit 108 or the next speaker estimation unit 108A, by using the gaze target label information theta _{v, k,} gaze target transition pattern _{E a} of each user _{P a,} generates a _k. The generation of the gaze target transition pattern is performed by generating a transition n-gram in consideration of the temporal order, using the gaze target label R _GL as a component. Here, n is a positive integer. For example, in the example of FIG. 13, the gaze target transition pattern E _{1, k} generated from the gaze target label of the user P1 is L ₁ -L _2M . Similarly, gaze target transition pattern _{E 2} of the user _{P 2, k} is _L 1 -S _M, gaze target transition patterns _{E 3, k} of the user _{P 3} is S, gaze target transition pattern E of the user _{P 4 4, k} is X.

The gaze target transition pattern E _{a, k} is, for example, a speech period U _k and its speaker P _uk after the speech period U _{k + 1} is started, and a next utterer P _{uk +1} and a next utter that perform speech corresponding to the speech period U _{k + 1} It is sent to the database together with information representing the start timing T _{uk + 1} . In the database, the gaze target transition pattern E _{a, k} is merged with the parameters λ _{a, k} , and part or all of the information representing E _{a, k} , λ _{a, k} , U _k , P _uk , P _{uk + 1} , T _{uk + 1} Is kept in the database.

The next speaker estimation unit 108 or the next speaker estimation unit 108A receives the gaze target label information θ _{v, k} as input, and generates time structure information Θ _{v, k} for each gaze target label. The temporal structure information is information representing the temporal relationship of the user's gaze behavior, (1) time length of the gaze target label, (2) an interval between the gaze target label and the start time or end time of the speech section, 3) It has an interval between the start time or end time of the gaze target label and the start time or end time of another gaze target label as a parameter.

Specific parameters of time structure information are shown below. In the following, the start time of the speech section is defined as ST_U, and the end time of the speech section is defined as ET_U.
_{· INT1 (= ET_R GL -ST_R GL} ): gazing target label _{R GL} of the start time ST_R _GL and end time ET_R interval of _{GL · INT2 (= ST_U-ST_R} GL): start time ST_R _GL of the gaze target label _{R GL} utterance How long ago the start time ST_U of the section was • INT3 (= ET_U-ST_R _GL ): how long the start time ST_R _GL of the gaze target label R _GL was before the end time ET_U of the speech section INT4 (= ET_R _GL -ST_U): gazing target label _{R GL} of the end time ET_R _GL Do · INT5 was after much than the start time ST_U of the speech segment (= ET_U-ET_R _GL): end time ET_R _GL is the utterance section of the gaze target label _{R GL} Any more than the end time ET_U of Have either · _INT6 had been before _{(= ST_R GL -ST_R GL ')} : the gaze target label _{R GL} of the start time ST_R _GL other of the gaze target label _{R GL'} of the start time ST_R _GL or was after much than _'· INT7 ( = ET_R _{GL '-ST_R GL):} gazing target label _{R GL} of the start time ST_R _GL other of the gaze target label _{R GL'} of the end time ET_R _{GL 'or} was before much than _{· INT8 (= ET_R GL -ST_R GL} ' ): gaze target label _{R GL} of the end time ET_R _GL is gazing target label _{R GL 'of} the start time ST_R _GL' or was after much than _{· INT9 (= ET_R GL -ET_R GL} '): the end of the gazing target label _{R GL} time ET_R _GL is none than the _'end time ET_R _{GL of'} gaze target label _{R GL} Did even after leprosy

In addition, about INT6-INT9, it acquires with respect to the combination with the gaze object label of all the users. In the example of FIG. 13, since there are six gaze target label information in total (L ₁ , L _{2 M} , L ₁ , S _M , S, X), each of INT 6 to INT 9 has 6 × 5 = 30 pieces of data. It is generated.

Temporal structure information Θ _{v, k} is information composed of parameters INT 1 to INT 9 for gaze target label information θ _{v, k} . Each of the above-mentioned parameters constituting the time structure information Θ _{v, k} will be concretely shown using FIG. FIG. 14 is a diagram showing time structure information of the gaze target label L1 of the user P1 (R = S) who is the current speaker. That is, it is the time structure information in _RGL = _SL1 . Note that the INT6～INT9, for simplicity of illustration, gaze target label L1 of the user P2, ie the only relationship between _R GL _{= L L1.} In the example of FIG. 14, it is understood that INT1 to INT9 can be obtained as follows.
· INT1 = ET_S _L1- ST_S _L1
・ INT2 = ST_U-ST_S _L1
・ INT3 = ET_U-ST_S _L1
· INT4 = ET_S _L1- ST_U
・ INT5 = ET_U-ET_S _L1
-INT6 = ST_S _L1- ST_L _L1
-INT7 = ET_L _L1- ST_S _L1
-INT 8 = ET_S _L1- ST_L _L1
・ INT9 = ET_S _L1 -ET_L _L1

Time structure information theta _{v, k,} for example after the speech segment _{U k + 1} is started, the speech segment _{U k} and its speaker _{P uk,} next speaker _{P uk + 1} and the next utterance start performing speech corresponding to the speech segment _{U k + 1} It is sent to the database together with information representing the timing T _{uk + 1} . In the database, temporal structure information Θ _{v, k} is merged with the parameters λ _{a, k} and _one of the information representing Θ _{v, k} , λ _{a, k} , U _k , P _uk , U _{k + 1} , P _{uk + 1} , T _{uk + 1} Part or all is kept in the database.

The next-speaker estimation unit 108 or the next-speaker estimation unit 108A selects the gaze target transition pattern E _{a, k} , the time structure information Θ _{v, k} , the speaker information P _uk , the utterance section U _k , and the suction section of the user P _a I _a, suction amount of breath at _k, the suction section _{I a,} the length of _k, the suction section _{I a,} suction amount of time variation of the breath at _k, and speech periods _{U k} and the suction section _{I a,} and _k Machine learns a model for obtaining estimated information for the feature quantity f _{a, k} corresponding to at least a part of the time relationship of, and uses the model to estimate the next speaker probability P ^ns _i (t) Obtain and output.

Although the above-described next-speaker estimation unit 108 or next-speaker estimation unit 108A estimates the user and timing to start speaking next based on the observed value of respiratory motion and the observed value of gaze, Information on the movement of the head of the person may be used. This utilizes the fact that a person tends to crawl in front of his / her speech. The next-speaker estimation unit 108 or the next-speaker estimation unit 108A analyzes the image data of each user from the video input unit 102, and determines whether the user looked up or not depending on whether the head moved up and down. judge. If the next speaker estimation unit 108 or the next speaker estimation unit 108A determines that the user i has visited several seconds before the time t, the next speaker probability P ^ns _i (t) at the time t of the user i A process of adding a predetermined value to Further, the next speaker estimation unit 108 or the next speaker estimation unit 108A determines the next speaker probability P based on at least one of the observation value of the respiratory movement, the observation value of the line of sight, and the information on the head movement of the user. You may calculate ^ns _i (t).

  In addition, when the next speaker estimation unit 108 or the next speaker estimation unit 108A uses at least one of the observation value of the breathing movement, the observation value of the line of sight, and the information on the movement of the head of the user, Depending on the information used by the person estimation unit 108 or the next speaker estimation unit 108A, the sensor 103 may be any one of the position measurement device 201, the respiration movement measurement device 202, the gaze target detection device 203, and the head movement detection device 204. Alternatively, a plurality of components may be provided.

  The robot 100 according to the first embodiment and the robot 100A according to the second embodiment include the microphone 101, the camera 102, the sensor 103, the voice input unit 104, the video input unit 105, the sensor input unit 106, and the speech segment detection unit 107. Although the configuration is such that the next speaker estimation unit 108 or the next speaker estimation unit 108A and the speech control unit 109 or the control unit 109A are built in, the present invention is not limited to this configuration. Microphone 101, camera 102, sensor 103, voice input unit 104, video input unit 105, sensor input unit 106, speech section detection unit 107, next speaker estimation unit 108 (or next speaker estimation unit 108A), and speech control unit 109 The speech control device including (or the control unit 109A) may be provided separately from the robot 100 (or the robot 100A). The speech control device is configured to be able to communicate with the robot 100 (or the robot 100A), and transmits the control signal from the speech control unit 109 (or the control unit 109A) to the robot 100 (or the robot 100A). The speech of (or the robot 100A) is controlled.

  The robot 100 and the robot 100A may be configured to display a part of the body on a display unit such as a display, and the whole body is displayed on the display unit as an agent that is a virtual person. It may be To represent a part of the body of the robot 100 and the robot 100A on the display unit, for example, the entire face is a display unit, and a configuration in which an image of the face is displayed on the display unit can be considered. Various expressions can be made by changing the image of the face displayed on the display unit. Like the robot 100, the display device for displaying the agent as the speaker on the display unit includes the microphone 101, the camera 102, the sensor 103, the voice input unit 104, the video input unit 105, and the sensor input unit 106. , An utterance section detection unit 107, a user information acquisition unit 108, an operation control unit 109, a sound control unit 110, a mouth control unit 111, a sight control unit 112, a head control unit 113, and a trunk A control unit 114 and a speaker 115 are provided. The agent is, for example, a person who has a face having an mouth including a mouth and an eye including an eye, and a torso having a hand, an arm, and a foot under the head including the face. According to control signals from the mouth control unit 111, the sight control unit 112, the head control unit 113, and the trunk control unit 114, the display device displays the agent's mouth, eye gaze, head and so on. It further comprises an image processing unit that moves the torso (including hands, arms, legs, etc.). The robot 100 and the robot 100A may not be provided with the plurality of microphones 101 and sensors 103. For example, the robot 100 and the plurality of microphones 101 and sensors 103 installed outside the robot 100A may be wired or wireless. May be configured to transmit and receive signals.

  In the robot 100 in the first embodiment and the robot 100A in the second embodiment, a normal robot other than the functions shown in FIG. 1 and FIG. May be provided. For example, the robot 100 in the first embodiment may be configured to be able to perform the same operation as a human at the time of conversation such as a breathing operation like the robot 100A in the second embodiment.

  Each function part with which robot 100 or robot 100A in this embodiment mentioned above is provided is realizable with a computer, for example. In that case, a program for realizing this function may be recorded in a computer readable recording medium, and the program recorded in the recording medium may be read and executed by a computer system. Here, the “computer system” includes an OS and hardware such as peripheral devices. The term "computer-readable recording medium" refers to a storage medium such as a flexible disk, a magneto-optical disk, a ROM, a portable medium such as a ROM or a CD-ROM, or a hard disk built in a computer system. Furthermore, “computer-readable recording medium” dynamically holds a program for a short time, like a communication line in the case of transmitting a program via a network such as the Internet or a communication line such as a telephone line. It may also include one that holds a program for a certain period of time, such as volatile memory in a computer system that becomes a server or a client in that case. Further, the program may be for realizing a part of the functions described above, or may be realized in combination with the program already recorded in the computer system. It may be realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes design and the like within the scope of the present invention.

  The present invention can be applied to control of a robot that talks with a user, and can be applied to control of movement of an agent (virtual person) displayed on a display device that talks to the user.

51a: right eye, 51b: left eye, 52: mouth, 53: head, 54: neck, 55: trunk, 100, 100A: robot, 101: microphone, 102: camera, 103: sensor, 104: voice input Unit 105 105 Video input unit 106 Sensor input unit 107 Speech section detection unit 108, 108A Next speaker estimation unit 109, 301 Speech control unit 109A Control unit 110, 110A Sound control Unit 111 111 mouth control unit 112 eye gaze control unit 113 head control unit 114 body control unit 115 speaker (sound generation unit) 116 mouth drive unit 117 eye unit drive unit , 118: head drive unit, 119: trunk unit drive unit, 201: position measurement device, 202: respiratory movement measurement device, 203: gaze target detection device, 204: head movement detection device, 30 2 Operation pattern information storage unit 303 Operation control unit 304 Sensor signal conversion unit 401 Speech analysis unit 402 Conversation information generation unit 403, 403A Conversation information DB 404, 404A Speech information generation unit , 405 ... sound signal generator

Claims (4)

An utterance control unit for controlling an utterance of a robot talking with a plurality of users, or an utterance of a speaker displayed on a display device talking with a plurality of users;
A next speaker estimation unit that acquires a first next speaker probability that is a probability that the user will be the next speaker at any time;
Equipped with
The speech control unit controls the speech of the robot or the speaker based on the first next speaker probability of the user acquired by the next speaker estimation unit .
The next speaker estimation unit acquires the first next speaker probability based on the non-verbal behavior of the user,
The robot or the speaker can be controlled to perform an action corresponding to the non-verbal action,
The second speaker estimation unit is a second probability that the robot or the speaker will be the next speaker at the time based on the information on the motion corresponding to the non-verbal behavior of the robot or the speaker. Get next speaker probability,
The utterance control unit is configured to compare the robot or the speaker based on comparison between the first speaker next probability of the plurality of users and the robot or the second speaker next probability of the speaker. A speech control system that controls the speech of the person. The utterance control unit acquires the utterance necessity degree, which is an index indicating the necessity of the utterance of the robot or the speaker according to the utterance content, and the next speaker estimation unit according to the acquired utterance necessity degree The speech control system according to claim 1 , wherein the value of the second next speaker probability acquired by the user is changed. An utterance control unit for controlling an utterance of a robot talking with a plurality of users, or an utterance of a speaker displayed on a display device talking with a plurality of users;
A next speaker estimation unit that acquires a first next speaker probability that is a probability that the user will be the next speaker at any time;
Equipped with
The speech control unit controls the speech of the robot or the speaker based on the first next speaker probability of the user acquired by the next speaker estimation unit .
The next speaker estimation unit acquires the first next speaker probability based on the non-verbal behavior of the user,
The robot or the speaker can be controlled to perform an action corresponding to the non-verbal action,
The second speaker estimation unit is a second probability that the robot or the speaker will be the next speaker at the time based on the information on the motion corresponding to the non-verbal behavior of the robot or the speaker. Get next speaker probability,
The utterance control unit is configured to compare the robot or the speaker based on comparison between the first speaker next probability of the plurality of users and the robot or the second speaker next probability of the speaker. Utterance control device which controls the utterance of. A speech control program for causing a computer to function as the speech control system according to claim 1 or 2.

Images