JP2018185362A - Robot and control method of the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To correspond to speech of a user even during self-speech.SOLUTION: A robot 1 interacting with a user comprises: an image analysis section 106 for analyzing a face image of the user; voice analysis sections 105 and 111 for estimating a sound source direction from ambient voice and performing voice recognition processing; a start determination section 110 for determining start and stop of the voice recognition processing by the voice analysis sections; and a speech generation section 115 for generating a message in accordance with a recognition result of the voice recognition processing and making a speech. The start determination section temporarily stops the speech by the speech generation section and starts the voice recognition processing by the voice analysis sections when a prescribed condition showing that the user is speaking is satisfied from an operation state of the speech generation section, an analysis result of the face image of the user by the image analysis section and an estimation result of the sound source direction by the voice analysis section.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a robot and a control method thereof.

  Robots that interact with users are becoming widespread, but if a user speaks while the robot is speaking, the accuracy of the speech recognition process may be reduced and the conversation may not be established. This is because the speech of the robot is input as noise for the speech recognition processing for recognizing the speech uttered by the user.

  For this reason, when the user utters, the robot utterance is stopped by touching the robot or the like, and then the voice recognition processing of the user voice is executed.

  Although not a robot, an interactive guidance device stops guidance (utterance) of an interactive guidance device when a user's utterance is detected by pressing a talk switch in order to improve the recognition rate of voice recognition processing. (Patent Document 1).

  On the other hand, there is also known a robot equipped with processing such as an adaptive filter and echo cancellation so that speech and voice recognition can be processed simultaneously (Patent Document 2).

  In addition, although it is not a robot, the navigation apparatus which determines a user's speech start by detecting the motion of a user's mouth is also known (patent document 3).

Japanese Patent No. 3654045 Japanese Patent No. 5178370 JP 2009-98217 A

  In the prior art described in Patent Document 1, when there is an instruction to start voice input by some operation, the volume of the guidance device is reduced or silenced, and the permission of voice input is notified when the noise estimation section ends. However, every time the user wants to speak, it is necessary to stop the utterance of the guidance device itself (self-speaking) by performing some operation on the guidance device. Prone to unnatural dialogue.

  In the prior art described in Patent Document 2, a signal by self-speech among signals input to speech recognition processing can be removed using an adaptive filter or echo cancellation, so that the user's speech is recognized even during self-speech. can do. However, it is difficult to accurately recognize the voice of the user who is currently speaking even if processing such as an adaptive filter or echo cancellation is used, and implementation of the adaptive filter or echo cancellation takes time and costs. In particular, in addition to basic control processing such as image processing (face recognition), voice recognition, and motion, processing such as adaptive filters is added to inexpensive and small robots with limited resources such as memory size and CPU (Central Processing Unit). Is not easy to implement.

  In the prior art described in Patent Literature 3, since the voice recognition process is started by detecting the user's speech movement (movement of the mouth), a special operation for starting the voice recognition process is unnecessary. However, it is difficult to detect the user's utterance only by the movement of the user's mouth.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a robot capable of interacting in response to a user's utterance even during his / her utterance and a control method thereof.

  A robot according to one aspect of the present invention is a robot that interacts with a user, an image analysis unit that analyzes a user's face image, and a voice analysis unit that performs sound source direction estimation and voice recognition processing from surrounding sounds And an activation determination unit that determines the start and stop of voice recognition processing by the voice analysis unit, and an utterance generation unit that generates a message and utters according to the recognition result of the voice recognition process. When the predetermined condition indicating that the user is speaking is satisfied based on the operation state of the generation unit, the analysis result of the face image of the user by the image analysis unit, and the estimation result of the sound source direction by the voice analysis unit, the utterance generation unit The speech is paused and the speech recognition process by the speech analysis unit is started.

  The predetermined condition is an image in which the utterance generation unit is speaking, the user's face is located in front, and the analysis result of the user's face image by the image analysis unit indicates the operation of the user's mouth, and The estimation result of the sound source direction by the voice analysis unit may indicate the front.

  The movable part movable with respect to a main-body part may be provided, and the starting determination part may implement | achieve the predetermined speech recognition start operation | movement which shows the start of a speech recognition process in a movable part.

  During the operation of the utterance generation unit, a speech recognition stop operation that indicates the stop of the speech recognition process may be realized by operating the movable unit.

  The voice recognition start operation may be to stop the voice recognition stop operation.

  If the utterance generation unit can generate a message corresponding to the result of the speech recognition processing by the speech analysis unit, the message is uttered, a new conversation with the user is started, and the message corresponding to the result of the speech recognition processing by the speech analysis unit If the utterance generation unit cannot generate the utterance, the utterance generation unit may resume the paused utterance.

  A robot control method according to another aspect of the present invention is a method in which a robot that interacts with a user is controlled by a robot controller, and the robot controller monitors whether the user has spoken during the utterance of the robot. When an utterance is detected, the utterance is paused, and at the same time, the voice recognition stop operation indicating that the voice recognition processing is stopped is paused by moving at least the movable part of the robot, which is performed during the robot's utterance. The voice to be uttered is acquired, the voice of the acquired user is subjected to voice recognition processing, it is determined whether a message corresponding to the result of the voice recognition processing can be generated, and if it is determined that the message can be generated, the message is uttered and the user When it is determined that a new conversation is started and a message cannot be generated, the paused speech is resumed and voice recognition is stopped. To resume the work.

The figure which shows the whole structure of a robot. The flowchart which shows a process when a user talks during robot speech. Explanatory drawing which shows typically the relationship between the speech of a robot, and a user's talk. The flowchart of the process which concerns on 2nd Example. The flowchart of the process which concerns on 3rd Example. It is explanatory drawing which shows an example which concerns on 4th Example and a user's face is located in the front of a robot. It is explanatory drawing which shows the other example in which a user's face is located in front of a robot, and the example which is not located.

  In the present embodiment, as will be described later, the user's mouth is moving and the voice comes from the user's direction while the robot 1 is speaking while looking at the front of the user's (speaker) face. Are simultaneously detected, the utterance of the robot 1 is temporarily stopped and voice recognition is started. Thereby, even when the user interrupts and speaks while the robot 1 is speaking (during self-speaking), it is possible to recognize the user's speech and continue a natural conversation. Hereinafter, temporarily stopping the speech of the robot 1 is expressed as “pause” or “interrupt”.

  FIG. 1 shows the overall configuration of the robot 1. The robot 1 is configured as a so-called communication robot that can communicate with a user. The robot 1 can interact with a user or exercise and play in, for example, a general home, an office, various commercial facilities, a hospital, a care facility, a nursery school, a kindergarten, and a school. Hereinafter, a person who communicates with the robot 1 through dialogue is referred to as a user.

  The robot 1 includes one or a plurality of applications (services). Examples of applications include news, recreation, quizzes, games, gymnastics, and dance. When the user issues a command associated with the application, the robot 1 recognizes the voice and executes the application corresponding to the command.

  The robot 1 can be roughly divided into a robot control unit 10 and a robot body 11. The robot control unit 10 controls the robot body 11, and details thereof will be described later. The robot body 11 includes, for example, a body 12, legs 13R and 13L, hands 14R and 14L, and a head 15. Hereinafter, when the left and right are not distinguished, they are referred to as both legs 13 and both hands 14. The body 12 corresponds to a “main part”. Both the legs 13, both hands 14, and the head 15 are provided so as to be movable with respect to the body 12 and correspond to “movable parts”.

  The robot control unit 10 is provided in the robot body 11. The robot 1 is provided with a camera 121 and a microphone 120 described later.

  The robot control unit 10 includes, for example, a microprocessor (hereinafter referred to as CPU) 101, a memory 102, an SSD (Solid State Drive) 103, an integrated control unit 104, a voice signal processing unit 105, an image recognition unit 106, a voice synthesis unit 107, a voice output, and the like. Unit 108, LED (Light Emitting Diode) drive unit 109, voice recognition activation determination unit 110, voice recognition unit 111, dictionary database 112, acoustic model 113, language model 114, utterance generation unit 115, utterance database 116, actuator control unit 117 , An actuator driving unit 118, a motion database 119, a communication unit (not shown), a power supply unit, and the like.

  The CPU 101 reads and executes computer programs and operation control data stored in the memory 102 or the SSD 103, thereby executing applications such as news, recreation, quiz, dance, and gymnastics. In this embodiment, the SSD 103 is used as the auxiliary storage device, but a storage device other than the SSD may be used.

  When the voice recognition result of the voice recognition unit 111 is a command, the integrated control unit 104 starts the operation in accordance with the command, corresponding processing units such as the LED driving unit 109, the utterance generation unit 115, and the actuator control unit 117. It is a function to instruct to. If the result of speech recognition is a word other than the command, the integrated control unit 104 sends the user's word to the utterance generation unit 115.

  The voice signal processing unit 105 and the voice recognition unit 111 correspond to a “voice analysis unit”. The audio signal processing unit 105 acquires sound signals from the plurality of microphones 120 and analyzes them. For example, one microphone 120 is provided on each of the robot body 11 before and after the portion corresponding to the left and right ears and the portion corresponding to the neck. That is, the robot 1 is provided with a plurality of microphones 120 at different locations so that the direction of the sound source can be estimated.

  The audio signal processing unit 105 includes, for example, a feature vector extraction unit 1051 and a sound source direction estimation unit 1052. In the figure, the word “part” is omitted as appropriate.

  The feature vector extraction unit 1051 extracts a feature amount for speech recognition from the speech acquired by the microphone 120 as a feature vector. The extracted feature vector is sent to the speech recognition unit 111. Note that the feature vector extraction unit 1051 can perform beam forming using the estimation result of the sound source direction by the sound source direction estimation unit 1052, and can generate a feature vector based on a signal that emphasizes speech coming from the sound source direction. . Thereby, the feature amount of the voice of the user talking to the robot 1 can be extracted more clearly.

  The sound source direction estimation unit 1052 estimates the direction of the sound source by analyzing the time difference between the sound waves from the sound source reaching each microphone 120. The estimation result of the direction of the sound source is sent to the speech recognition activation determination unit 110. For example, the direction of the sound source can be estimated by calculating the arrival time difference between the audio signals received by the two microphones 120 mounted on the left and right ears of the robot body 11 (delay time estimation method). Furthermore, it is possible to distinguish the front and rear in the direction in which the voice arrives based on the strength of the voice signal received by the two microphones 120 mounted before and after the neck of the robot body 11. Note that the sound source direction may be estimated by calculating the arrival time difference of the audio signals received by all four microphones 120.

  The image recognition unit 106 corresponds to an “image analysis unit” and analyzes image data (hereinafter also referred to as an image) captured by the camera 121. The analysis result of the image recognition unit 106 is sent to the voice recognition activation determination unit 110. For example, at least one camera 121 is provided in front of the robot head 15 (an area corresponding to the face of the robot). The image recognition unit 106 includes, for example, a face detection unit 1061 and a mouth movement detection unit 1062.

  The face detection unit 1061 extracts the user's face (human face) from the image acquired from the camera 121. Specifically, the face detection unit 1061 outputs a face area detected by a rectangle, coordinates indicating the positions of the nose and mouth, and a face direction.

  The mouth movement detection unit 1062 detects the presence / absence of movement of the user's mouth based on images continuously captured by the camera 121. Specifically, it is determined whether or not there is a movement in the user's mouth from the continuously captured images using the coordinates of the nose and mouth detected by the face detection unit 1061.

  The voice synthesizer 107 synthesizes a voice signal corresponding to a message (such as a response sentence) input from the utterance generator 115. The voice signal synthesized by the voice synthesizer 107 is output from the speaker 122 to the outside via the voice output unit 108.

  The LED driving unit 109 drives the LED 123 provided in the robot body 11. For example, at least one LED 123 can be provided in front of the robot head 15 (an area corresponding to a face). Instead of this, for example, LEDs may be provided on the back surface of the robot head 15, the periphery of the neck, the chest of the body 12, and the like. Furthermore, instead of the LED 123 or together with the LED 123, a liquid crystal display, a display unit such as an OLED (Organic Light Emitting Diode), and a light emitting unit may be provided.

  The speech recognition activation determination unit 110 corresponds to an “activation determination unit”. Hereinafter, it is also referred to as an activation determination unit 110. The activation determination unit 110 estimates the user (speaker) status based on the sound received from each microphone 120 and the image captured by the camera 121, and determines whether the speech recognition process is started or stopped. Based on the determination result of the activation determination unit 110, the feature vector extraction unit 1051 and the speech recognition unit 111 start or stop their operations.

  The speech recognition unit 111 recognizes speech based on feature vectors acquired from the speech signal processing unit 105. The voice recognition unit 111 can use, for example, a dictionary database 112, an acoustic model 113, and a language model 114.

  The acoustic model 113 is a database in which text reading and a waveform when the text is pronounced are stored in association with each other, and defines what kind of waveform sound is recognized as what word. The language model 114 is a database that stores word arrangement (grammar) and the like for each language. The dictionary database 112 holds general dictionary data. For example, by preparing the dictionary database 112, the acoustic model 113, and the language model 114 for each language, it is possible to support multiple languages.

  The utterance generation unit 115 generates a response sentence (message) corresponding to the words recognized by the voice recognition unit 111 and outputs the response sentence to the outside from the speaker 122 via the voice synthesis unit 107 or the like. Specifically, the utterance generation unit 115 refers to the utterance database 116, selects a sentence example that responds to the recognized word, and generates a response sentence from the selected sentence example.

  The actuator control unit 117 is connected to the plurality of actuators 124 via the actuator driving unit 118 and controls each actuator 124. Examples of the actuator 124 include a DC servo motor for moving the joints of each part. For example, an ultrasonic motor, a piezoelectric actuator, a solenoid, or the like may be used as the actuator. Hereinafter, a joint motor will be described as an example of an actuator. Therefore, the joint motor may be referred to as a joint motor 124 by adding a reference numeral 124.

  The actuator control unit 117 refers to the motion database 119 to control the joint motor 124 of each unit in order to realize the operation instructed by the integrated control unit 104. In the motion database 119, control information (rotation angle, rotation time, rotation speed, sequence) of the joint motor 124 corresponding to various operations is registered.

  Here, when the activation determination unit 110 determines that the voice recognition process should be started, the integrated control unit 104 instructs the actuator control unit 117 to execute the voice recognition start motion 1191. On the other hand, when the activation determination unit 110 determines that the voice recognition process should be stopped, the integrated control unit 104 instructs the actuator control unit 117 to execute the voice recognition stop motion 1192.

  The voice recognition start motion 1191 is a motion executed at the timing of starting the voice recognition process, and corresponds to a “voice recognition start operation”. The voice recognition stop motion 1192 is a motion that is executed at the timing of stopping the voice recognition processing (that is, the timing of starting speech), and corresponds to the “voice recognition stop operation”. Since the voice recognition stop motion 1192 is a motion performed during the self-speaking of the robot 1, it can also be referred to as a speech motion.

  In the voice recognition start motion 1191 and the voice recognition stop motion 1192, for example, the user can be alerted by moving the movable part of the robot body 11 or turning on the LED 123.

  The user can learn the start timing and stop timing of the voice recognition processing through the operation of the robot 1 through experience. By visually recognizing the voice recognition start motion 1191, the user can know that the voice recognition processing has started, and can speak at an appropriate timing. By visually recognizing the voice recognition stop motion 1192, the user can know that the voice recognition process has been stopped. Even a user who is less interested in the performance of the robot 1 can experience whether or not his / her utterance has been recognized and the relationship between these motions 1191 and 1192. Can be expected to learn naturally.

  An example of the voice recognition start motion 1191 will be described. In the voice recognition start motion 1191, for example, the arm 14 is moved during self-speaking, and when the voice recognition process starts, the movement of the arm 14 is stopped to notify the start of the voice recognition process. Instead of stopping the operation of the arm 14 or simultaneously with stopping the operation of the arm 14, the start of the voice recognition process may be notified by giving a gesture to listen to the user of the sound source.

  An example of the voice recognition stop motion 1192 will be described. In the voice recognition stop motion 1192, for example, the arm 14 is not moved during the voice recognition process, and the movement of the arm 14 is started when the voice recognition process is stopped and the self-speaking is started. Notify that the speech recognition process has stopped. Instead of starting the operation of the arm 14 or simultaneously with the start of the operation of the arm 14, the gesture that is listening to the user of the sound source during the speech recognition process may be stopped and returned to the normal state. It is also possible to notify the user of the start or stop of voice recognition by operating a movable part other than the arm 14, for example, the leg 13 or the head 15.

  In the voice recognition start motion 1191, as described above, the operation of the arm 14 as the movable portion (for example, the operation of swinging the arm up and down, also referred to as the arm swing motion) is stopped, but the joint motor 124 is moved to a predetermined position. The joint motor 124 may be stopped in the brake mode.

  In the voice recognition start motion 1191 of the present embodiment, the H bridge inside the joint motor 124 configured as a DC servo motor is short-circuited to function as a generator instead of a motor. Thereby, the noise at the time of stop control of a DC servo motor can be reduced, and the fall of the precision of voice recognition processing can be controlled. Furthermore, when stopping control of the DC servo motor, it is necessary to specify the angle to be stopped and instruct driving, and the control processing is complicated and takes time, but in the brake mode, only the H bridge is short-circuited. Well, it can be stopped easily and quickly. Furthermore, in the brake mode, since a load is required to rotate the DC servo motor, it is possible to suppress the amount that the arm 14 is naturally rotated by gravity.

  With reference to FIG. 2, the dialogue control process according to the present embodiment will be described. The overall flow will be described later in another embodiment.

  First, it is assumed that the robot 1 is speaking in the initial state (S10). It is assumed that the robot 1 reads out news or is talking to the user, for example. During the speech of the robot 1, that is, during the self-speech, a speech motion (voice recognition stop motion 1192) is executed.

  The robot control unit 10 monitors whether a predetermined condition for determining whether the user talks to the robot 1 is satisfied (S11). The predetermined condition is, for example, that the user's face is directed in a predetermined direction, a movement is detected in the user's mouth, and the direction of the sound source detected by the microphone 120 is in front of the robot 1. There is. Specifically, this is a case where the user's face is in front of the robot 1, the user's mouth is moving, and voice is detected from the front front of the robot 1. Here, in the present specification, the front front of the robot 1 is not based on the body of the robot 1 but refers to a direction (viewing angle shooting range) in which the camera 121 mounted on the robot 1 can shoot. An example of “when the user's face is located in front” will be described later with reference to FIGS.

  When the predetermined condition is satisfied, it can be estimated that the user has started speaking while the robot 1 is speaking. When the detection timings of the face direction, the mouth movement, and the sound source direction coincide with each other, it is determined that a predetermined condition is satisfied (S11). This is to suppress false detections and prevent the conversation from being interrupted. Whether the timings match can be determined, for example, by determining whether the difference between the time stamp of the audio data and the time stamp of the image data falls within a predetermined time.

  When the robot control unit 10 determines that the predetermined condition is satisfied (S11: YES), the robot control unit 10 notifies the user that the voice recognition process has been started using the expression ability of the robot 1 (S12). For example, the robot control unit 10 pauses self-speaking, stops the utterance motion, performs a short inquiry operation to the user, performs an LED display indicating a voice input wait, and transmits a voice through each microphone 120. Collect (S12).

  Here, the stop of the speech motion means that the speech recognition stop motion 1192 is stopped. In a short questioning operation to the user, for example, a short word such as “n?” Is issued, and even if it is suddenly issued during the dialogue, a word that seems not to disturb the flow of the dialogue is emitted. In the LED display indicating voice input waiting, for example, the LED 123 is lit blue to notify the user that voice input is waiting.

  The robot control unit 10 determines whether voice can be collected from each microphone 120 in a state where the user's face is visible in front of the robot 1 (S13). When the voice cannot be collected while the user's face is visible in front of the robot 1 (S13: NO), the robot control unit 10 determines whether a predetermined time t1 has elapsed from the start of voice collection (S14).

  The predetermined time t1 can be set to about several seconds such as a few seconds. This is because when the silent state continues, the utterance (self-utterance) of the robot 1 temporarily stopped in step S12 is immediately resumed. By setting the predetermined time t1 short, it is possible to prevent the conversation from being interrupted for a long time when the predetermined condition is satisfied by chance, and it is possible to naturally return to the paused utterance.

  The robot controller 10 repeats step S13 until a predetermined time t1 has elapsed since the start of voice collection (S14: NO). If the voice cannot be collected even after the predetermined time t1 has elapsed (S14: YES), the robot control unit 10 resumes the self-speech suspended in step S12 (S15), and returns to step S10.

  In step S15 for resuming self-speech, the speech motion (speech recognition stop motion 1192) is resumed, and the LED 123 is lit red, for example, to inform the user that the speech recognition process has been stopped.

  On the other hand, when the robot control unit 10 can collect voice from each microphone 120 in a state where the user's face is visible in front of the robot 1 before the predetermined time t1 elapses (S13: YES), voice recognition processing is performed. Is started (S16).

  Although not shown, it is not necessary to perform all of the speech recognition processing inside the robot 1, and at least a part of the processing may be executed by a speech recognition processing server provided outside the robot 1. . When all or part of the voice recognition process is executed by an external server, it takes some time until the result of the voice recognition process is obtained. Therefore, in step S16, an operation of moving the robot head 15 by tilting it forward (whipping motion) may be executed. When the robot 1 performs the whispering motion, even when the voice recognition processing takes time, it is possible to keep a gap and continue natural conversation. Even when all of the voice recognition processing is executed in the robot 1, the whispering motion is executed during the voice recognition processing, so that the user is directed to listen to the user's words. be able to.

  The robot control unit 10 determines whether a voice recognition result has been obtained (S17). When the voice recognition result is not output (S17: NO), the robot control unit 10 determines whether the predetermined time t2 has elapsed from the start of the voice recognition process in step S16 (S18). The robot control unit 10 waits for a predetermined time t2 until a voice recognition result is obtained.

  The predetermined time t2 can be set to a value (for example, about 10 seconds) longer than the predetermined time t1 described in step S14. Under a situation where there is a high possibility that the user is talking to the robot 1, by waiting for a relatively long time t2, it is possible to accept a talk (interrupt) from the user and lead to a new conversation. . If the waiting time t2 in step S18 is set short, there is a high possibility that the operation that the user tried to talk to is canceled, and there is a possibility that natural dialogue will be hindered.

  When the speech recognition result is obtained before the predetermined time t2 has elapsed (S17: YES), the robot control unit 10 determines whether the utterance generation unit 115 can generate a message corresponding to the speech-recognized word. (S19).

  Specifically, the robot control unit 10 recognizes speech capable of effective conversation by performing morphological analysis on the speech-recognized word and determining whether or not a keyword matching the result of the morphological analysis is stored in the speech database 116. It is determined whether or not (S19).

  When the speech-recognized word includes a keyword stored in the utterance database 116 (S19: YES), the robot control unit 10 starts a new conversation with the user based on the recognized speech (S20).

  On the other hand, the robot control unit 10 could not recognize the speech capable of effective conversation when the words recognized in step S16 do not include any keywords stored in the utterance database 116. In the case (S19: NO), the process proceeds to step S15, and the speech of the robot 1 that has been paused is resumed.

  FIG. 3 is a time chart showing one specific example of the processing described in FIG. Although illustration is omitted, the user first makes a statement such as “Tell me about tomorrow's weather”, and the robot 1 responds to this statement by collecting weather forecast data from a weather forecast server or the like on the network. An example of the case will be described.

  In case (a), it is assumed that the robot 1 reads, for example, a news such as “This morning in the Kanto region seems to be sunny” as a self-utterance W1. The robot 1 can appropriately acquire information from an information source such as an external news distribution site using a communication function (not shown).

  In the case (b), when a situation in which the user starts talking to the robot 1 appears by chance during the self-utterance W1 of the robot 1, that is, the predetermined condition described in step S11 in FIG. 2 is temporarily established. Shows the case.

  In the case (b), when a predetermined condition is satisfied, the robot 1 temporarily suspends the self-utterance W1 and issues a short response “?” (S12 in FIG. 2). At the same time, the robot 1 stops the voice recognition stop motion 1192 such as moving the arm 14 up and down and starts the voice recognition process (S12). However, after the voice recognition process is started, if the silent time during which no voice can be acquired continues for a predetermined time t1 (YES in S14 in FIG. 2), the voice recognition process is stopped and the suspended utterance W1a is resumed from the subsequent W1b ( S15 of FIG. 2).

  In other words, even if it is erroneously determined that the user has spoken, as in case (b), the robot 1 returns a very short and natural response of “n?” And reacts as if the ears are stood away for a moment. Return to the original utterance promptly after showing. Therefore, even when the utterance of the robot 1 is divided into the pre-interruption part W1a and the post-interruption part W1b, the possibility of giving the user unnaturalness can be reduced.

  Case (c) shows a case where a predetermined condition is satisfied, a response “n?” Is issued, and voice recognition processing is started, but no recognizable voice can be collected within a predetermined time t2 (FIG. 2). NO at S17 and YES at S18).

  In the case (c), after a predetermined time t2 has elapsed from the start of the speech recognition process, the robot 1 resumes the suspended utterance from the suspended position (S15). Note that switching between the voice recognition stop motion 1192 and the voice recognition start motion 1191 has been described with reference to FIG.

  In the case (d), when a predetermined condition is satisfied and a response “n?” Is issued and the speech recognition processing is started, but speech capable of valid conversation cannot be collected, that is, in the speech database 116. The case where the voice corresponding to the registered keyword could not be recognized is shown (YES in S17 of FIG. 2, NO in S19). If the keyword cannot be extracted from the user's voice, the robot 1 resumes the suspended utterance from the suspended position (S15).

  Case (e) shows a case where the robot 1 turns to a new topic in response to a conversation from the user detected during the speech of the robot 1.

  After the robot 1 issues a short response “No?”, If the voice collected from the user includes a keyword capable of effective conversation (YES in S19 in FIG. 2), the robot 1 talks to the user. A new conversation W2 is started according to the received voice (S20 in FIG. 2). The remaining portion W1b of the first utterance W1 is not uttered.

  In the case (e), it is possible to move to a new topic in response to the conversation from the user, and smooth communication can be continued.

  As described above, during the speech recognition, the robot 1 of the present embodiment (1) cannot detect the user's face or voice (S13: NO), and (2) cannot detect the detected voice (S17: NO). (3) When the conversation corresponding to the speech-recognized word cannot be made (S19: NO), the speech recognition process is stopped and the speech recognition start motion 1191 is switched to the speech recognition stop motion 1192.

  When a new conversation is started (S20), the speech recognition process is stopped while the robot 1 is speaking. After the utterance of the robot 1 is finished, the voice recognition stop motion 1192 is switched to the voice recognition start motion 1191, and voice recognition is started.

  According to this embodiment configured as described above, when a predetermined condition is satisfied (the user's mouth is moving during self-speaking while looking in front of the user's face as a speaker), and the robot 1 When the voice is input from the front of the robot 1), the speech of the robot 1 is paused and the voice recognition process is started. Therefore, according to the present embodiment, it is possible to respond naturally to the conversation from the user during the self-speaking without using the function (barge-in function) capable of recognizing the voice during the self-speaking, and perform smooth communication. Can do.

  According to the present embodiment, even an inexpensive robot that is difficult to provide a barge-in function due to large resource constraints can cope with a conversation from a user who is speaking. Therefore, it is possible to improve the interactive performance of the robot 1 without increasing the cost so much.

  According to the present embodiment, when all the conditions of the user's face direction, mouth movement, and voice arrival direction are all determined, it is determined that the user is speaking toward the robot 1, so that the user's utterance is accurate. It can detect well and follow changes in the topic.

  According to the present embodiment, the speech recognition start motion 1191 indicating that speech recognition is enabled when the speech of the robot 1 is paused and the speech recognition process is started is performed. For this reason, the user can naturally learn the timing at which the robot switches from the speech mode (voice recognition stop mode) to the voice recognition mode. Through the experience that the user does not recognize the voice until the mode is changed to the voice recognition mode, the user can use an utterance method (interrupt method) according to the performance of the robot 1 such that it is wasted even if an important word is suddenly spoken during the utterance of the robot 1. Can learn. As a result, it is possible to expect an effect that the user can naturally and smoothly interact with the robot.

  In this embodiment, the voice recognition start motion 1191 is executed to notify the start of the voice recognition process, and the movement of the part (for example, the arm 14) moved during the speech is stopped. Thereby, according to the present embodiment, it is possible to notify the user that the mode has been switched in a natural dialogue operation.

  According to the present embodiment, the speech recognition result of words uttered by the user during the self-speaking of the robot 1 is the topic range (the keyword range stored in the utterance database 116) that the robot 1 can respond to. In the case of responding to the new topic, on the other hand, if it is not within the range of the topic that can be responded or if speech recognition is not possible, the paused self-utterance is resumed from the continuation. As a result, the conversation can be continued according to the interest of the mobile user, and when the user emits an unclear sound, the communication returns to the original topic without disrupting the conversation tempo. Can continue.

  According to the present embodiment, when the speech recognition process is stopped and the utterance of the robot 1 is started, the user executes the speech recognition stop motion 1192 indicating that the user's speech is not recognized. The timing at which the robot 1 switches from the voice recognition mode to the speech mode (voice recognition stop mode) can be naturally learned.

  According to the present embodiment, in the voice recognition start motion 1191, the movable part such as the arm 14 is stopped, so that the operation sound of the joint motor 124 is prevented from being collected by the microphone 120, and the voice recognition accuracy is improved. it can.

  According to the present embodiment, when the arm 14 is stopped, the joint motor 124 is stopped in the brake mode. Therefore, the stopped state can be maintained against gravity without consuming electric energy.

  A second embodiment will be described with reference to FIG. Each of the following embodiments including this embodiment corresponds to a modification of the first embodiment, and therefore, differences from the first embodiment will be mainly described. In this embodiment, according to the number of users around the robot 1, the dialogue control method described in the first embodiment, that is, the dialogue processing corresponding to the case where the user speaks during the self-speaking of the robot 1 (hereinafter, referred to as the dialogue processing) Controls the activation of the interrupt handling dialog control process).

  FIG. 4 shows a part of processing executed by the robot 1 according to the present embodiment. The robot control unit 10 acquires the environment around the robot 1 from the image captured by the camera 121 and the sound collected by the microphone 120 (S30). For example, the robot control unit 10 acquires the image of the user existing around the robot 1 by looking around the robot 1 by rotating the head 15 equipped with the camera 121 in a horizontal direction by a predetermined angle.

  The robot control unit 10 detects a candidate user who can be a conversation target from the image acquired in step S30 (S31). For example, the robot control unit 10 estimates the distance from the robot 1 based on the size of the user's face image and the like, and is a user who is located within a predetermined distance from the robot 1 and exists near the front of the robot 1. Then, it is detected as a user candidate who can talk with the robot 1 (S31).

  The robot control unit 10 determines whether the number of users detected in step S31 is equal to or less than a preset threshold value ThU (S32). When the number of users who can talk is equal to or less than the threshold value ThU (S32: YES), the robot control unit 10 sets the interrupt handling dialogue control process (S10 to S20 in FIG. 2) described in the first embodiment to be executable. (S33).

  On the other hand, when the number of users who can talk is larger than the threshold value ThU (S32: NO), the robot control unit 10 skips step S33. Therefore, the robot 1 cannot execute the interrupt handling dialogue control process.

  After determining whether or not to execute the interrupt-corresponding dialog control process, the robot control unit 10 interacts with the user (S34). When the user utters a command requesting execution of an application such as “quiz” or “dance”, the robot control unit 10 executes the application corresponding to the command.

  Configuring this embodiment like this also achieves the same operational effects as the first embodiment. Furthermore, in this embodiment, when the number of users who can interact with the robot 1 is equal to or less than the threshold value ThU, the interrupt handling dialog control process described in the first embodiment can be executed. Therefore, when the robot 1 performs “recreation” or the like with more users than the threshold ThU, it is possible to suppress the progress of the moderator of the robot 1 from being disturbed by the speech of surrounding users. .

  A third embodiment will be described with reference to FIG. In this embodiment, whether or not to execute the interrupt-compatible dialog control process is determined according to the type of application.

  FIG. 5 shows a part of processing executed by the robot 1 according to the present embodiment. The robot control unit 10 determines the mode (S40). Here, for example, it is assumed that there are a free conversation mode (S41, S42), a news mode (S43, S44), a recreation mode (S45, S46), a charge request mode (S47, S48), and the like.

  In the free conversation mode, the robot 1 has a free conversation with the user. The news mode is performed when the user utters a command such as “read news”. In the news mode, the robot control unit 10 acquires news from a news distribution server (not shown) and reads out the acquired news.

  The recreation mode can be set by a manager of the robot 1 (for example, a robot person in charge at a facility where the robot 1 is installed) by determining the date and time in advance. The recreation mode is executed in, for example, a nursing facility, a school, a hospital, a supermarket, a department store, an amusement park, and the like. In the recreation mode, the robot 1 serves as a moderator and executes programs such as “gymnastics”, “choral”, and “quiz competition”.

  The charge request mode is executed when the SOC (State Of Charge) of the storage battery (not shown) of the robot 1 is lowered to a predetermined threshold value and charging is necessary. When the charging request mode is set, the robot control unit 10 notifies the administrator that the charging is necessary by turning on the LED 123 or the like.

  In the free conversation mode (S41: YES), the robot control unit 10 freely interacts with the user in a state where the interrupt-corresponding dialog control process can be executed (S42).

  In the news mode (S43: YES), the robot control unit 10 reads the news in a state where the interrupt-corresponding dialogue control process can be executed (S44).

  In the recreation mode (S45: YES), the robot control unit 10 serves as a recreation moderator without executing the interrupt-corresponding dialogue control process (S46).

  In the charge request mode (S47: YES), the robot control unit 10 requests the administrator to charge without executing the interrupt handling dialogue control process (S48).

  Configuring this embodiment like this also achieves the same operational effects as the first embodiment. Furthermore, in the present embodiment, the robot 1 can communicate with the user according to the situation because it determines whether or not to execute the interrupt-corresponding dialog control process according to the mode (according to the type of application).

  For example, in the recreation mode, many users are often dealt with. Therefore, if the interrupt-corresponding dialogue control process is set to be executable, the moderation of the robot 1 is frequently interrupted by the user's speech, and smooth Recreational activities may not be possible.

  In the charge request mode, since the remaining amount of the storage battery is low, the remaining amount of the storage battery is lost earlier if the user responds to the talks one by one, and the function of the robot 1 may stop.

  However, in this embodiment, since the robot 1 is controlled in a mode in which the interrupt-corresponding dialogue control process should be executed and a mode in which it is not executed, smooth communication is realized according to the situation where the robot 1 is placed. Can do.

  A fourth embodiment will be described with reference to FIGS. In this embodiment, the “predetermined condition” will be described in detail.

  As described above, when a predetermined condition indicating that the user is speaking is satisfied, the utterance by the utterance generation unit 115 is temporarily stopped, and the voice recognition process by the voice analysis unit (105, 111) is started.

  As described above, the predetermined condition is that the utterance generation unit 115 is speaking and the user's face is positioned in front, and the analysis result of the user's face image by the image recognition unit 106 as the image analysis unit is This is an image showing the motion of the mouth and the estimation result of the sound source direction by the speech analysis unit indicates the front.

  As shown in FIG. 6, “the state where the user's face is located in front” is a state where the user is looking at the robot 1. Specifically, the state in which the user is looking at the robot 1 is a state in which the front of the user's face is facing the direction of the robot 1 in a range that can be photographed by the camera 121 of the robot 1 (field angle photographing range). . Since the camera 121 of the robot 1 only needs to be able to capture the face in front of the user, not only when the user is positioned directly in front of the robot 1 (A in FIG. 6), but also within a field angle shooting range even at positions other than the front. The case where the face in front of the user is located (B, C in FIG. 6) is also included.

  As shown in the explanatory diagram of FIG. 7, when the user's face is located within the field angle shooting range but the user is not facing the robot 1 (D in FIG. 7), “the user's face is the front It is not included in the “state located at”. Further, when the user faces the robot 1 but is out of the field angle shooting range (E in FIG. 7), it is not included in “when the user's face is located in front”.

  Therefore, a specific example of “when the user's face is located in front” corresponds to A to C in FIG. 6 and A ′ and B ′ in FIG. 7, and D and E in FIG. 7 do not correspond. Although not shown, even when the user within the field-of-view photographing range is turning his back to the robot 1 or when the user is looking up at the ceiling or looking at the floor, “when the user's face is in front” Not applicable.

  The present invention is not limited to the above-described embodiment. A person skilled in the art can make various additions and changes within the scope of the present invention. The robot does not need to be humanoid, and may be a geometric object such as an animal such as a dog, a cat, a bird, or a fish, a plant such as a sunflower or corn, a cylinder, a polygon, or a cube.

  1: Robot, 10: Robot controller, 11: Robot body, 12: Body, 13L, 13R: Leg, 14L, 14R: Hand, 15: Head, 104: Integrated controller, 105: Audio signal processor, 106 : Image recognition unit, 110: voice recognition activation determination unit, 111: voice recognition unit, 115: utterance generation unit, 116: utterance database, 119: motion database

Claims (8)

A robot that interacts with the user,
An image analysis unit for analyzing a user's face image;
A speech analysis unit that performs sound source direction estimation and speech recognition processing from surrounding speech,
An activation determination unit that determines start and stop of the speech recognition processing by the speech analysis unit;
An utterance generator that generates a message and utters according to the recognition result of the voice recognition process;
With
The activation determination unit is a predetermined state indicating that the user is speaking from the operation state of the utterance generation unit, the analysis result of the user's face image by the image analysis unit, and the estimation result of the sound source direction by the voice analysis unit. If the condition is satisfied, temporarily stop the utterance by the utterance generation unit, and activate the speech recognition processing by the speech analysis unit,
robot. The predetermined condition is an image in which the utterance generation unit is uttering and the user's face is located in front, and the analysis result of the user's face image by the image analysis unit indicates the operation of the user's mouth. There is a case where the estimation result of the sound source direction by the voice analysis unit indicates the front,
The robot according to claim 1. It has a movable part that can move relative to the main body,
The activation determination unit causes the movable unit to realize a predetermined voice recognition start operation indicating the start of the voice recognition process.
The robot according to claim 2. During the operation of the utterance generation unit, a speech recognition stop operation indicating stop of the speech recognition process is realized by operating the movable unit.
The robot according to claim 3. The voice recognition start operation is to stop the voice recognition stop operation.
The robot according to claim 4. If the utterance generation unit can generate a message corresponding to the result of the voice recognition processing by the voice analysis unit, utter the message and start a new conversation with the user,
If the utterance generation unit cannot generate a message corresponding to the result of the speech recognition processing by the voice analysis unit, the utterance generation unit resumes the paused utterance,
The robot according to claim 1 or 2. It has a movable part that can move relative to the main body,
The activation determination unit, when the utterance generation unit resumes the paused utterance, realizes a voice recognition stop operation indicating the stop of the voice recognition processing by operating the movable unit,
The robot according to claim 6. A method of controlling a robot that interacts with a user by a robot control unit,
The robot controller is
Monitoring whether the user speaks during the utterance of the robot,
When the user's utterance is detected, the utterance is paused and the voice recognition stop operation indicating that the voice recognition process is stopped by moving at least the movable part of the robot, which is performed during the utterance of the robot, is paused. Let
Get the voice that the user utters,
Voice recognition processing is performed on the acquired user voice,
Determining whether a message corresponding to the result of the voice recognition process can be generated;
If it is determined that the message can be generated, utter the message to start a new conversation with the user,
When it is determined that the message cannot be generated, the speech that has been paused is resumed and the speech recognition stop operation is resumed.
Robot control method.

Images