JP2006251266A - Audio-visual coordinated recognition method and device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately detect a person, to whom conversation is to be conducted, using voice information and image information in a robot and to conduct the conversation. <P>SOLUTION: In the robot, specific voice, that indicates the beginning of the conversation being uttered by a speaker, is recognized, direction to the speaker is detected by estimating the sound source direction, movement is made toward the direction of the detected speaker, the speaker's face is detected from the image inputted from a camera after the movement is made and interactive processing is conducted when the face is detected. In the interactive processing, priority is given to the direction along which the face is detected and voice recognition is conducted by limiting the directionality of the voice recognition toward the face direction. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a human interface technique using both image recognition and voice recognition. In particular, the present invention relates to a robot having an image recognition function, a voice recognition function, and a mechanism control function.

  Recent robots are progressing from performing only predetermined tasks like industrial robots to determining situations and communicating with humans. For dialogue with humans, technology for detecting a conversation partner from the surrounding environment and technology for recognizing speech are required.

  In the conventional dialogue between a robot and a human, the person who talks in front of the robot is a predetermined person, and the speaker attaches a close-up microphone so that the robot recognizes only the voice of the conversation person. By using a close-up microphone, speaker detection technology is no longer necessary, and only the voice of the speaker can be recognized without inputting ambient noise.

  Next, as a means for detecting the direction of the conversation partner without using the close-talking microphone, there is a technique of arranging a plurality of microphones and estimating the direction of the sound source using the phase difference of each microphone. Further, there is a technique for detecting the position of a speaker by detecting a human face using a camera.

  However, in the case of only voice, there is a problem that it reacts erroneously to noise other than human voice. Further, in the case of only an image, there are problems that a complicated background is misrecognized as a face, and the face cannot be detected due to conditions such as illumination and face angle. For this reason, a technique has been devised that uses both image recognition and voice recognition, and if there is a face in the voice direction, the person with that face is the speaker. Known examples in this respect include Patent Documents 1 to 3. In Patent Document 1, control is performed to move to any one of a moving object detection result, a face detection result, and a sound source direction detection result. When a face is detected during movement, the face moves in the direction of the face and stops when it falls within a predetermined range. In Patent Document 2, a speaker is identified by using face recognition and sound source direction estimation. In Patent Literature 3, a speaker is tracked using one or more results of output of recognition functions such as face recognition and voice recognition.

JP 2004-130427 A

JP 2002-264051 A JP 2004-283959 A JP-A-8-272773 Toshiro Oga, Yoshio Yamazaki, Yutaka Kaneda, "Acoustic systems and digital processing," IEICE, pp.203-209, 1995/3/25

A first problem to be solved by the present invention is to accurately detect a conversation partner and execute a conversation in a situation where there are noises around the robot and there are a plurality of persons.
In Patent Literature 1, when the face is detected and the face is detected and the face is detected, the face is moved in the face direction. However, in the method of Patent Document 1, the face found when facing the sound source direction is the speaker. Since the accuracy of estimation of the sound source direction is not high, there is a problem that a speaker cannot be specified accurately when there are a plurality of persons in the sound source direction.

  The second problem to be solved by the present invention is to correctly recognize the speaker even in a situation where the speaker is outside the field of view of the camera and a person who is not the speaker is in the field of view. In Patent Document 3, a speaker is tracked using at least one result of face recognition and voice recognition. In this method, there is no detailed description about the priority of recognition processing. If the face recognition is given priority in the above situation, there is a problem that a speaker is detected incorrectly.

  The third problem to be solved by the present invention is to detect a speaker by performing appropriate control even when voice recognition and face recognition are misrecognized or unrecognizable. In face recognition, depending on conditions such as lighting conditions, distance to the speaker, and face angle, even if a face is present in the field of view, it may not be detected or may be detected incorrectly. In speech recognition, only the voice of the speaker is detected in a quiet room, but noise other than the voice of the speaker is present outdoors and the like, so the voice of the speaker may be mistaken for the noise. In such a situation, there is a problem that the conventional method which assumes that recognition works correctly cannot be dealt with.

  In Patent Document 1, since it is a precondition that the sound source direction and the face direction coincide with each other, there is a problem that the speaker cannot be specified if any of the recognition functions is erroneously recognized or cannot be recognized. is there.

  In Patent Document 2, a speaker is identified in a situation where there are a plurality of persons using face recognition and sound source direction estimation. However, this method assumes that each recognition function such as face recognition and sound source direction estimation is always correctly recognized. However, when recognition is performed in the above environment, an error in speech recognition and an error in image processing are unavoidable, and there is a problem that the method of Patent Document 2 cannot be recognized correctly.

  In Patent Literature 3, a speaker is tracked using one or more results of output of recognition functions such as face recognition and speech recognition. However, detailed control of face recognition and voice recognition is not described, and there is a problem that proper control cannot be performed in the case of misrecognition or recognition failure only with this example.

  An audiovisual linkage recognition method invented to solve such a problem is an audiovisual linkage recognition method that executes processing based on voice input and image input, and is a specific method used at the beginning of a conversation uttered by a speaker. Recognize the voice of a word or sentence and its direction, and if the recognition fails, return to the initial state, and if the recognition succeeds, point the camera in the direction of the detected voice and input images from the camera while moving The face of the person is detected, and if no face is detected, the process returns to the initial state, and if a face is detected, a dialogue process is performed.

  The effect of the present invention is that an interlocutor can be identified from a remote location in an environment with ambient noise. More specifically, it is possible to correctly distinguish between ambient noise and a conversation person's voice without using a close-up microphone. The second effect of the present invention is that a conversation person can be specified even when there are a plurality of persons around. The third effect of the present invention is that the robustness of recognition is high even when the environmental conditions are bad. When there is noise in the vicinity or when there are a plurality of persons, one or both of voice recognition and face recognition may be wrong or unrecognizable. Even in such a situation, it is possible to control autonomously to succeed in a correct dialogue.

  The following provide some specific details for an overall description of the invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without these specific details. The descriptions and drawings in this specification are typical means used by those skilled in the art to disclose the subject matter to others skilled in the art. In addition, when there is a description of “one embodiment” in the present specification, it does not necessarily apply only to the same embodiment, and individual embodiments are not limited to each other. Furthermore, the processing order of the processing according to the embodiment of the present invention is illustrative and not limited.

  First, FIG. 1 shows the configuration of an audiovisual cooperation apparatus according to the present invention. Reference numeral 100 denotes a microphone array composed of a plurality of microphones installed at different positions. At 110, the direction of the sound source is detected from the audio signal from the microphone array. At 120, the sound in the direction of the sound source is recognized. Reference numeral 130 denotes a camera for inputting an image. In 140, a face area is detected from the input image. In 140, in addition to the face detection, it may have a function of identifying who the registered face is registered. A mechanism control unit 150 moves the orientation and position of the microphone array 100 and the camera 130. The movement includes rotational movement in the horizontal and vertical directions, moving movement such as forward / backward, up / down / left / right. In 160, the dialogue is controlled based on the position of the speaker and the content of the speaker's story. The utterance content of the robot determined by 160 is uttered as a robot voice through the speaker 170. Reference numeral 180 denotes an overall control unit. The results of speech recognition and face recognition are integrated to detect the direction of the speaker, and the microphone array 100 and the camera 130 are moved through the mechanism control unit 150 based on the result, and the conversation control unit 160 has a conversation with the speaker. .

  First, features of the present embodiment will be described. The first feature of the present embodiment is that, in order to accurately estimate the direction of the speaker, the dialogue is performed only when both the sound source direction by voice recognition and the face direction by face detection match. Since coincidence is a condition, there are fewer false recognitions than when only one of them is used.

  The second feature is that the direction of the sound source direction is preferentially moved for detection of the speaker direction, and after turning around the direction of the speaker, the direction of the face is preferentially determined with priority on the face direction. That is. Thereby, a speaker can be specified even when there are a plurality of persons in the field of view.

  The third feature is that the recognizable words are limited to the speaker's call instead of performing the sound source direction estimation with wide directivity. Although it becomes difficult to discriminate between ambient noise and the voice of the speaker by expanding the range of sound source direction estimation, the discrimination performance is improved by limiting the words.

The fourth feature is that directivity is limited only to the speaker direction after the start of the dialogue. By limiting the directivity, the ambient noise and the voice of the speaker can be clearly separated.
With the above features, highly accurate speaker detection and dialogue can be realized.

  As a first embodiment of the present invention, a processing flow for detecting a speaker and having a conversation will be described with reference to FIGS. FIG. 2 is an overall flow, and FIG. 3 is a flow of only the dialog processing part of step 250.

  First, FIG. 2 will be described. The processing from step 200 to step 240 is processing for detecting the direction of the speaker. In step 200, a voice calling from the speaker to the robot is detected. In this process, the estimation range of the sound source direction is set in a wide range. The range of audio input can be controlled by combining the inputs from the microphone array. In this process, it is possible to limit words and sentences used for calling in order to accurately distinguish between noise and calling. A method for estimating the sound source direction will be described later. In step 210, it is determined whether the call voice and direction are recognized. If no call is detected, the process returns to step 200. That is, since the robot does not react at all, the speaker calls again.

  If a call is detected in step 210, the process moves in the call direction in step 220. The movement includes operations such as angle change, forward / backward movement, up / down / left / right. The movement may use distance information with respect to the target object and surrounding objects. Examples of obtaining distance information include obtaining from voice information in step 200, using a laser radar or the like, and stereo viewing using a plurality of cameras. In step 230, face detection is performed in the calling direction. As an example of the face detection process, there is a technique disclosed in Japanese Patent Application Laid-Open No. 8-272773 (Patent Document 4). This is to detect a face area by determining a skin color area from an image, and to detect a face area from image information of H (hue), S (saturation), and V (lightness) of the image. is there. In order to use input from a normal camera, the RGB signal may be converted into an HSV signal. In step 240, it is determined whether a face has been detected. If no face is detected, the process returns to step 200. That is, since the robot does not react at all, the speaker calls again. If a face is detected in step 240, it is determined that a dialogue partner has been detected, and the dialogue processing in step 250 is performed.

  FIG. 3 is a detailed flow of the dialog processing in step 250 of FIG. First, in step 300, the robot speaks to the caller. If the caller has a conversation corresponding to the utterance content in step 300, the voice is recognized in step 310. At this time, if the range of the sound source direction in which speech recognition is possible is limited to the viewing angle or the calling direction, the accuracy of separating the ambient noise and the speech of the conversation is increased. In step 310, the sound source direction is also estimated again. In step 320, it is determined whether the voice recognition is successful. If not successful, the process returns to step 310. That is, since the robot does not react at all, the speaker speaks again.

  If the speech recognition is successful, face detection is performed in step 330. In step 340, it is determined whether a face has been detected from the image. If no face is detected, the robot speaks at step 350. Examples of utterances in this case include “Cannot detect face” to tell you that the face cannot be detected, “Look here” to have you face the front, and change the standing position to light Or change the background, such as “Please move to the right (or left)”. If the process of step 350 is repeated a predetermined number of times, it may be determined that there is no speaker and the conversation may be terminated. Further, step 350 may be omitted, and if detection fails, the process may return directly to step 330. If a face is detected in step 340, it is determined in step 360 whether or not the sound source direction obtained in step 310 matches the face direction obtained in step 330.

  If the directions do not match, move to the speaker direction in step 370. This movement is to move the camera so that the position of the face is at the center of the field of view of the camera. The reason for performing step 370 is that when both the sound source direction and the face direction are detected, the detection accuracy of the face direction is higher. Thereby, even when there are a plurality of persons in the field of view, the speaker can be accurately identified. In the subsequent processing, in the speech recognition in step 310, it is possible to recognize with higher accuracy by making the directivity stronger in the speaker direction. Further, in step 370, it is only necessary to determine the conversation partner within the field of view without actually moving.

  If the directions match in step 360, it is determined in step 380 whether or not to continue the dialogue according to the content of the dialogue. If the dialogue is continued, the process returns to step 300. In this case, the content of the utterance in step 300 differs depending on the content of the dialogue. If the dialogue is not continued, the dialogue is terminated. After completion of the dialog, the process may return to step 200 in FIG. 2 and wait for the start of the next dialog.

As a second embodiment of the present invention, a flow of processing for detecting a speaker and performing a dialogue will be described with reference to FIG. In FIG. 4, the same numbers as those in FIG.
If no face is detected in step 240, the face may not be detected depending on the illumination conditions and the face angle. In order to cope with such a situation, the number of retries described later is limited in step 260. If the retry is within the specified number of times, in step 270, an utterance requesting the user to change the posture is performed in order to improve face detection. Specific examples include “Turn here” and “Move to the right (or left) slightly (due to lighting effects)”. If the specified number of times is exceeded, it is determined that the detection of the calling voice in step 200 is incorrect, and the process returns to step 200.

  Hereinafter, an embodiment of the sound source direction estimation technique used in Embodiments 1 and 2 will be described. As an example of a technique for estimating the direction of a sound source using a plurality of microphones, there is a blind spot forming type sound source localization technique shown in Non-Patent Document 1. In this technique, a blind spot is formed in a sound source direction that exists in a direction other than the direction of the determination target, and only the sound in the direction of the determination target is extracted, thereby calculating the sound power for each direction. Then, the sound source direction is estimated from the sound power for each direction. It is known that the blind spot forming type sound source localization technology can estimate the sound source direction with high accuracy when the number of sound sources is less than the number of microphones.

It is a block diagram of the present invention. It is a figure of the audio-visual cooperation flow which shows a 1st Example. It is a figure of the flow of the dialogue process in a 1st Example. It is a figure of the audiovisual cooperation flow which shows a 2nd Example.

Explanation of symbols

  100: Microphone array, 110: Sound source direction estimation unit, 120: Speech recognition unit, 130: Camera, 140: Face recognition unit, 150: Mechanism control unit, 160: Dialog control unit, 170: Overall control unit

Claims (4)

An audiovisual linkage recognition method for executing processing based on audio input and image input,
Based on the voice input from the microphone array, it recognizes the voice and direction of a specific word or sentence used at the beginning of the conversation made by the speaker,
If the speech recognition fails, it returns to the initial state,
If the speech recognition is successful, point the camera in the direction of the detected speech,
Detecting a human face from an image input from the camera,
If no face is detected, it returns to the initial state,
An audio-visual cooperative recognition method characterized in that dialogue processing is performed when a face is detected. An audiovisual linkage recognition method for executing processing based on audio input and image input,
Based on the voice input from the microphone array, it recognizes the voice and direction of a specific word or sentence used at the beginning of the conversation made by the speaker,
If recognition fails, it returns to the initial state,
If recognition succeeds, point the camera in the direction of the detected voice,
Detecting a human face from an image input from the camera,
If no face is detected, inform the user that the face cannot be detected and return to the face detection process.
An audio-visual cooperative recognition method characterized in that dialogue processing is performed when a face is detected. In the dialogue processing of the audiovisual linkage method according to claim 1 or 2, utterance is performed in response to a talk from a speaker,
Limiting directivity to the speaker direction detected in claim 1 or claim 2, recognizing the voice and direction of the speaker,
If voice recognition fails, it returns to the voice recognition standby state,
If speech recognition is successful, face detection processing is performed.
If no face is detected, tell the utterance that no face is detected and return to the face detection process.
If a face is detected, determine whether the direction of the speaker and the direction of the speaker detected in the voice recognition match,
If they do not match, correct the speaker direction recognized in claim 1 or claim 2 and move to the detected face direction,
If they match, decide whether to continue the conversation,
To continue the dialogue, select the utterance content according to the state transition of the dialogue, return to the utterance,
An audiovisual linkage recognition method characterized in that the processing is terminated if not continued. An audio-visual cooperative recognition device that executes processing based on audio input and image input,
Collecting voice information from a plurality of microphones, having a sound source direction estimation unit that estimates the direction of the voice from the collected voice information, and a voice recognition unit that recognizes the voice from the estimated sound source direction,
It has a face recognition unit that collects image information from the camera and detects a human face area from the collected image information,
A mechanism control unit for moving the position and direction of the camera and microphone;
The sound source direction estimation unit, the speech recognition unit, and the result of the face recognition unit are integrated to detect the speaker direction, the mechanism control unit of the device, and the overall control unit to understand the conversation content,
A dialog control unit that determines the state transition of the dialog according to the speech recognition result and selects the utterance content, and a speaker that utters,
The sound source direction estimating unit changes the directionality of direction estimation according to the state of dialogue;
In the mechanism control unit, a process of moving the device in the sound source direction estimated in the control unit;
The audio-visual recognizing and recognizing portable apparatus characterized in that the overall control unit executes a process of performing a dialogue only when both of the speech direction estimation results using the voice information and the image information match.

Images