JP2002264051A - Robot audio-visual system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a robot audio-visual system ensuring the tracing of an object by integrating visual and auditory information on the object. SOLUTION: An auditory module 20 extracts an auditory event 28 by identifying the sound source of a speaker by pitch extraction and the separation and orientation of the sound source from an acoustic signal of a microphone. A visual module 30 extracts a visual event 39 by identifying each speaker by the face identification and orientation of the speaker from the image of a camera. A motor control module 40 extracts a motor event 49 from the rotating position of a drive motor. An association module 60 creates an auditory stream 65 and a visual stream 66 from the auditory event, visual event and motor event and creates an association stream 67 by associating these streams. The direction of each speaker is thereby determined on the basis of the sound source orientation of the auditory event and the face orientation of the visual event to perform attention control for the planning of drive motor control on the basis of these streams.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an audiovisual system for a robot, particularly a humanoid or animal robot.

[0002]

2. Description of the Related Art In recent years, attention has been paid to active perception of sight and hearing in such humanoid or animal type robots. Active perception is to control the posture of a head that supports these perception devices, such as a head, by a drive mechanism so that the perception device that is responsible for perception such as robot vision and robot hearing follows an object to be perceived. .

Here, with regard to active vision, at least a camera, which is a perceptual device, holds its optical axis direction toward a subject by attitude control by a driving mechanism, and automatically focuses, zooms in, and zooms out on the subject. As a result, the object is imaged by a camera, and various studies have been made.

[0004] On the other hand, regarding active hearing, at least a microphone, which is a perceptual device, maintains its directivity toward an object by attitude control by a drive mechanism, and sounds from the object are collected by the microphone. Such active hearing is disclosed, for example, in Japanese Patent Application No. 2000-2267 by the present applicant.
No. 7 (robot hearing system), which directs a sound source with reference to visual information.

[0005]

Incidentally, these active vision and active hearing are closely related to a motor control module for changing the orientation (horizontal direction) of the robot, and the active vision and the active hearing are performed for a specific object. In order to activate active hearing, it is necessary to aim the robot at a specific target, that is, to perform attention control. However, in order for the robot to accurately identify each speaker of interest based on the surrounding situation, it is necessary to integrate visual and auditory information, for example, as if multiple people are talking with each other. In such a situation, it has not been performed to identify each person by real-time processing and perform active hearing.

SUMMARY OF THE INVENTION In view of the above, an object of the present invention is to provide a robot audio-visual system which integrates visual and auditory information on an object to reliably track the object.

[0007]

According to the present invention, there is provided a hearing module including at least a pair of microphones for collecting external sounds, a visual module including a camera for capturing an image in front of the robot, and a robot. A motor control module that includes a drive motor that rotates the camera horizontally, an association module that integrates events from the auditory module, the vision module, and the motor control module to generate a stream, and a stream generated by the association module. An attention control module for performing attention control, wherein the hearing module determines a direction of at least one speaker from pitch extraction, sound source separation and localization based on an acoustic signal from a microphone. Determined that hearing eve Extract the door, visual module,
Based on the image captured by the camera, each speaker is identified from the face identification and localization of each speaker and its visual event is extracted, and the motor control module generates a motor event based on the rotational position of the drive motor. The extraction allows the association module to determine the direction of each speaker from the auditory event, the visual event, and the motor event based on the direction information of the sound source localization of the auditory event and the face localization of the visual event. Generating a visual stream and further associating them to generate an association stream, wherein the attention control module performs attention control for planning the drive motor control of the motor control module based on these streams. With the robot audiovisual system, It is made.

Further, according to the present invention, there is provided a hearing module including at least a pair of microphones for collecting external sounds, a visual module including a camera for capturing an image in front of the robot, and A motor control module including a driving motor to be rotated; an association module that integrates events from the hearing module, the vision module, and the motor control module to generate a stream; and performs attention control based on the stream generated by the association module. At least one speaker from pitch extraction, sound source separation and localization based on acoustic signals from the microphone. Direction The auditory event is extracted, the visual module identifies each speaker from the face identification and localization of each speaker based on the image captured by the camera, and extracts the visual event thereof. By extracting the motor event based on the rotational position of, the association module determines the direction of each speaker based on the direction information of the sound source localization of the auditory event and the face localization of the visual event from the auditory event, the visual event and the motor event. The determination generates an audio stream and a visual stream, and further associates them to generate an association stream, and the attention control module uses the streams to determine an attention for planning the drive motor control of the motor control module. Specially to control The robot audiovisual system that is achieved.

A robot audiovisual system according to the present invention comprises:
Preferably, the association module synchronizes the asynchronously generated auditory, visual, and motor events with each other when generating the audio and visual streams.

[0010] The robot audiovisual system according to the present invention comprises:
Preferably, the auditory module identifies each speaker by detecting the MFCC of the audio from the audio signal, and the association module identifies the speaker based on the speaker identification of the auditory event and the speaker identification of the visual event. By specifying, the auditory and visual streams to which the auditory and visual events are to be connected are selected.

[0011] The robot audiovisual system according to the present invention comprises:
Preferably, when the plurality of streams approach each other, the association module refers to a temporal flow of the auditory event and the visual event, and selects an auditory stream and a visual stream to which the auditory event and the visual event are to be connected. .

A robot audiovisual system according to the present invention comprises:
Preferably, the association module associates an auditory stream and a visual stream that are strongly associated with each other to generate an association stream, and cancels the association when the association between the auditory stream and the visual stream that constitute the association stream is weakened. To destroy the association stream.

[0013] According to the above configuration, the auditory module obtains the direction for each sound source by performing pitch extraction using the harmonic structure from the sound from the external object collected by the microphone, and obtains the individual speech. The direction of the person is determined and the auditory event is extracted. In addition, the visual module identifies each speaker from face identification and localization of each speaker by pattern recognition from an image captured by the camera, and extracts a visual event of each speaker. Further, the motor control module extracts a motor event by detecting a direction of the robot based on a rotation position of a drive motor that rotates the robot in a horizontal direction. Note that the event means that a sound or face is detected at each time point, features such as pitch and direction are extracted, and speaker identification and face identification are performed.
Alternatively, it indicates a state in which the drive motor is rotated, and the stream indicates a temporally continuous event.

Here, the association module is
By determining the direction of each speaker based on the direction information of the sound source localization of the auditory event and the face localization of the visual event based on the auditory event, the visual event, and the motor event thus extracted, the speaker's An audio stream and a visual stream are generated, and the streams are associated to generate an association stream. At this time, the association module determines the direction of each speaker based on the sound source localization of the auditory event and the face localization of the visual event, i.e., the auditory and visual direction information, and refers to the determined direction of each speaker. An association stream will be generated. Then, the attention control module performs planning of drive motor control of the motor control module by performing attention control based on these streams. Attention is the audible and / or visual "attention" of the speaker to which the robot is intended. Attention control is when the robot changes its orientation by means of a motor control module so that the robot can It is to pay attention to.

Then, the attention control module comprises:
By controlling the drive motor of the motor control module based on this planning, the direction of the robot is directed to the target speaker. With this, the robot can directly face the target speaker, so that the hearing module can accurately collect and localize the voice of the speaker by the microphone in a highly sensitive frontal direction, The module can better image the speaker with the camera.

Therefore, by the cooperation of the hearing module, the visual module, and the motor control module with the association module and the attention control module, based on the direction information of the sound source localization of the auditory event and the speaker localization of the visual event, By determining the direction of the speakers, the ambiguities of the hearing and vision of the robot are complemented with each other, so-called robustness is improved, and even if there are a plurality of speakers, each speaker is Can be reliably perceived. Also, for example, even when one of the auditory event or the visual event is missing, the attention control module can track the target speaker based on only the remaining visual event or the auditory event, It is possible to accurately grasp the direction of the object and control the motor control module.

In the case where the association module synchronizes an audio event, a visual event, and a motor event that are generated asynchronously with each other when generating the audio stream and the visual stream, the audio event generated asynchronously, By synchronizing the visual and motor events with each other, different generation periods and delay times of these events in the association module will be absorbed, and the direction information of the sound source localization of the auditory event and the speaker localization of the visual event will be absorbed. Is used to determine the direction of each speaker more accurately. Therefore, when the auditory stream composed of the auditory event and the visual stream composed of the visual event exist at a distance close to each other, it is possible to associate them with each other to generate a higher-order association stream.

[0018] The association module identifies each speaker by the hearing module detecting MFCC of speech from the audio signal, and the association module determines based on speaker identification of the auditory event and speaker identification of the visual event. By specifying the speaker, when an auditory stream and a visual stream to which the auditory event and the visual event are to be connected are selected, the speaker identification can be performed by the MFCC of the audio from the auditory event. Visual events will identify individual speakers. Thus, by connecting each auditory event and visual event to the same speaker's auditory stream and visual stream, respectively, it is possible to more accurately identify each speaker even when there are multiple speakers, for example. , An auditory stream and a visual stream can be generated, and even if one of the auditory event and the visual event is interrupted halfway, the other event can continue speaker identification. This enables more accurate tracking of each speaker by identifying each speaker and performing higher-order integration of hearing and vision even when voices of multiple speakers are detected from the same direction. Becomes possible.

When the plurality of streams approach each other, the association module refers to a temporal flow of the auditory event and the visual event, and selects an auditory stream and a visual stream to which the auditory event and the visual event are to be connected. In this case, even when a plurality of speakers are close to each other and the auditory stream and the visual stream of these speakers are close to each other and intersect, the range of movement of the speaker is predicted. In this range, by retaining the auditory stream or the visual stream, the auditory stream and the visual stream can be generated more accurately. Therefore, the ambiguity of the auditory stream and the visual stream is complemented with each other, so-called robustness is improved, and a plurality of speakers can be reliably tracked.

The association module associates an auditory stream and a visual stream that are strongly associated with each other to generate an association stream. When the association between the auditory stream and the visual stream that constitute the association stream is weakened, the association module releases the association. Therefore, when the association stream is extinguished, the association stream can be accurately generated for each speaker, so that the ambiguity of the auditory stream and the visual stream is eliminated as much as possible, and the accurate speaker identification is performed. Can be performed. Furthermore, in this case, by appropriately selecting the predetermined angle, even if the speaker is moving, the movement of the speaker is surely captured, so to speak, the movement of the speaker is predicted. Can be specified.

[0021]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail based on embodiments shown in the drawings. 1 to 4 show the overall configuration of an experimental humanoid robot provided with an embodiment of the robot audiovisual system according to the present invention. In FIG. 1, the humanoid robot 10 has 4 DOF (degree of freedom).
The robot includes a base 11, a body 12 supported on the base 11 so as to be rotatable around one axis (vertical axis), and a three-axis direction (vertical axis, (A horizontal axis in the left-right direction and a horizontal axis in the front-rear direction).

The base 11 may be fixedly arranged, and may be operable as a leg. Also base 1
1 may be mounted on a movable cart or the like.
The body portion 12 is rotatably supported as shown by an arrow A in FIG. 1 around a vertical axis with respect to the base 11, and is driven to rotate by driving means (not shown). , Covered by a soundproof exterior.

The head 13 is supported by the body 12 via a connecting member 13a.
Can swing about a horizontal axis in the front-rear direction, as shown by an arrow B in FIG. 1, and can swing about a horizontal axis in the left-right direction, as shown by an arrow C in FIG. In addition, the connecting member 13a is swingably supported as shown by an arrow D in FIG. It is rotationally driven in the directions of the arrows A, B, C, and D by the non-driving means.

Here, the head 13 is entirely covered with a soundproof exterior 14, as shown in FIG.
A camera 15 is provided on the front side as a visual device in charge of robot vision, and a pair of microphones 16 (16a, 16b) are provided on both sides as hearing devices in charge of robot hearing.

The exterior 14 is made of a sound-absorbing synthetic resin such as a urethane resin, for example. The interior of the head 13 is sound-insulated by almost completely sealing the interior of the head 13. Have been. The exterior of the body 12 is also made of a synthetic resin having a sound absorbing property. The camera 15 has a known configuration, and for example, a commercially available camera having a so-called pan, tilt, and zoom 3DOF (degree of freedom) can be applied.

The microphones 16 are mounted on the side surfaces of the head 13 so as to have directivity toward the front. Here, the left and right microphones 16a and 16b of the microphone 16 are, as shown in FIG. 1 and FIG.
b, the front sound is collected through the through holes provided inside the steps 14a and 14b and attached to the inside.
The sound is shielded by appropriate means so as not to pick up the sound inside the exterior 14. Thereby, the microphones 16a, 16b
Are configured as so-called binaural microphones. In the vicinity of the mounting positions of the microphones 16a and 16b,
The exterior 14 may be formed in the shape of a human outer ear.

FIG. 4 shows an electrical configuration of the robot audiovisual system including the microphone 16 and the camera 15. In FIG. 4, the audiovisual system 17 is configured as a party reception and companion robot, and includes an audio module 20, a visual module 30, a motor control module 40, a dialog module 50, and an association module 60. Hereinafter, a further description will be given with reference to FIGS. For convenience of explanation, the auditory module 20 is block 1
5, the visual module 30 is shown as a block 2 in FIG. 6, the motor control module 40 is shown as a block 3 in FIG. 7, and the dialogue module 50 is a block 4 in FIG. Shown enlarged,
Further, the association module 60 is set to block 5
FIG. 9 is an enlarged view of FIG. Here, the association module 60 (block 5, FIG. 9) is composed of a server, and has other modules, ie, the hearing module 20 (block 1, FIG. 5), the visual module 30 (block 2, FIG. 6), and the motor. The control module 40 (block 3, FIG. 7) and the dialogue module 50 (block 4, FIG. 8) are each composed of a client and operate asynchronously with each other.

The server and each client are:
For example, it is configured by a personal computer,
LA, for example, by a TCP / IP protocol via a network 70 such as 100Base-T.
N connections are made. In addition, each module 20, 30, 4
0, 50, and 60 are hierarchically distributed, and specifically, are device layers, process layers, feature layers,
It consists of an event layer.

As shown in FIG. 5, the hearing module 2
0 denotes a microphone 16 as a device layer, a peak extraction unit 21, a sound source localization unit 22, a sound source separation unit 23, and a speaker identification unit 23a as process layers, a pitch 24 as a characteristic layer (data), and a horizontal direction 25. And an auditory event generator 26 and a viewer 27 as an event layer.

Here, the hearing module 20 operates as shown in FIG. That is, in FIG. 10, the hearing module 20 is, for example,
The sound signal from the microphone 16 sampled at z, 16 bits is subjected to frequency analysis by FFT (Fast Fourier Transform) as indicated by a symbol X2, and a spectrum is generated for each of the left and right channels as indicated by a symbol X3. . Then, the hearing module 20 extracts a series of peaks for each of the left and right channels by the peak extracting unit 21 and pairs the same or similar peaks in the left and right channels. here,
The peak extraction is a band that transmits only data under the condition that the power is equal to or higher than the threshold value and is a local peak, and the frequency is, for example, 90 Hz to 3 kHz in order to cut low-frequency noise and high-frequency band with low power. This is done by using a filter. This threshold is defined as a value obtained by measuring background noise in the surroundings and adding a sensitivity parameter, for example, 10 dB.

Using the fact that each peak has a harmonic structure, the hearing module 20 extracts local peaks having a harmonic structure in order from the lowest frequency, and extracts the peaks of the extracted peaks. Assuming the set as one sound, the sound source separation unit 23 applies an inverse FFT (fast Fourier transform) as indicated by reference sign X4, thereby obtaining a sound signal for each sound source from a mixed sound from each sound source as indicated by reference sign X5. Is separated.

Thus, the auditory module 20 selects the sound signal of the same frequency from the left and right channels as indicated by the symbol X6 by the sound source localization unit 22 for the sound signal of each sound source, for example, every five degrees. IPD (interaural phase difference) and IID (interaural intensity difference) are determined. Then, using a so-called auditory epipolar geometry, the sound source localization unit 22 of the auditory module 20 uses the hypothesis inference indicated by the symbol X7 in a range of ± 90 ° with the front of the robot 10 being 0 °.
Generate a hypothesis for D Ph

(Equation 1) Then, the distance d (θ) between the sound separated by the above and each hypothesis is calculated. Here, n _f <1.5 kHz means that the frequency is 1.5 kHz
Hz. This is the left and right microphone 15
Since the IPD is effective for frequencies of 1.2 to 1.5 kHz or less from the baseline, the experiment was performed at 1.5 kHz or less in this experiment.

The IID is obtained from the power difference between the left and right channels of each overtone of the separated sound, similarly to the IPD. However, IID is not hypothetical inference,

(Equation 2) The sound source is determined to be either left or right using a discriminant function based on That is, the IID of each overtone of the frequency f is I
_{Assuming that s} (f), the sound source exists in the left direction of the robot when I is positive, in the right direction when I is negative, and in the front direction when I is almost zero. Where IID
Since the generation of a hypothesis requires an enormous amount of calculation considering the shape of the head of the robot 10, a hypothesis inference similar to that of the IPD is not performed in consideration of real-time processing. In this way, the matching between the IPD and the IID is performed, as indicated by reference numeral X8.

Then, the sound source localization unit 22 of the hearing module 20 calculates the probability density function from the distance d (θ), as indicated by the symbol X9.

(Equation 3) Is used to calculate the _IPD certainty factor BF _IPD (θ). Here, m and s are the average and variance of d (θ), respectively, and n is the number of d. In addition, the confidence B of the IID
F _IID (θ) is 30 degrees <θ ≦ 90 degrees, 0.35 when I is +, 0.65 when −, and −30 degrees <θ ≦ 30.
In degrees, 0.5 when I is +, 0.5 when −, -9
0 degree <θ ≦ −30 degrees and 0.65−
It becomes 0.35 at the time of.

Then, the confidence BF _IPD (θ) of the IPD obtained in this way and the confidence BF of the IID
_IID (θ) is represented by a symbol X10,

(Equation 4) Are integrated according to the Dempster-Shafer theory shown by the formula (1) to generate a certainty factor BF _{IP D + IID} (θ). Further, the speaker identification unit 23a obtains, for example, an MFCC (mel frequency cepstrum coefficient) from the acoustic signal from the microphone 16 and compares the MFCC with the MFCC of the speaker registered in advance to identify the speaker itself. Identify. As a result, the auditory module 20 outputs the auditory event generator 26
Thus, the auditory event 28 is generated by the top 20 confidence factors BF _{IPD + IID} (θ), the list of the directions (θ), the pitch, and the speaker identification in the descending order of the likelihood as the sound source direction.

In this way, the hearing module 20
Based on the sound signal from the microphone 16, at least one speaker is specified (speaker identification) from the pitch extraction, sound source separation and localization, and MFCC, and the auditory event is extracted. 60. The above-described processing in the hearing module 20 is performed every 40 msec.

The viewer 27 displays the auditory event 28 thus generated on the screen of the client. Specifically, the power spectrum and the extracted peak of the auditory event 28 are displayed in the left window. In the window on the right, auditory events having a relative azimuth on the vertical axis and a pitch (frequency) on the horizontal axis are displayed. Here, the auditory event is represented by a circle having the certainty factor of the sound source localization as the diameter of the circle.

As shown in FIG. 6, the visual module 3
0 indicates a camera 15 as a device layer, a face detection unit 31, a face identification unit 32, and a face localization unit 33 as process layers;
A face ID 34 and a face direction 35 as feature layers (data);
It comprises a visual event generator 36 and a viewer 37 as an event layer.

Thus, the visual module 30 detects the face of each speaker by, for example, extracting skin color by the face detecting unit 31 based on the image signal from the camera, and the face registered in advance by the face identifying unit 32. When there is a matched face searched by the database 38, the face ID 34 is determined to identify the face, and the face direction 35 is determined (localized) by the face localization unit 33. Here, when the face finding unit 31 finds a plurality of faces from the image signal, the visual module 30 performs the above-described processing, that is, identification, localization, and tracking for each face. At this time, since the size, direction, and brightness of the face detected by the face detection unit 31 often change, the face detection unit 31 detects the face area and extracts the combination of the skin color and the pattern matching based on the correlation operation. Thus, a plurality of faces can be accurately detected within 200 ms.

The face identifying unit 32 projects each face area image detected by the face finding unit 31 into the discrimination space, and calculates a distance d from the face data registered in the face database 38 in advance. Since this distance d depends on the number of registered faces (L),

(Equation 5) Is converted to a certainty factor Pv independent of parameters. Here, the discriminant matrix serving as the basis of the discriminant space can be updated by a known online LDA with a smaller number of calculations compared to a normal LDA, so that face data can be registered in real time.

The face localization unit 33 converts the face position on the two-dimensional image plane into a three-dimensional space, and the width and height of the face at (x, y) on the image plane are X and Y, respectively. Assuming that there are w × w pixels, the face position in the three-dimensional space is obtained as a set of an azimuth angle θ, a height φ, and a distance r given by the following equations.

(Equation 6)

(Equation 7)

(Equation 8) Here, C ₁ and C ₂ are constants defined by the search image size (X, Y), the angle of view of the camera, and the actual face size.

Then, the visual module 30 generates a visual event 39 from the face ID (name) 34 and the face direction 35 by the visual event generator 36 for each face. In detail, the visual event 39 includes, for each face, the top five face IDs (names) 34 with certainty factors and positions (distance r, horizontal angle θ, and vertical angle φ).

The viewer 37 displays a visual event on the screen of the client. Specifically, the viewer 37 displays an image obtained by the camera 15, a face ID of a face extracted with certainty of face identification, and a localization result. Display a list of locations that are. Here, in the image obtained by the camera 15, the face that has been found and identified is displayed surrounded by a rectangular frame. When a plurality of faces are found, a rectangular frame indicating the identification and a list as a result of the localization are displayed for each face.

As shown in FIG. 7, the motor control module 40 includes a motor 41 and a potentiometer 42 as device layers, a PWM control circuit 43, an AD conversion circuit 44 and a motor control unit 45 as process layers, and a characteristic layer. , A motor event generator 47 as an event layer, and a viewer 48.

Thus, the motor control module 40
Is controlled by the motor control unit 45 based on a command from the attention control module 64 (described later).
The drive of the motor 41 is controlled via the
1 is detected by the potentiometer 42,
A motor control unit 45 extracts a robot direction 46 via an AD conversion circuit 44, and a motor event generation unit 47 generates a motor event 49 including motor direction information.

The viewer 48 displays the motor event three-dimensionally on the client screen. Specifically, the viewer 48 displays the direction and the operation speed of the robot by the motor event 49, for example, in a tertiary system implemented by OpenGL. The original viewer is used to display three-dimensional images in real time.

As shown in FIG.
0 is a speaker 51 and a microphone 16 as device layers.
And a speech synthesis circuit 52 as a process layer, a dialogue control circuit 53, a self-voice suppression circuit 54, and a speech recognition circuit 55.

Thus, the dialogue module 50 controls the dialogue control circuit 53 by the later-described association module 60 and the speaker 51 by the speech synthesis circuit 52.
, A predetermined sound is emitted to the target speaker, the sound from the speaker 51 is removed from the acoustic signal from the microphone 16 by the self-voice suppression circuit 54, and then the target Recognize the speaker's voice. The interaction module 50 does not have a feature layer and an event layer as a hierarchy.

Here, for example, in the case of a party reception robot, the highest priority is given to the current attention in the case of a party reception robot. for,
Attention controlled.

As shown in FIG. 9, the association module 60 is hierarchically positioned higher than the above-described auditory module 20, visual module 30, motor control module 40, and dialog module 50. The stream layer, which is the upper layer of the 20, 30, 40, and 50 event layers, is configured. Specifically, the association module 60 synchronizes the asynchronous event 61a from the auditory module 20, the visual module 30, and the motor control module 40, that is, the auditory event 28, the visual event 39, and the motor event 49 into a synchronous event 61b. A circuit 62, a stream generation unit 63 that correlates these synchronization events 61b to generate an auditory stream 65, a visual stream 66, and an association stream 67; an attention control module 64;
It has.

The synchronizing circuit 62 is connected to the hearing module 20.
The synchronizing auditory event 28, the visual event 38 from the visual module 30, and the motor event 49 from the motor control module 40 are synchronized to generate a synchronous auditory event, a synchronous visual event, and a synchronous motor event. At this time, the coordinate system of the auditory event 28 and the visual event 39 is converted into the absolute coordinate system by the synchronous motor event.

Here, the delay time from the actual observation of each event to the arrival at the association module 60 via the network 70 is, for example, 40 ms for the auditory event 28 and 200 m for the visual event 39.
Second, the motor event 49 is 100 m, the delay in the network 70 is 10 to 200 msec, and the arrival period is also different. Therefore,
To synchronize each event, the hearing module 20,
The auditory event 28, the visual event 39, and the motor event 49 from the visual module 30 and the motor control module 40 each have time stamp information indicating the actual observation time.
For example, it is temporarily stored only for two seconds.

Then, the synchronization circuit 62 converts each event stored in the short-term storage circuit into a synchronization process so as to provide a delay time of 500 ms in comparison with the actual observation time in consideration of the delay time described above. Remove by As a result, the response time of the synchronization circuit 62 becomes 500 ms. Further, such a synchronization process operates at a period of, for example, 100 ms. Since each event arrives at the association module 60 asynchronously with each other, an event at the same time as the observation time for synchronization is not always present. Therefore, in the synchronization process, interpolation is performed by linear interpolation on events that occur before and after the observation time for synchronization.

Further, as shown in FIG. 11, the stream generator 63 reads the auditory event S and the visual event V from the short-term storage circuit M, and generates the streams 65, 66, 67 based on the following points. . 1. The auditory event 28 is connected to the nearest auditory stream 65 with equal or overtone pitches and within ± 10 degrees in direction, as indicated by the symbol Y1. still,
Values within ± 10 degrees are selected taking into account the accuracy of the auditory epipolar geometry. 2. The visual event 39 has a common face ID 34 and is connected to the closest visual stream 66 within a range of 40 cm, as indicated by the symbol Y2. The value in the range of 40 cm is selected on the assumption that a human does not move at a speed of 4 m or more per second. 3. As a result of searching all the streams, if there is an event for which there is no connectable stream 65, 66, the event 28, 39 constitutes a new stream 65, 66, as indicated by reference numeral Y3. Will be. 4. If there are no events 28 and 39 connected to these streams 65 and 66, the existing streams 65 and 66 remain for a maximum of 500 msec, as indicated by reference numeral Y4a. If it continues, it will disappear as shown by code | symbol Y4b. 5. Auditory stream 65 and visual stream 66 are ± 1
If the state of approaching within 0 degrees lasts for 500 ms or more in one second, the auditory stream 65 and the visual stream 66 are considered to be derived from the same speaker, and as shown by a symbol Y5, Associated with each other, an association stream 67 is generated. 6. The association stream 67 is dissociated if the auditory event 28 or visual event 39 is not connected for more than 3 seconds, and only the existing auditory stream 65 or visual stream 66 remains. 7. The association stream 67 has a direction difference between the auditory stream 65 and the visual stream 66 of 3 seconds, ±
If the angle exceeds 30 degrees, the association is released, and the process returns to the individual auditory streams 65 and the visual streams 66.

Thus, based on the synchronous auditory event and the synchronous visual event from the synchronous circuit 62, the stream generation unit 63 connects the events in consideration of their temporal connection, thereby forming the synchronous auditory event and the synchronous visual event. By connecting the events to the same speaker's auditory stream 65 and the visual stream 66, the auditory stream 65 and the visual stream 66 are generated, and the strongly interconnected auditory stream 65 and the visual stream 66 are associated to associate. Stream 6
7 is generated. Conversely, when the connection between the auditory stream 65 and the visual stream 66 constituting the association stream 67 is weakened, the association is released. Accordingly, even when the target speaker is moving, the movement of the speaker is predicted, and if the target speaker is within the angle range which is the moving range, the streams 65, 66, and 67 described above are transmitted. By performing the generation, the movement of the speaker can be predicted and tracked.

The attention control module 64
Performs attention control for planning the drive motor control of the motor control module 40. At this time, the attention control is performed by preferentially referring to the association stream 67, the auditory stream 65, and the visual stream 66 in this order. Then, the attention control module 64 performs the operation planning of the robot 10 based on the state of the auditory stream 65 and the visual stream 66 and the presence or absence of the association stream 67. If the operation of the drive motor 41 is required, the motor control module 40 , A motor event as an operation command is transmitted via the network 70.

Here, the attention control module 64
Attention control in is based on continuity and triggers, trying to keep the same state with continuity and trying to track the objects of most interest with triggers. Therefore, attention control includes: The presence of the association stream indicates that the person who is speaking directly to the robot 10 is still present or has been present in the near future, It is necessary to focus attention on a person who has high priority and perform tracking. 2. Since the microphone 16 is omnidirectional, there is no detection range such as the viewing angle of the camera, and a wide range of auditory information can be obtained. Therefore, the priority of the auditory stream should be higher than that of the visual stream. In consideration of these two points, a stream to which attention is directed is selected and tracking is performed according to the following principle. 1. Prioritize association stream tracking. 2. If there is no association stream,
Prioritize audio stream tracking. 3. If the association stream and the auditory stream do not exist, priority is given to tracking the visual stream. In this way, the attention control module 64
By performing attention control, the motor control module 4
0, the control of the drive motor 41 is planned, a motor command 66 is generated based on the plan, and the motor control module 4 is controlled via the network 70.
Transmit to 0. Thereby, the motor control module 40
Then, based on the motor command 66, the motor control unit 45 performs PWM control to rotate and drive the drive motor 41 so as to direct the robot 10 in a predetermined direction.

The viewer 68 displays each stream generated in this way on the screen of the server. Specifically, the viewer 68 displays a radar chart and a stream chart. Here, the radar chart shows the state of the stream at that moment, more specifically, the viewing angle of the camera and the sound source direction, and the stream chart shows the association stream (shown in bold lines), the auditory stream and the visual stream (shown in thin lines). ing.

The humanoid robot 10 according to the embodiment of the present invention is configured as described above, and operates as follows with reference to FIG. 12 for a speaker to be a party reception robot. First, as shown in FIG.
The robot 10 is arranged in front of the entrance of the party venue. Then, as shown in FIG. 12B, the party participant P approaches the robot 10,
Has not yet recognized the participant P. Here, the participant P has talks with the robot 10, for example, "Hello", the robot 10 includes a microphone 16 picking up the sound of the participant P, generate an auditory event 28 hearing module 20 involves a sound source direction Then, the data is transmitted to the association module 60 via the network 70.

As a result, the association module 60 generates the auditory stream 29 based on the auditory event 28. At this time, the visual module 30 does not generate the visual event 39 because the participant P is not in the field of view of the camera 15. Accordingly, the association module 60 generates the auditory stream 29 based only on the auditory event 28, and the attention control module 64 uses the auditory stream 29 as a trigger to generate the auditory stream 29.
Attention control is performed such that the robot 10 is directed toward the participant P.

In this way, as shown in FIG. 12C, the robot 10 faces the direction of the participant P, and tracking by a so-called voice is performed. Then, the visual module 30 captures the image of the face of the participant P by the camera 15, generates a visual event 39, searches the face of the participant P by the face database 38, performs face identification, and determines the result. Is transmitted to the association module 60 via the network 70. If the face of the participant P is not registered in the face database 38, the visual module 30 transmits the fact to the association module 60 via the network 70.

At this time, the robot 10 has generated an association stream 65 based on the auditory event 28 and the visual event 39, and the attention control module 64 uses the association stream 65 to generate the association stream 65.
Since the attention control is not changed, the robot 10
Keeps pointing in the direction of the participant P. Therefore, even if the participant P moves, the robot 10 tracks the participant P by controlling the motor control module 40 by the association stream 65, and the camera 15 of the visual module 30 continues the participant P. It is possible to take an image.

Then, the association module 60
Gives an input to the speech recognition circuit 55 of the hearing module 20, and the speech recognition circuit 55 gives the speech recognition result to the dialogue control circuit 53. As a result, the dialog control circuit 53 performs voice synthesis and utters voice from the speaker 51. At this time, the voice recognition circuit 55 reduces the sound from the speaker 51 from the sound signal from the microphone 16 by the self-voice suppression circuit 54, so that the robot 10 ignores its own utterance,
The other party's voice can be more accurately recognized.

Here, the utterance by the speech synthesis is based on the participant P
Is registered in the face database 38 or not. If the face of the participant P is registered in the face database 38, the association module 60
Is based on the face ID 24 from the vision module 30,
To control the interaction module 50, to ask questions to participants P as "Hello .XXX-san?" By the speech synthesis. On the other hand, if the participant P answers "Yes.", The dialogue module 50 recognizes "Yes" by the voice recognition circuit 55 based on the acoustic signal from the microphone 16, and the voice is synthesized by the dialogue control circuit 53. And the speaker 5
From 1, he says, "Welcome XXX, please enter the room."

The face of the participant P is stored in the face database 38.
If that is not registered, the association module 60 is to control the interaction module 50, the question for the "Hello. Can you tell me your name?" And participants P by speech synthesis. On the other hand, when the participant P answers his name "XXX.", The dialogue module 50 recognizes "XXX" by the voice recognition circuit 55 based on the acoustic signal from the microphone 16, and the dialogue is performed. Speech synthesis is performed by the control circuit 53, and the speaker 51 utters "Welcome XXX, please enter the room." In this way, the robot 10
Performs recognition of the participant P to guide entry to the party venue, and causes the visual module 30 to register the face image and the name “XXX” of the participant P in the face database 38.

The humanoid robot 10 operates as a companion robot as follows. in this case,
The humanoid robot 10 aurally and visually recognizes a plurality of speakers based on an auditory event 28 by the auditory module 20 and a visual event 39 by the visual module 30 and an association stream 65 by the association module 60. One of the speakers can be tracked, or the other speaker can be switched and tracked along the way. In the case of a companion robot, the robot 10 plays a passive role, that is, plays only "listen" or "looks at the speaker" of the party participant, and does not utter by the dialog module 50. .

The humanoid robot 10 as the companion robot may share the face database 38 with the party reception robot, or the face database 38 of the party reception robot may be transferred or copied. In this case, the humanoid robot 10 as the companion robot can always recognize all the party participants by face identification.

As described above, according to the humanoid robot 10 according to the embodiment of the present invention, the association module 60 uses these directional information and individual information based on the auditory and visual events from the auditory module 20 and the visual module 30. From the speaker identification of the speaker, the auditory stream, the visual stream, and the association stream are generated in consideration of these temporal flows, so that a plurality of speakers are recognized. Multiple times in real time, for example, when the speaker moves and becomes `` invisible '' due to hearing or when the speaker becomes `` inaudible '' without speaking, Speakers can be tracked audibly and / or visually.

In the above-described embodiment, the humanoid robot 10 is configured to have 4 DOF (degree of freedom). However, the present invention is not limited to this. The robot according to the present invention may be a robot configured to perform an arbitrary operation. It is also possible to incorporate a hearing system. Further, in the above-described embodiment, the case where the robot audiovisual system according to the present invention is incorporated in the humanoid robot 10 has been described. However, the present invention is not limited to this. Obviously, it is also possible to incorporate.

[0070]

As described above, according to the present invention,
The auditory module identifies the sound source of each speaker by obtaining the direction of each sound source by performing pitch extraction using the harmonic structure from the sound from the external object collected by the microphone, and identifying the sound source of each speaker. Is extracted. Also, the visual module
From the image captured by the camera, each speaker is identified from the face identification and localization of each speaker by pattern recognition, and the visual event of each speaker is extracted. Further, the motor control module extracts a motor event by detecting a direction of the robot based on a rotation position of a drive motor that rotates the robot in a horizontal direction.

Here, the association module is
Based on the auditory event, visual event, and motor event thus extracted, an audio stream and a visual stream of each speaker are generated with reference to the direction information and the speaker identification, and these streams are further generated. By generating association streams in association with each other, the attention control module performs attention control based on these streams, thereby planning the drive motor control of the motor control module. At this time, the association module determines the direction of each speaker based on the sound source localization of the auditory event and the face localization of the visual event, that is, the audio and visual direction information, and generates an auditory stream, a visual stream, and an association stream. Will do.

Then, the attention control module:
By controlling the drive motor of the motor control module based on this planning, the direction of the robot is directed to the target speaker. With this, the robot can directly face the target speaker, so that the hearing module can accurately collect and localize the voice of the speaker by the microphone in a highly sensitive frontal direction, The module can better image the speaker with the camera.

Therefore, the cooperation of the hearing module, the visual module and the motor control module with the association module and the attention control module makes it possible to refer to the directional information, the speaker identification and the temporal flow of the auditory and visual events. By tracking the speakers, the ambiguities of the robot's hearing and vision are complemented by each other, and the so-called robustness is improved. Can be reliably perceived. Also, for example, even when either the auditory event or the visual event is lost, the association module can perceive the target speaker based on only the remaining visual event or the auditory event, so that accurate Then, the direction of the target can be grasped and the motor control module can be controlled. Thus, according to the present invention, there is provided an extremely excellent robot audio-visual system that integrates visual and auditory information on an object to reliably track the object.

[Brief description of the drawings]

FIG. 1 is a front view showing the appearance of a humanoid robot incorporating a first embodiment of a robot hearing device according to the present invention.

FIG. 2 is a side view of the humanoid robot of FIG. 1;

FIG. 3 is a schematic enlarged view showing a configuration of a head in the humanoid robot of FIG. 1;

FIG. 4 is a block diagram showing an electrical configuration of a robot audiovisual system in the humanoid robot of FIG. 1;

5 is a block diagram of an electrical configuration showing an enlarged view of a hearing module of block 1 in FIG. 4;

6 is a block diagram of an electrical configuration showing a visual module of a block 2 in FIG. 4 in an enlarged manner.

FIG. 7 is a block diagram of an electric configuration showing a motor control module of a block 3 in FIG. 4 in an enlarged manner.

FIG. 8 is a block diagram of an electric configuration showing a dialog module of block 4 in FIG. 4 in an enlarged manner.

9 is an enlarged block diagram of an electrical configuration showing an association module of a block 5 in FIG. 4;

FIG. 10 is a diagram illustrating peak extraction, sound source localization, and sound source separation by an auditory module in the robot audiovisual system of FIG. 4;

FIG. 11 is a diagram showing stream generation by an association module in the robot audiovisual system of FIG. 4;

12 is a diagram illustrating an operation example as a party reception robot in the robot audiovisual system of FIG. 4;

[Explanation of symbols]

 Reference Signs List 10 humanoid robot 11 base 12 torso 13 head 13a connecting member 14 exterior 15 camera (robot vision) 16, 16a, 16b microphone (robot hearing) 17 robot audiovisual system 20 hearing module 30 visual module 40 motor control module 50 dialog module 60 Association Module 64 Attention Control Module 70 Network

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification code FI Theme coat ゛ (Reference) G06T 7/00 300 G06T 7/00 300F 5L096 7/60 150 7/60 150B G10L 15/28 H04N 7/18 Z 17/00 G10L 3/00 511 15/00 545F 15/22 551H 15/20 571T 21/02 3/02 301C 15/02 9/00 301A H04N 7/18 F term (reference) 2C150 BA11 CA01 CA02 CA04 DA04 DA05 DA24 DA25 DA26 DA27 DA28 DF03 DF04 DF06 DF33 ED42 ED47 ED52 EF07 EF16 EF17 EF23 EF29 EF33 EF36 3C007 AS34 AS36 CS08 JS03 KS04 KS08 KS18 KS39 KT01 LT08 NS01 WA02 WA03 CC05 CB19 CC05 CCB5 CC05 CCB5 CC05 CCB5 CC05 CCB5 CC05 CC05 CCB5 CC05 CCB5 CC05 CCB5 CC05 CCB5CC 5D015 AA03 CC13 DD02 EE04 KK01 KK04 LL06 5L096 BA05 BA16 BA18 CA02 DA05 FA67 HA05 JA11 LA12

Claims (6)

[Claims] 1. A hearing module including at least a pair of microphones for collecting external sounds, a visual module including a camera for capturing an image in front of the robot, and a motor control module including a drive motor for rotating the robot in a horizontal direction. An association module that integrates events from the hearing module, the vision module, and the motor control module to generate a stream, and an attention control module that performs attention control based on the stream generated by the association module. A robot audiovisual system, wherein the auditory module determines a direction of at least one speaker from pitch extraction, sound source separation and localization based on an acoustic signal from a microphone, and extracts the auditory event, Visual module Based on the image captured by the camera, identify the sound source of each speaker from the face identification and localization of each speaker, extract the visual event, the motor control module, the rotation position of the drive motor Extracting the motor event based on the audio event, the association module determines the direction of each speaker from the auditory event, the visual event and the motor event based on the direction information of the sound source localization of the auditory event and the face localization of the visual event. Generating an audio stream and a visual stream, and further associating them to generate an association stream, wherein the attention control module is configured to generate an attention stream based on the streams for planning the drive motor control of the motor control module. Performing control. Tsu door audiovisual system. 2. A hearing module including at least a pair of microphones for collecting external sounds, a visual module including a camera for capturing an image in front of the robot, and a motor control module including a driving motor for rotating the robot in a horizontal direction. An association module that integrates events from the hearing module, the vision module, and the motor control module to generate a stream, and an attention control module that performs attention control based on the stream generated by the association module. An audiovisual system for a humanoid or animal robot, wherein the auditory module determines a direction of at least one speaker from pitch extraction, sound source separation and localization based on an acoustic signal from a microphone. , Extract that auditory event The visual module identifies each speaker from the face identification and localization of each speaker based on an image captured by a camera, and extracts a visual event of the speaker. By extracting a motor event based on the rotational position of the audio event, the association module can determine each story based on the sound source localization of the auditory event and the face localization direction information of the visual event from the auditory event, the visual event, and the motor event. Generating an auditory stream and a visual stream by determining the direction of the driver, and further generating an association stream by associating the streams with the audio stream and the visual stream, wherein the attention control module controls the drive motor control of the motor control module based on the streams. Perform attention control for planning. Characterized in that, the robot audiovisual system. 3. The method as claimed in claim 1, wherein the association module synchronizes the asynchronously generated auditory, visual, and motor events with each other when generating the auditory stream and the visual stream.
Or the robot audiovisual system according to 2. 4. The hearing module detects a MFCC of speech from an audio signal to identify each speaker, and the association module determines the speaker based on an auditory event speaker identification and a visual event speaker identification. The robot audiovisual system according to any one of claims 1 to 3, wherein an audio stream and a visual stream to which the audio event and the visual event are to be connected are selected by specifying a speaker. 5. The association module, when a plurality of streams approach each other, refers to a temporal flow of the auditory event and the visual event, and generates an auditory stream and a visual stream to which the auditory event and the visual event should be connected. The robot audiovisual system according to any one of claims 1 to 4, wherein the robot audiovisual system is selected. 6. The association module generates an association stream by associating an auditory stream and a visual stream that are strongly associated with each other, and when the association between the auditory stream and the visual stream that make up the association stream is weakened,
The robot audiovisual system according to any one of claims 1 to 5, wherein the association is canceled to cause the association stream to disappear.