JP2002264052A - Robot audio-visual system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a robot audio-visual system allowing real-time processing for the visual and auditory tracing of an object and visualizing the real-time processing. SOLUTION: An auditory module 20 extracts an auditory event 28 by identifying the sound source of a speaker from an acoustic signal of a microphone. A visual module 30 extracts a visual event 39 by the face identification and orientation of the speaker. A motor control module 40 extracts a motor event 49. An association module 60 creates an auditory stream 65 and a visual stream 66 from the auditory event, visual event and motor event and creates an association stream 67 by associating these streams. An attention control module 64 performs attention control for the planning of drive motor control, and display parts 27, 37, 48, 68 are provided for displaying at least part of auditory information, visual information, motor information and stream information.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an audiovisual system for a robot, particularly a humanoid or animal robot.

[0002]

2. Description of the Related Art In recent years, attention has been paid to active perception of sight and hearing in such humanoid or animal type robots. Active perception is to control the posture of a head that supports these perception devices, such as a head, by a drive mechanism so that the perception device that is responsible for perception such as robot vision and robot hearing follows an object to be perceived. .

Here, with regard to active vision, at least a camera, which is a perceptual device, holds its optical axis direction toward a subject by attitude control by a driving mechanism, and automatically focuses, zooms in, and zooms out on the subject. As a result, the object is imaged by a camera, and various studies have been made.

[0004] On the other hand, regarding active hearing, at least a microphone, which is a perceptual device, maintains its directivity toward an object by attitude control by a drive mechanism, and sounds from the object are collected by the microphone. Such active hearing is disclosed, for example, in Japanese Patent Application No. 2000-2267 by the present applicant.
No. 7 (robot hearing system), which directs a sound source with reference to visual information.

[0005]

Incidentally, these active vision and active hearing are closely related to a motor control module for changing the orientation (horizontal direction) of the robot, and the active vision and the active hearing are performed for a specific object. In order to activate active hearing, it is necessary to aim the robot at a specific target, that is, to perform attention control. However, in order to integrate vision and hearing with the control of the motor control module, real-time processing for tracking vision and hearing is required, and such visual and auditory tracking status is visualized in real-time processing. It is very useful to understand the real-time processing inside the robot, but in the conventional robot development, one that performs real-time processing for a single sound source has been realized. In a situation where humans are talking to each other, it is not performed to identify each person by real-time processing and perform active hearing,
Further, visualization of such real-time processing has not been performed.

In view of the above points, the present invention provides a robot audio-visual system that enables real-time processing for tracking the visual and auditory sense of an object and visualizes the real-time processing. It is intended to be.

[0007]

According to the present invention, there is provided a hearing module including at least a pair of microphones for collecting external sounds, a visual module including a camera for capturing an image in front of the robot, and a robot. A motor control module that includes a drive motor that rotates the camera horizontally, an association module that integrates events from the auditory module, the vision module, and the motor control module to generate a stream, and a stream generated by the association module. An attention control module for performing attention control, wherein the hearing module is configured to detect at least one speaker's sound source from pitch extraction, sound source separation and localization based on a sound signal from a microphone. To identify the auditory The visual module extracts the visual event from the face identification and localization of each speaker based on the image captured by the camera, and the motor control module outputs the motor event based on the rotational position of the drive motor. By extracting the events, the association module divides the auditory, visual and motor events into auditory and visual streams,
An association control stream is generated by associating them with each other, and the attention control module performs attention control for planning the drive motor control of the motor control module based on these streams. Information and motor information by motor module,
The present invention is attained by a robot audio-visual system that includes a display unit that displays at least a part of stream information by an association module.

Further, according to the present invention, there is provided a hearing module including at least a pair of microphones for collecting external sounds, a visual module including a camera for capturing an image in front of the robot, and A motor control module including a driving motor to be rotated; an association module that integrates events from the hearing module, the vision module, and the motor control module to generate a stream; and performs attention control based on the stream generated by the association module. An audio-visual system for a humanoid or animal-type robot, comprising: an attention control module, the audio-visual module comprising at least one talker based on an acoustic signal from a microphone, from pitch extraction, sound source separation and localization. Identify the sound source of the person To extract the auditory event, visual module, based on the image captured by the camera,
Extracting the visual event from each speaker's face identification and localization,
The motor control module extracts a motor event based on the rotational position of the drive motor, so that the association module converts the auditory stream, the visual stream, and the motor event from the auditory stream, the visual stream, and the association stream that associates the audio stream and the visual stream with each other. Then, the attention control module performs attention control for the planning of the drive motor control of the motor control module based on these streams, and outputs the auditory information by the auditory module, the visual information by the visual module, and the motor information by the motor module. And a robot audiovisual system, comprising a display unit for displaying at least a part of the stream information by the association module,
Achieved.

[0009] In the robot hearing device according to the present invention, preferably, the display unit includes an auditory display unit that displays a spectrum of an acoustic signal from a sound source, an extracted peak, and an auditory event as auditory information.

In the robot auditory apparatus according to the present invention, preferably, the auditory display unit comprises an auditory event represented by a circle having a vertical axis representing a relative azimuth about the robot, a horizontal axis representing a pitch, and a diameter representing a certainty factor. Is displayed.

Preferably, in the robot hearing device according to the present invention, the display unit includes a camera image indicating the extracted face as a frame as visual information and a visual display unit for displaying a visual event.

In the robot hearing device according to the present invention, preferably, the visual display unit displays a visual event by a list of face identification and face localization extracted with certainty.

[0013] In the robot hearing device according to the present invention, preferably, the display unit includes a motor display unit that three-dimensionally displays the direction and the operation speed of the robot in real time as motor information.

In the robot hearing device according to the present invention, preferably, the display unit includes a stream display unit that displays a stream chart and a radar chart as stream information.

[0015] In the robot hearing device according to the present invention, preferably, the stream display unit displays stream information in the form of a stream chart with each of an auditory stream, a visual stream, and an association stream.

[0016] In the robot hearing device according to the present invention, preferably, the stream display unit includes a radar chart.
The stream state at that time is displayed by the camera's visual field and sound source localization.

According to the above configuration, the auditory module obtains the direction of each sound source by performing pitch extraction using the harmonic structure from the sound from the external object collected by the microphone, thereby obtaining the direction of each sound source. And the auditory event is extracted. Further, the visual module extracts, from the image captured by the camera, a visual event of each speaker from face identification and localization of each speaker by pattern recognition. further,
A motor control module extracts a motor event by detecting a direction of the robot based on a rotation position of a drive motor that rotates the robot in a horizontal direction.
Note that the event indicates a state in which a sound or face is detected at each time point, features such as pitch and direction are extracted, speaker identification and face identification are performed, or a drive motor is rotated. The stream indicates an event that is temporally continuous.

Here, the association module is
Based on the auditory event, visual event, and motor event thus extracted, an auditory stream and a visual stream of each speaker are generated, and the streams are associated with each other to generate an association stream. However, by performing attention control based on these streams, planning of drive motor control of the motor control module is performed. Attention refers to the auditory and / or visual "attention" of the speaker to which the robot is intended. Attention control means that the robot changes its orientation by means of a motor control module so that the robot can speak. To pay attention to the person. Then, the attention control module controls the drive motor of the motor control module based on the planning, thereby directing the robot to the target speaker. This allows the robot to face the target speaker so that the auditory module can accurately collect the speaker's voice with the microphone, and the visual module can capture the speaker's image with the camera. Good imaging can be performed.

Therefore, the cooperation of the hearing module, the vision module, and the motor control module with the association module and the attention control module complements the ambiguity of the hearing and vision of the robot. Robustness is improved, and even if there are a plurality of speakers, each speaker can be perceived. Also, for example, even when either the auditory event or the visual event is missing, the attention control module can track the target speaker based on only the visual event or the auditory event, so that The control of the motor control module can be performed.

Further, the display unit displays at least a part of the auditory information by the auditory module, the visual information by the visual module, the motor information by the motor module, and the stream information by the association module, and visualizes the real-time processing by the association module. This makes it possible to visually and intuitively grasp the state of the real-time processing.

In the case where the display unit includes an auditory display unit that displays, as auditory information, a spectrum of an acoustic signal from a sound source, an extracted peak, and an auditory event.
When the auditory display unit displays an auditory event by a circle having a relative azimuth centered on the robot on the vertical axis, a pitch on the horizontal axis, and a certainty factor on the diameter, look at the auditory display unit. Thereby, the auditory information can be grasped intuitively.

When the display unit includes a camera image indicating the extracted face as a frame as visual information, and a visual display unit for displaying a visual event, the visual display unit includes:
When a visual event is displayed based on the list of face identification and face localization extracted with certainty, visual information can be intuitively grasped by looking at the visual display unit. When the display unit includes a motor display unit that three-dimensionally displays the direction and the speed of the robot in real time as the motor information, the motor information is intuitively grasped by looking at the motor display unit. be able to.

When the display unit has a stream display unit for displaying a stream chart and a radar chart as stream information, and the stream display unit displays each of an auditory stream, a visual stream, and an association stream in a stream chart. When the stream information is displayed by, the stream display unit further displays the stream state by the camera chart and the sound source localization in the radar chart. The state and the change state of each stream can be grasped intuitively.

As described above, since the information of symbolic events and streams is handled as the auditory information and the visual information, the data amount is greatly reduced as compared with the case where the raw audio data and the image data are handled. Therefore, real-time display on the display unit becomes possible.

[0025]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail based on embodiments shown in the drawings. 1 to 4 show the overall configuration of an experimental humanoid robot provided with an embodiment of the robot audiovisual system according to the present invention. In FIG. 1, the humanoid robot 10 has 4 DOF (degree of freedom).
The robot includes a base 11, a body 12 supported on the base 11 so as to be rotatable around one axis (vertical axis), and a three-axis direction (vertical axis, (A horizontal axis in the left-right direction and a horizontal axis in the front-rear direction).

The base 11 may be fixedly arranged, and may be operable as a leg. Also base 1
1 may be mounted on a movable cart or the like.
The body portion 12 is rotatably supported as shown by an arrow A in FIG. 1 around a vertical axis with respect to the base 11, and is driven to rotate by driving means (not shown). , Covered by a soundproof exterior.

The head 13 is supported on the body 12 via a connecting member 13a.
Can swing about a horizontal axis in the front-rear direction, as shown by an arrow B in FIG. 1, and can swing about a horizontal axis in the left-right direction, as shown by an arrow C in FIG. In addition, the connecting member 13a is swingably supported as shown by an arrow D in FIG. It is rotationally driven in the directions of the arrows A, B, C, and D by the non-driving means.

Here, the head 13 is entirely covered with a soundproof exterior 14 as shown in FIG.
A camera 15 is provided on the front side as a visual device in charge of robot vision, and a pair of microphones 16 (16a, 16b) are provided on both sides as hearing devices in charge of robot hearing.

The exterior 14 is made of a sound-absorbing synthetic resin such as a urethane resin, for example. The interior of the head 13 is sound-insulated by almost completely sealing the interior of the head 13. Have been. The exterior of the body 12 is also made of a synthetic resin having a sound absorbing property. The camera 15 has a known configuration, and for example, a commercially available camera having a so-called pan, tilt, and zoom 3DOF (degree of freedom) can be applied.

The microphones 16 are mounted on the side surfaces of the head 13 so as to have directivity toward the front. Here, the left and right microphones 16a and 16b of the microphone 16 are, as shown in FIG. 1 and FIG.
b, the sound is collected by a suitable means so as to collect sound from the front through through holes provided in the steps 14a and 14b and not to pick up sound inside the exterior 14. Thus, the microphones 16a and 16b
It is configured as a so-called binaural microphone. In addition,
In the vicinity of the mounting positions of the microphones 16a and 16b, the exterior 14 may be formed in the shape of a human outer ear.

FIG. 4 shows an electrical configuration of the robot audiovisual system including the microphone 16 and the camera 15. In FIG. 4, the audiovisual system 17 is configured as a party reception and companion robot, and includes an audio module 20, a visual module 30, a motor control module 40, a dialog module 50, and an association module 60. Hereinafter, a further description will be given with reference to FIGS. For convenience of explanation, the auditory module 20 is block 1
5, the visual module 30 is shown as a block 2 in FIG. 6, the motor control module 40 is shown as a block 3 in FIG. 7, and the dialogue module 50 is a block 4 in FIG. Shown enlarged,
Further, the association module 60 is set to block 5
FIG. 9 is an enlarged view of FIG. Here, the association module 60 (block 5, FIG. 9) is composed of a server, and has other modules, ie, the hearing module 20 (block 1, FIG. 5), the visual module 30 (block 2, FIG. 6), and the motor. The control module 40 (block 3, FIG. 7) and the dialogue module 50 (block 4, FIG. 8) are each composed of a client and operate asynchronously with each other.

The server and each client are composed of, for example, a personal computer, and are mutually LA-connected via a network 70 such as 100Base-T using, for example, the TCP / IP protocol.
N connections are made. In addition, each module 20, 30, 4
0, 50, and 60 are hierarchically distributed, and specifically, are device layers, process layers, feature layers,
It consists of an event layer.

As shown in FIG. 5, the hearing module 2
0 denotes a microphone 16 as a device layer, a peak extraction unit 21, a sound source localization unit 22, and a sound source separation unit 23 as process layers, a pitch 24 as a characteristic layer (data), a horizontal direction 25, and an event layer as an event layer. Auditory event generator 26
And a viewer 27.

Thus, the hearing module 20 extracts a series of peaks for each of the left and right channels by the peak extracting unit 21 based on the acoustic signal from the microphone 16, and pairs the same or similar peaks in the left and right channels. And Here, the peak extraction is performed by using a bandpass filter that transmits only data under the condition that the power is equal to or higher than the threshold value and has a maximum value, for example, a frequency between 90 Hz and 3 kHz. This threshold is
It is defined as a value obtained by measuring ambient noise and adding a sensitivity parameter, for example, 10 dB.

The hearing module 20 uses the fact that each peak has a harmonic structure to find a more accurate pair of peaks between the left and right channels, and for each pair of peaks of the left and right channels. The sound source separation unit 23 applies an inverse FFT (Fast Fourier Transform) to separate a sound having a harmonic structure from a mixed sound from each sound source. Thereby, the hearing module 20 selects the sound signal of the same frequency from the left and right channels by the sound source localization unit 22 for each of the separated sounds, and for example, performs the IPD every 5 degrees.
(Interaural phase difference) and IID (aural intensity difference).

The sound source localization unit 22 of the auditory module 20 generates a hypothesis of the IPD Ph by hypothesis inference within a range of ± 90 degrees with the front of the robot 10 being 0 degrees by using a so-called auditory epipolar geometry. ,

(Equation 1) Then, the distance d (θ) between the sound separated by the above and each hypothesis is calculated. Here, n _f <1.5 kHz means that the frequency is 1.5 kHz
Hz. This is the left and right microphone 15
Since the IPD is effective for frequencies of 1.2 to 1.5 kHz or less from the baseline, the experiment was performed at 1.5 kHz or less in this experiment.

The IID is obtained from the power difference between the left and right channels of each overtone of the separated sound, similarly to the IPD. However, IID is not hypothetical inference,

(Equation 2) The sound source is determined to be either left or right using a discriminant function based on That is, the IID of each overtone of the frequency f is I
_{Assuming that s} (f), the sound source exists in the left direction of the robot when I is positive, in the right direction when I is negative, and in the front direction when I is almost zero. Where IID
Since the generation of a hypothesis requires an enormous amount of calculation considering the shape of the head of the robot 10, a hypothesis inference similar to that of the IPD is not performed in consideration of real-time processing.

The sound source localization unit 22 of the hearing module 20 calculates the probability density function from the distance d (θ).

(Equation 3) Is used to calculate the _IPD certainty factor BF _IPD (θ). Here, m and s are the average and variance of d (θ), respectively, and n is the number of d. In addition, the confidence B of the IID
F _IID (θ) is 30 degrees <θ ≦ 90 degrees, 0.35 when I is +, 0.65 when −, and −30 degrees <θ ≦ 90.
In degrees, 0.5 when I is +, 0.5 when −, -9
0 degree <θ ≦ −30 degrees and 0.65−
It becomes 0.35 at the time of.

Then, the confidence BF _IPD (θ) of the IPD thus obtained and the confidence BF of the IID
_IID (θ)

(Equation 4) Are integrated according to the Dempster-Shafer theory shown by the formula (1) to generate a certainty factor BF _{IP D + IID} (θ). As a result, the auditory module 20 can generate the auditory event generator 2
6, the auditory event 28 is generated from the list of the top 20 certainty factors BF _{IPD + IID} (θ) and the directions (θ) in the descending order of the likelihood as the sound source direction, and the pitch.

In this way, the hearing module 20
Based on the sound signal from the microphone 16, at least one speaker's sound source is identified from pitch extraction, sound source separation and localization, and its auditory event is extracted.
0 to the association module 60. The above-described processing in the hearing module 20 is performed every 40 msec.

The viewer 27 displays the auditory event 28 generated as described above on the screen of the client. Specifically, as shown in FIG. Hearing event 2
8 is displayed by, for example, a black curve 27b, and its peak is displayed by, for example, a blue vertical line 27b.
c, the automatically measured background noise level is displayed by, for example, a red curve 27d.
In FIG. 7e, a graph of the auditory event 28 having a relative azimuth on the vertical axis and a pitch (frequency) on the horizontal axis is displayed. Here, each auditory event is represented by a circle 27f in which the certainty factor of the sound source localization is the diameter of the circle. Thereby, by looking at the display of the viewer 27, the auditory event 28 can be visually and intuitively grasped by the power spectrum of the auditory event 28, the extracted peak, and the circle display on the graph. .

As shown in FIG. 6, the visual module 3
0 indicates a camera 15 as a device layer, a face detection unit 31, a face identification unit 32, and a face localization unit 33 as process layers;
A face ID 34 and a face direction 35 as feature layers (data);
It comprises a visual event generator 36 and a viewer 37 as an event layer.

Thus, the visual module 30 detects the face of each speaker by, for example, extracting skin color by the face detecting unit 31 based on the image signal from the camera, and the face registered in advance by the face identifying unit 32. When there is a matched face searched by the database 38, the face ID 34 is determined to identify the face, and the face direction 35 is determined (localized) by the face localization unit 33. When the face identification unit 32 searches the face database 38 and finds no matching face, the face learning unit 38a registers the face detected by the face finding unit 31 in the face database 38.

Here, when the face finding unit 31 finds a plurality of faces from the image signal, the visual module 30 performs the above-described processing, that is, identification, localization, and tracking for each face.
At this time, since the size, direction, and brightness of the face detected by the face detection unit 31 often change, the face detection unit 31
Is capable of accurately detecting a plurality of faces within 200 msec by performing face area detection and combining pattern extraction based on skin color extraction and correlation calculation.

The face identifying section 32 projects each face area image detected by the face finding section 31 into the discrimination space, and calculates a distance d from the face data registered in the face database 38 in advance. Since this distance d depends on the number of registered faces (L),

(Equation 5) Is converted to a certainty factor Pv independent of parameters. Here, the discriminant matrix serving as the basis of the discriminant space can be updated by a known online LDA with a smaller number of calculations compared to a normal LDA, so that face data can be registered in real time.

The face localization unit 33 converts the face position on the two-dimensional image plane into a three-dimensional space, and the width and height of the face at (x, y) on the image plane are X and Y, respectively. Assuming that there are w × w pixels, the face position in the three-dimensional space is obtained as a set of an azimuth angle θ, a height φ, and a distance r given by the following equations.

(Equation 6)

(Equation 7)

(Equation 8) Here, C ₁ and C ₂ are constants defined by the search image size (X, Y), the angle of view of the camera, and the actual face size.

Then, the visual module 30 generates a visual event 39 from the face ID (name) 34 and the face direction 35 for each face by the visual event generator 36. In detail, the visual event 39 includes, for each face, the top five face IDs (names) 34 with certainty factors and positions (distance r, horizontal angle θ, and vertical angle φ).

The viewer 37 displays a visual event on the screen of the client.
As shown in FIG. 1 (B), the image 37a by the camera 15
, A list 37b of face IDs and positions of the extracted faces, a face ID of each extracted face, and a certainty factor. A list 37c of distances is displayed. Here, in the image 37a by the camera 15, the face found and identified is displayed by being surrounded by a rectangular frame. In the case shown in the figure, since a plurality of faces are found, rectangular frames 37d (for example, red display) and 37e (for example, yellow display) indicating the identification are displayed for each face. Along with this, in the case shown, the list 37b is also displayed for each face. By viewing the display of the viewer 37, the frame 37c. The visual event 39 can be visually and intuitively grasped by the extracted face indicated by 37d, the face localization list 37b, and the face ID list extracted with certainty.

As shown in FIG. 7, the motor control module 40 includes a motor 41 and a potentiometer 42 as device layers, a PWM control circuit 43, an AD conversion circuit 44 and a motor control unit 45 as process layers, and a characteristic layer. , A motor event generator 47 as an event layer, and a viewer 48.

Thus, the motor control module 40
Is controlled by the motor control unit 45 based on a command from the attention control module 64 (described later).
The drive of the motor 41 is controlled via the
1 is detected by the potentiometer 42,
A motor control unit 45 extracts a robot direction 46 via an AD conversion circuit 44, and a motor event generation unit 47 generates a motor event 49 including motor direction information.

The viewer 48 is for displaying motor events three-dimensionally on the screen of the client. Specifically, as shown in FIG.
Using the three-dimensional viewer implemented by L, the direction of the robot 10 due to the motor event 49 is determined by the direction of the robot 48a in the three-dimensional display and the direction of the arrow 48b in the red display, and the operation speed of the robot 10 Three-dimensional display is performed in real time by the length of the arrow 48b. Thus, by viewing the display of the viewer 48, the three-dimensional display of the robot 10 by the motor event 49 allows the motor event 49 to be grasped visually and intuitively. Note that the viewpoint of the robot 10 can be arbitrarily changed, and zoom-up and zoom-out are also possible.

As shown in FIG.
0 is a speaker 51 and a microphone 16 as device layers.
And a speech synthesis circuit 52 as a process layer, a dialogue control circuit 53, a self-voice suppression circuit 54, and a speech recognition circuit 55.

Thus, the dialogue module 50 controls the dialogue control circuit 53 by the later-described association module 60 and the speaker 51 by the voice synthesis circuit 52.
, A predetermined sound is emitted to the target speaker, the sound from the speaker 51 is removed from the acoustic signal from the microphone 16 by the self-voice suppression circuit 54, and then the target Recognize the speaker's voice. The interaction module 50 does not have a feature layer and an event layer as a hierarchy.

Here, for example, in the case of a party reception robot, the top priority is to continue the current attention, but in the case of a party robot, the dialog control circuit 53 assigns the most recently associated stream to the stream. Attention control is performed.

As shown in FIG. 9, the association module 60 is hierarchically positioned higher than the above-described auditory module 20, visual module 30, motor control module 40, and dialog module 50. The stream layer, which is the upper layer of the 20, 30, 40, and 50 event layers, is configured. Specifically, the association module 60 synchronizes the asynchronous event 61a from the auditory module 20, the visual module 30, and the motor control module 40, that is, the auditory event 28, the visual event 39, and the motor event 49 into a synchronous event 61b. A circuit 62, a stream generation unit 63 that correlates these synchronization events 61b to generate an auditory stream 65, a visual stream 66, and an association stream 67; an attention control module 64;
It has.

The synchronizing circuit 62 is connected to the hearing module 20.
The synchronizing auditory event 28, the visual event 38 from the visual module 30, and the motor event 49 from the motor control module 40 are synchronized to generate a synchronous auditory event, a synchronous visual event, and a synchronous motor event. At that time, the coordinate system of the auditory event 28 and the visual event 38 is converted into the absolute coordinate system by the synchronous motor event.

Here, the delay time from when each event is actually observed to when it reaches the association module 60 via the network 70 is, for example, 40 msec for the auditory event 28 and 200 m for the visual event 39.
Second, the motor event 49 is 100 m, the delay in the network 70 is 10 to 200 msec, and the arrival period is also different. Therefore,
To synchronize each event, the hearing module 20,
The auditory event 28, the visual event 39, and the motor event 49 from the visual module 30 and the motor control module 40 each have time stamp information indicating the actual observation time.
For example, it is temporarily stored only for two seconds.

The synchronization circuit 62 compares the events stored in the short-term storage circuit with a delay time of 500 ms in comparison with the actual observation time in consideration of the delay time described above. Remove by As a result, the response time of the synchronization circuit 62 becomes 500 ms. Further, such a synchronization process operates at a period of, for example, 100 ms. Since each event arrives at the association module 60 asynchronously with each other, an event at the same time as the observation time for synchronization is not always present. Therefore, in the synchronization process, interpolation is performed by linear interpolation on events that occur before and after the observation time for synchronization.

The stream generator 63 generates streams 65, 66, and 67 based on the following points. 1. The auditory event 28 is connected to the nearest auditory stream 65 with pitches that are equivalent or harmonically related and within ± 10 degrees of direction. The value within ± 10 degrees is selected in consideration of the accuracy of the auditory epipolar geometry. 2. The visual event 39 has a common face ID 34 and is connected to the closest visual stream 66 within 40 cm. The value in the range of 40 cm is selected on the assumption that a human does not move at a speed of 4 m or more per second. 3. As a result of searching all the streams, if there is an event for which there is no connectable stream 65, 66, the event 28, 39 constitutes a new stream 65, 66. 4. The streams 65 and 66 that have already existed remain for a maximum of 500 ms when there are no events 28 and 39 connected to them, but disappear when the event continues to be disconnected. 5. Auditory stream 65 and visual stream 66 are ± 1
If the proximity state within 0 degrees continues for 500 ms or more in one second, the auditory stream 65 and the visual stream 66 are considered to be of the same speaker, and are associated with each other and associated. A stream 67 is generated. 6. The association stream 67 is dissociated if the auditory event 28 or visual event 39 is not connected for more than 3 seconds, and only the existing auditory stream 65 or visual stream 66 remains. 7. The association stream 67 has a direction difference between the auditory stream 65 and the visual stream 66 of 3 seconds, ±
If the angle exceeds 30 degrees, the association is released, and the process returns to the individual auditory streams 65 and the visual streams 66.

Thus, based on the synchronous auditory event and the synchronous visual event from the synchronous circuit 62, the stream generation unit 63 connects the events in consideration of their temporal connection, thereby forming the auditory stream 65 and the visual stream. In addition to generating the association stream 67, the association stream 67 is generated by associating the auditory stream 65 and the visual stream 66 that are strongly connected to each other.
Auditory stream 65 and visual stream 66 constituting
When the connection becomes weak, the association is broken.

The attention control module 64
Performs attention control for planning the drive motor control of the motor control module 40. At this time, the attention control is performed by preferentially referring to the association stream 67, the auditory stream 65, and the visual stream 66 in this order. Then, the attention control module 64 performs the operation planning of the robot 10 based on the state of the auditory stream 65 and the visual stream 66 and the presence or absence of the association stream 67. If the operation of the drive motor 41 is required, the motor control module 40 , A motor event as an operation command is transmitted via the network 70.

Here, the attention control module 64
Attention control in is based on continuity and triggers, trying to keep the same state with continuity and trying to track the objects of most interest with triggers. Therefore, attention control includes: The presence of the association stream indicates that the person who is speaking directly to the robot 10 is still present or has been present in the near future, It is necessary to focus attention on a person who has high priority and perform tracking. 2. Since the microphone 16 is omnidirectional, there is no detection range such as the viewing angle of the camera, and a wide range of auditory information can be obtained. Therefore, the priority of the auditory stream should be higher than that of the visual stream. In consideration of these two points, a stream to which attention is directed is selected and tracking is performed according to the following principle. 1. Prioritize association stream tracking. 2. If there is no association stream,
Prioritize audio stream tracking. 3. If the association stream and the auditory stream do not exist, priority is given to tracking the visual stream. 4. If there are multiple streams of the same type, priority is given to tracking the oldest stream. In this way, the attention control module 64
By performing attention control, the motor control module 4
0, the control of the drive motor 41 is planned, a motor command 66 is generated based on the plan, and the motor control module 4 is controlled via the network 70.
Transmit to 0. Thereby, the motor control module 40
Then, based on the motor command 66, the motor control unit 45 performs PWM control to rotate and drive the drive motor 41 so as to direct the robot 10 in a predetermined direction.

The viewer 68 displays each stream generated in this way on the screen of the server. Specifically, as shown in FIG. 12 (B), the viewer 68 uses a radar chart 68a and a stream chart 68b. indicate. Here, the radar chart 68a shows the state of the association stream at that moment, for example, with the camera viewing angle 68a1 shown as a wide and bright (pink in the figure) fan shape and the stream direction 68a2 shown as a narrow dark fan shape. . Here, the stream direction 68a2 is displayed, for example, in red when there is an auditory stream and a visual stream, is displayed in blue, for example, when there is only an auditory stream, and is displayed in green, for example, when there is only a visual stream. The stream chart 68b shows an association stream 68b1 indicated by a thick line and an auditory stream or visual stream 68b2 indicated by a thin line. Here, the association stream 68b1 is displayed, for example, in red when the auditory stream and the visual stream exist, is displayed in blue, for example, in the case of only the audio stream, and is displayed, for example, in green when the audio stream is only the visual stream. Also, a stream 68b2 formed by a thin line
Are displayed, for example, in blue in the case of an auditory stream, and in green, for example, in the case of a visual stream. Thus, by looking at the display of the viewer 68, the visual stream and the auditory stream at that time can be visually and intuitively grasped by the radar chart 68a, and the stream chart 68b
Thereby, the temporal flow of the visual stream and the auditory stream can be grasped more intuitively by visual observation. At this time, by visually recognizing the color of the display, it is possible to easily grasp in which stream the attention control is being performed.

The humanoid robot 10 according to the embodiment of the present invention is configured as described above, and operates as follows with reference to FIG. 5 for a target speaker as a party reception robot. First, as shown in FIG. 10A, the robot 10 is disposed in front of the entrance of the party venue. Then, as shown in FIG. 10B, the party participant P approaches the robot 10,
Has not yet recognized the participant P. Here, the participant P has talks with the robot 10, for example, "Hello", the robot 10 includes a microphone 16 picking up the sound of the participant P, generate an auditory event 28 hearing module 20 involves a sound source direction Then, the data is transmitted to the association module 60 via the network 70.

As a result, the association module 60 generates an auditory stream 29 based on the auditory event 28. At this time, the visual module 30 does not generate the visual event 39 because the participant P is not in the field of view of the camera 15. Accordingly, the association module 60 may use only the auditory event 28
The auditory stream 29 is generated, and the attention control module 64 uses the auditory stream 29 as a trigger to perform attention control such that the robot 10 is directed to the participant P.

In this way, as shown in FIG. 10C, the robot 10 faces the participant P, and tracking is performed by a so-called voice. Then, the visual module 30 captures the image of the face of the participant P by the camera 15, generates a visual event 39, searches the face of the participant P by the face database 38, performs face identification, and determines the result. Is transmitted to the association module 60 via the network 70. If the face of the participant P is not registered in the face database 38, the visual module 30 transmits the fact to the association module via the network 70.

At this time, the robot 10 has generated an association stream 65 based on the auditory event 28 and the visual event 39, and the attention control module 64 uses the association stream 65 to generate the attention stream.
Since the attention control is not changed, the robot 10
Keeps pointing in the direction of the participant P. Therefore, even if the participant P moves, the robot 10 tracks the participant P by controlling the motor control module 40 by the association stream 65, and the camera 15 of the visual module 30 continues the participant P. It is possible to take an image.

Then, the association module 60
Gives an input to the speech recognition circuit 55 of the hearing module 20, and the speech recognition circuit 55 gives the speech recognition result to the dialogue control circuit 53. As a result, the dialog control circuit 53 performs voice synthesis and utters voice from the speaker 51. At this time, the voice recognition circuit 55 reduces the sound from the speaker 51 from the sound signal from the microphone 16 by the self-voice suppression circuit 54, so that the robot 10 ignores its own utterance,
The other party's voice can be more accurately recognized.

Here, the utterance by the speech synthesis is based on the participant P
Is registered in the face database 38 or not. If the face of the participant P is registered in the face database 38, the association module 60
Is based on the face ID 24 from the vision module 30,
To control the interaction module 50, to ask questions to participants P as "Hello .XXX-san?" By the speech synthesis. On the other hand, if the participant P answers "Yes.", The dialogue module 50 recognizes "Yes" by the voice recognition circuit 55 based on the acoustic signal from the microphone 16, and the voice is synthesized by the dialogue control circuit 53. And the speaker 5
From 1, he says, "Welcome XXX, please enter the room."

The face of the participant P is stored in the face database 38.
If that is not registered, the association module 60 is to control the interaction module 50, the question for the "Hello. Can you tell me your name?" And participants P by speech synthesis. On the other hand, when the participant P answers his name "XXX.", The dialogue module 50 recognizes "XXX" by the voice recognition circuit 55 based on the acoustic signal from the microphone 16, and the dialogue is performed. Speech synthesis is performed by the control circuit 53, and the speaker 51 utters "Welcome XXX, please enter the room." In this way, the robot 10
Performs recognition of the participant P, guides entry to the party venue, and uses the visual module 30 to
The face image and the name “XXX” are registered in the face database 38.

The humanoid robot 10 operates as a companion robot as follows, for example, with reference to FIG. First, the humanoid robot 10 does not have a particularly clear scenario. For example, in FIG. 13, the humanoid robot 10 tracks one speaker with respect to the four speakers appearing, and tracks other speakers along the way. The operation is performed so as to switch the attention. The operation is performed by each viewer 27, 3
The display at 7, 48, 68 can be easily grasped and evaluated by visually recognizing the display. here,
Each of FIGS. 13A to 13H includes an upper left snapshot, an upper right viewer 68, a lower left viewer 27, and a lower right viewer 28. First, in FIG. 13A, the humanoid robot 10 detects the face of the leftmost speaker as indicated by a rectangular frame on the display of the lower right viewer 28 by the visual module 30. The visual event 38 of the speaker is
In the upper right stream chart 68b, a thin line 6
8b1.

Next, when the speaker starts speaking, FIG.
As shown in (B), the auditory module 20 detects the auditory event 28, and the auditory event 28 is displayed as a small circle in the right window 27b of the lower left viewer 27 and is displayed in the left window 27a. And extracted as a set of overtone peaks in the power spectrum. The auditory event 28 is converted into an auditory stream by the association module 60,
This is shown as a narrow and dark sector 68a2 in the radar chart 68a of the upper right viewer 68.

At this time, the association module 60 is obtained from the auditory event 28 and the visual event 38 of the speaker.
Since the auditory stream 65 and the visual stream 66 generated by the above have a common direction for a certain period of time or more, the stream generator 63 of the association module 60 generates the association stream 67, and FIG. As shown in ()), the association stream is displayed as a thick line on the stream chart 68b of the viewer 68 at the upper right, and attention control is performed on the speaker.

In this state, as shown in FIG. 13D, when the speaker interrupts the talk, the association of the association stream is released, the attention for the speaker is released, and the attention control is interrupted. Is done. Then, since the rightmost person started speaking, attention was directed to this speaker, and the humanoid robot 1
When 0 tries to turn around this speaker,
This speaker stopped talking. At that time, the humanoid robot 10 interrupted the horizontal rotation because the visual module 30 accidentally detected the face of the second speaker from the left.

Subsequently, as shown in FIG. 13E, the second speaker from the left began to speak, and an association stream of the speaker was generated, so that attention was directed to the speaker. Thereafter, as shown in FIG.
The speaker stopped talking, but the visual event 38 of the speaker
Is continuously detected, so that the association stream 67 has existed for several seconds.

Then, as shown in FIG. 13 (G), after the association stream 67 has disappeared, the third speaker from the left has begun to speak, so that the humanoid robot 10 tries to turn to that speaker. Since the visual module 30 cannot detect the speaker, no visual event 38 and no visual stream 66 are generated, and thus no association stream is generated.

Thereafter, as shown in FIG. 13 (H), when the leftmost speaker starts speaking again, the relevant speaker is not detected by the visual module 30, and therefore the visual event 3
8. Since the visual stream 66 has not been generated and the association stream has not been generated, the humanoid robot 10 turns to the speaker using the auditory event 28 of the speaker as a trigger.

In this way, the humanoid robot 10 is able to perform the auditory event 28 by the auditory module 20, the visual event 39 by the visual module 30, and the association stream 6 by the association module 60.
5 and a plurality of speakers can be recognized by hearing and vision, and one of the plurality of speakers can be tracked or switched to another speaker along the way. it can. In the case of a companion robot, the robot 10 plays a passive role, that is, plays only "listen" or "looks at the speaker" of the party participant, and does not utter by the dialog module 50. .

Further, the humanoid robot 10 as a companion robot may share the face database 38 with the party reception robot, or the face database 38 of the party reception robot may be transferred or copied. In this case, the humanoid robot 10 as the companion robot can always recognize all the party participants by face identification.

As described above, according to the humanoid robot 10 according to the embodiment of the present invention, based on the auditory event and the visual event from the auditory module 20 and the visual module 30, the association module 60 performs the auditory stream, the visual stream, and the association. Generating a stream recognizes multiple speakers, so if any of the events are missing or cannot be clearly recognized, for example, the speaker moves and becomes "invisible" In some cases, multiple speakers can be tracked audibly and / or visually in real time, and also visually, even if the speaker does not speak and becomes "inaudible." In addition, since the auditory information, visual information, motor information, and stream information are displayed by the viewer of each module, the information is intuitively grasped visually and the operation status of the humanoid robot 10 is easily evaluated. be able to. At this time, the display colors of the respective viewers differ depending on the status of the association stream, and are unified among the viewers, so that the status of the association stream can be easily grasped.

In the above-described embodiment, the humanoid robot 10 is configured to have 4 DOF (degree of freedom). However, the present invention is not limited thereto, and the robot according to the present invention may be configured to perform any operation. It is also possible to incorporate a robot hearing system. Further, in the above-described embodiment, the case where the robot audiovisual system according to the present invention is incorporated in the humanoid robot 10 has been described. However, the present invention is not limited to this. Obviously, it is also possible to incorporate. Further, in the above-described embodiment, when a plurality of streams of the same type exist during the attention control, the tracking of the oldest stream is prioritized. However, the present invention is not limited to this. For example, the tracking of the newest stream may be prioritized.

[0082]

As described above, according to the present invention,
The cooperation between the hearing module, the vision module, and the motor control module, and the association module and the attention control module complements the ambiguities of the hearing and vision of the robot, thereby improving the so-called robustness. Even speakers can perceive each speaker. Also, for example, even when either the auditory event or the visual event is missing, the association module can perceive the target speaker based on only the visual event or the auditory event. Control of the control module can be performed. Further, the display unit displays at least a part of auditory information by the auditory module, visual information by the visual module, motor information by the motor module, and stream information by the association module,
By visualizing the real-time processing by the association module, the state of the real-time processing can be visually and intuitively grasped. Thus, according to the present invention, an extremely excellent robot audio-visual system that enables real-time processing for performing visual and auditory tracking of an object and visualizes the real-time processing is provided.

[Brief description of the drawings]

FIG. 1 is a front view showing the appearance of a humanoid robot incorporating a first embodiment of a robot hearing device according to the present invention.

FIG. 2 is a side view of the humanoid robot of FIG. 1;

FIG. 3 is a schematic enlarged view showing a configuration of a head in the humanoid robot of FIG. 1;

FIG. 4 is a block diagram showing an electrical configuration of a robot audiovisual system in the humanoid robot of FIG. 1;

5 is a block diagram of an electrical configuration showing an enlarged view of a hearing module of block 1 in FIG. 4;

6 is a block diagram of an electrical configuration showing a visual module of a block 2 in FIG. 4 in an enlarged manner.

FIG. 7 is a block diagram of an electric configuration showing a motor control module of a block 3 in FIG. 4 in an enlarged manner.

FIG. 8 is a block diagram of an electric configuration showing a dialog module of block 4 in FIG. 4 in an enlarged manner.

9 is an enlarged block diagram of an electrical configuration showing an association module of a block 5 in FIG. 4;

FIG. 10 is a diagram showing an operation example as a party reception robot in the robot audiovisual system of FIG. 4;

11 is a diagram showing an example of a screen of a viewer of (A) an auditory module and (B) a viewer of a visual module in the robot audiovisual system of FIG. It is a figure which shows the example of a screen of the viewer of (C) motor control module and (D) association module.

12 is a diagram showing an example of a screen of a viewer of (C) a motor control module and (D) an association module in the robot audiovisual system of FIG. 4;

13 is a diagram illustrating an operation example as a companion robot in the robot audiovisual system of FIG. 4;

[Explanation of symbols]

 Reference Signs List 10 humanoid robot 11 base 12 torso 13 head 13a connecting member 14 exterior 15 camera (robot vision) 16, 16a, 16b microphone (robot hearing) 17 robot audiovisual system 20 hearing module 30 visual module 40 motor control module 50 dialog module 60 Association Module 70 Network

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) G10L 11/04 H04N 7/18 Z 5D045 13/00 13/00 5L096 15/28 G10L 3/00 C 17 / 00 Q 15/00 511 15/20 545F 21/02 551H 15/02 3/02 301C H04N 7/18 9/00 301A 13/00 F term (reference) 3C007 AS34 AS36 CS08 JS03 KS04 KS08 KS18 KS39 KT01 KT11 KT15 LT08 NS01 WA02 WA03 WB19 WC07 5B057 AA05 BA02 CA12 CA16 DA06 DB02 5C054 CC05 CD03 CG02 FC12 FD01 FE16 FF07 HA04 5C061 AA06 AA21 AA25 AB01 AB08 AB12 AB14 AB17 5D015 AA03 DD02 KK01 KK04 5D069 AB05

Claims (10)

[Claims] 1. A hearing module including at least a pair of microphones for collecting external sounds, a visual module including a camera for capturing an image in front of the robot, and a motor control module including a drive motor for rotating the robot in a horizontal direction. An association module that integrates events from the hearing module, the vision module, and the motor control module to generate a stream, and an attention control module that performs attention control based on the stream generated by the association module. A robot audiovisual system, wherein the auditory module identifies a sound source of at least one speaker and extracts its auditory event from pitch extraction, sound source separation and localization based on an acoustic signal from a microphone; Visual module By extracting the visual event from the face identification and localization of each speaker based on the image captured by the camera, the motor control module extracts the motor event based on the rotational position of the drive motor, The association module generates an audio stream and a visual stream and an association stream associating the audio stream and the visual stream from the audio event, the visual event, and the motor event, and the attention control module drives the motor control module based on the streams. While performing attention control for planning of motor control, the auditory information by the auditory module, the visual information by the visual module and the motor information by the motor module,
A robot audiovisual system, comprising: a display unit that displays at least a part of the stream information by the association module. 2. A hearing module including at least a pair of microphones for collecting external sounds, a visual module including a camera for capturing an image in front of the robot, and a motor control module including a driving motor for rotating the robot in a horizontal direction. An association module that integrates events from the hearing module, the vision module, and the motor control module to generate a stream, and an attention control module that performs attention control based on the stream generated by the association module. An audiovisual system for a humanoid or animal robot, wherein the auditory module identifies a sound source of at least one speaker based on a sound signal from a microphone from pitch extraction, sound source separation and localization. Extract auditory events The visual module extracts a visual event from face identification and localization of each speaker based on an image captured by a camera, and the motor control module extracts a motor event based on a rotational position of a driving motor. Accordingly, the association module generates an audio stream, a visual stream, and an association stream that associates the audio stream and the visual stream from the audio event, the visual event, and the motor event, and the attention control module performs motor control based on the streams. Performing attention control for planning of the drive motor control of the module, hearing information by the hearing module, visual information by the visual module, and motor information by the motor module;
A robot audiovisual system, comprising: a display unit that displays at least a part of the stream information by the association module. 3. The display unit according to claim 1, wherein the display unit includes an auditory display unit that displays, as auditory information, a spectrum of an acoustic signal from a sound source, an extracted peak, and an auditory event. 3. The robot audiovisual system according to 2. 4. The auditory display unit displays an auditory event as a circle having a vertical axis representing a relative azimuth around a robot, a horizontal axis representing a pitch, and a circle having a diameter as a certainty factor. The robot audiovisual system according to claim 3. 5. The display device according to claim 1, wherein the display unit includes a camera image indicating the extracted face as a frame as visual information and a visual display unit displaying a visual event. A robot audio-visual system according to the present invention. 6. The robot audiovisual system according to claim 5, wherein the visual display unit displays a visual event by using a list of face identification and face localization extracted with certainty. 7. The display device according to claim 1, wherein the display unit includes a motor display unit that three-dimensionally displays a direction and an operation speed of the robot in real time as motor information. The described robot audiovisual system. 8. The display unit, as stream information,
8. A stream display unit for displaying a stream chart and a radar chart.
Robot audiovisual system according to 4. 9. The robot audiovisual system according to claim 8, wherein the stream display unit displays stream information in the form of a stream chart, each of an auditory stream, a visual stream, and an association stream. 10. The robot audio-visual system according to claim 8, wherein the stream display unit displays a stream state at that time on a radar chart based on a camera visual field and a sound source localization.