JP6881267B2 - Controls, converters, control methods, conversion methods, and programs - Google Patents

Description

The present invention relates to a technique for estimating the direction of a speaker.

In applications such as voice dialogue agents and robot dialogues that use voice recognition, it is important to determine whether or not the sound that arrives at the robot is related to the dialogue in order to realize a smooth dialogue.

For example, in the prior art described in Patent Document 1 and Patent Document 2, a signal separated for each of a plurality of directions is generated based on a signal collected by a plurality of microphones, and the power of the separated signal is calculated. The direction of maximum power at a certain point is set as the direction related to the dialogue, and directional sound collection is performed so as to emphasize and collect the sound in that direction.

FIG. 1 shows the functional configuration of the conventional speaker direction determination device. The speaker direction determination device 9 of FIG. 1 includes a direction-specific preprocessing unit 91, a direction-specific power calculation unit 92, and an arrival direction selection unit 93. The direction-specific preprocessing unit 91 generates signals separated for each of a plurality of directions based on sound signals collected by a plurality of microphones. The direction-specific power calculation unit 92 calculates the power for each direction from the sound signal after separation. The arrival direction selection unit 93 selects the direction that becomes the maximum power at a certain time from the power for each direction as the direction related to the dialogue. The directional sound collecting unit 8 emphasizes and collects the sound in the arrival direction selected by the arrival direction selection unit 92 among the sound signals collected by the plurality of microphones.

Japanese Unexamined Patent Publication No. 2005-64968 Japanese Unexamined Patent Publication No. 2001-309483

However, since the conventional technology uses only the power of sound as a clue, when there is a target sound source and a sound source unrelated to dialogue, it is not possible to distinguish which is the target sound source, and the unrelated sound source. There is a possibility of malfunction such as emphasizing the side. For example, assuming a scene in which a robot surrounded by a plurality of people has a dialogue, a malfunction such as reacting to a speaker unrelated to the dialogue like a surrounding conversation may occur, and the dialogue may not be established.

In view of the above points, an object of the present invention is a speaker direction determination technique capable of preventing a malfunction by eliminating information irrelevant to the dialogue when a sound source irrelevant to the dialogue exists. Is to realize.

In order to solve the above problems, the converter of the present invention uses a camera to capture the direction of the desired sound source based on the microphone array estimated from the acoustic signals picked up by the plurality of microphones. It is a conversion device that converts the microphone coordinates into camera coordinates, which are the coordinates in the image, and includes a conversion unit that converts the microphone direction into camera coordinates according to the conversion rule. The conversion rule is emitted from a sound emitting unit consisting of at least three speakers. An acoustic signal set so that it can be identified from which of at least three speakers is emitted, and an image in which the sound emitting part is taken so that the individual positions of at least three speakers can be detected. It was obtained by associating.

In the speaker direction determination technique of the present invention, information irrelevant to dialogue can be eliminated by estimating the speaker direction using image recognition in addition to estimating the arrival direction of sound. Thereby, according to the speaker direction determination technique of the present invention, it is possible to prevent a malfunction even when a sound source unrelated to the dialogue exists.

FIG. 1 is a diagram illustrating a functional configuration of a conventional speaker direction determining device. FIG. 2 is a diagram illustrating a functional configuration of the speaker direction determining device of the first embodiment. FIG. 3 is a diagram illustrating a processing procedure of the speaker direction determination method of the first embodiment. FIG. 4 is a diagram for explaining the speaker direction estimation result and the calibration of the camera image. FIG. 5 is a diagram for explaining the calculation of the score using the image recognition result. FIG. 6 is a diagram illustrating the functional configuration of the conversion device of the modified example. FIG. 7 is a diagram illustrating a processing procedure of a conversion method of a modified example. FIG. 8 is a diagram illustrating the functional configuration of the speaker direction determining device of the second embodiment. FIG. 9 is a diagram illustrating a processing procedure of the speaker direction determination method of the second embodiment. FIG. 10 is a diagram for explaining the calculation of the score using the voice recognition result and the image recognition result. FIG. 11 is a diagram illustrating a functional configuration of the optimum arrangement acquisition device of the third embodiment. FIG. 12 is a diagram for explaining a method of acquiring the optimum arrangement of the speakers. FIG. 13 is a diagram for explaining a display method of the optimum arrangement of the speakers.

Hereinafter, embodiments of the present invention will be described in detail. In the drawings, the components having the same function are given the same number, and duplicate description will be omitted.

[First Embodiment]
The speaker direction determination device of the first embodiment is an arrival direction estimation result in order to prevent the direction estimation from malfunctioning due to a noise source or the like when the dialogue robot or the like estimates the speaker direction and performs directional sound collection. It is a device that determines the target speaker direction by performing image recognition based on the above.

As shown in FIG. 2, the speaker direction determination device 1 of the first embodiment is captured by M audio signals collected by M (≧ 2) microphones and at least K (≧ 1) cameras. K image signals are input, and the speaker direction estimated from the audio signal and the image signal is output to the directional sound collecting unit 8. It is assumed that the K cameras and the M microphones are installed at different positions, but they may be installed at positions that can be regarded as the same, for example, by installing the microphones in the housing of the camera. As the K cameras, a camera capable of photographing the entire circumference around the camera, such as an omnidirectional camera, may be used. The speaker direction determination device 1 includes an arrival direction estimation unit 11, a measurement coordinate correction unit 12, an image recognition unit 13, and a speaker direction estimation unit 14. The speaker direction determination method of the first embodiment is realized by the speaker direction determination device 1 performing the processing of each step illustrated in FIG.

The speaker direction determination device 1 is configured by loading a special program into a known or dedicated computer having, for example, a central processing unit (CPU), a main storage device (RAM: Random Access Memory), or the like. It is a special device. The speaker direction determination device 1 executes each process under the control of the central processing unit, for example. The data input to the speaker direction determination device 1 and the data obtained by each process are stored in, for example, the main storage device, and the data stored in the main storage device is read out to the central processing unit as needed. It is used for other processing. At least a part of each processing unit of the speaker direction determining device 1 may be configured by hardware such as an integrated circuit.

Hereinafter, the speaker direction determination method executed by the speaker direction determination device 1 of the first embodiment will be described with reference to FIG.

In step S11, the arrival direction estimation unit 11 first receives M audio signals from M microphones and converts them into L (≧ 2) direction-specific audio signals by signal processing such as beamforming. , Calculate the power for each direction from the audio signal for each direction. Next, the arrival direction estimation unit 11 compares the calculated power for each direction with a predetermined threshold value, and estimates the direction exceeding the threshold value as the arrival direction. Then, the arrival direction estimation unit 11 outputs the estimation result of the arrival direction to the measurement coordinate correction unit 12.

In step S12, the measurement coordinate correction unit 12 receives the arrival direction estimation result (angle information) from the arrival direction estimation unit 11 and calculates in advance so as to match the coordinate system on the images taken by the K cameras. Using the transformation matrix that has been set up, the arrival direction estimation result is calibrated to the same coordinate system as the camera. The measurement coordinate correction unit 12 outputs the calibrated arrival direction estimation result to the image recognition unit 13.

With reference to FIG. 4, the arrival direction estimation result (angle information) and the calibration method of the camera image will be described. In order to calibrate, the points (θ, φ) on the two-dimensional angle spectrum of the arrival direction estimation result calculated from the voice signal observed by the microphone (θ represents the horizontal angle and φ represents the elevation angle) and the camera. The conversion matrix with the pixels (x, y) on the captured image may be obtained. Here, a conversion method from a pixel of a camera image to a point on a two-dimensional angular spectrum is shown. When converting from a point on the two-dimensional angle spectrum to a pixel of a camera image, the reverse calculation may be performed.

As shown in FIG. 4, the calibration speakers are installed at three or more different positions. A marker (for example, a symbol such as “●”, “■”, “★”, etc.) that can distinguish each calibration speaker from the captured image is attached to each speaker. In addition, by emitting sounds in different frequency bands from each calibration speaker, it is possible to make a correspondence between the pixels on the image and the two-dimensional angle spectrum. _{Two-dimensional angle spectrum (θ i} , φ _i ) and camera image pixels (x _i , y _i ) (i are speakers) obtained by simultaneously collecting and photographing sound with a microphone and a camera using this calibration speaker. For (representing the index of), the transformation matrix represented by the following equation is obtained. Here, a, b, c, d, e, and f are conversion parameters obtained from the pair of pixels on the image and the two-dimensional angle spectrum of the arrival direction estimation result. The 3 × 3 matrix in which this transformation parameter is set corresponds to the transformation matrix K in FIG.

The degrees of freedom in coordinate conversion are a total of 6 degrees of freedom: rotation (1 degree of freedom), translation (2 degrees of freedom), scaling (1 degree of freedom), and shear (1 degree of freedom). By obtaining three or more pairs, the conversion matrix can be uniquely determined.

In step S13, the image recognition unit 13 receives the image signals from the K cameras, and receives the arrival direction estimation result whose coordinate axes are calibrated from the measurement coordinate correction unit 12. The image recognition unit 13 acquires an image for each arrival direction from the image received from the camera, performs face recognition on the acquired image, detects the face orientation in the image, and occupies the face portion in the entire screen. Is calculated. The image recognition unit 13 outputs the screen occupancy rate of the face portion and the face orientation detection result to the speaker direction estimation unit 19. As a method for detecting the face orientation and a method for calculating the screen occupancy rate, a method as in Reference 1 below can be used.
[Reference 1] Hiroyuki Arai, Naoki Ito, Kaori Kataoka, Yukinobu Taniguchi, "Advertising Effect Measurement Technology by Image Processing-Application of Number Measurement Technology / Face Image Technology", NTT Technology Journal 2013.1, vol. 25, pp. 61 -64, 2013

In step S14, the speaker direction estimation unit 14 estimates the speaker direction, which is the target of the directional sound collection, from the screen occupancy rate and the face orientation detection result for each arrival direction received from the image recognition unit 13. The speaker direction estimation method may be deterministic or probabilistic. For example, as shown in FIG. 5, a score is calculated from the screen occupancy rate and the face orientation detection result, and the direction of the image having the highest score is determined as the speaker direction. For example, in the example of FIG. 5, FIG. 5 (A), which faces the front and has a high screen occupancy rate, has the highest score, and FIG. 5 (D), which faces other than the front and has a low screen occupancy rate, has the lowest score. You can see that it is. The speaker direction estimation unit 14 outputs the determined speaker direction to the directional sound collecting unit 8.

In step S8, the directional sound collecting unit 8 emphasizes the sound in the speaker direction received from the speaker direction estimating unit 14 from the M audio signals collected by the M microphones and collects the sound. The directional sound collecting unit 8 collects the directional sound described in Reference 2 below, for example. The directional sound collecting unit 8 outputs the emphasized voice in which the sound in the speaker direction is emphasized.
[Reference 2] Japanese Unexamined Patent Publication No. 2009-445888

[Modification example]
An independent conversion device may be configured in which only the processing of the measurement coordinate correction unit 12 is extracted from the speaker direction determination device 1 of the first embodiment. As shown in FIG. 6, the conversion device 100 of the modified example includes a conversion unit 10. The conversion device 100 realizes the conversion method of the modified example by performing the processing of each step illustrated in FIG. 7.

The conversion device 100 takes the microphone direction as an input, and outputs the camera coordinates obtained by converting the microphone direction into the coordinate system on the image taken by the camera. The microphone direction is a direction of a desired sound source with reference to a microphone array estimated from acoustic signals picked up by a plurality of microphones. The camera coordinates are the coordinates in the image taken by the camera.

In step S10, the conversion unit 10 calibrates the input microphone direction (angle information) to the coordinate system of the image captured by the camera using the same conversion matrix as in the first embodiment, and converts the camera coordinates. It is output as the output of the device 100. Since the transformation matrix is the same as that of the first embodiment, it is set so that it can be identified from which of the at least three speakers the sound emitting unit is emitted from the sound emitting unit consisting of at least three speakers. It was obtained by associating the acoustic signal with an image in which the sound emitting portion was taken so that the individual positions of at least three speakers could be detected.

The conversion device 100 of the modified example receives, for example, the arrival direction estimation result output by the arrival direction estimation unit 11 of the speaker direction determination device 1, and calibrates the arrival direction estimation result to the coordinates on the image taken by the camera. , It is possible to function as an external device to be returned to the image recognition unit 13 of the speaker direction determination device 1. Further, it can be applied to other voice processing devices that convert the arrival direction of the sound collected by the microphone into the coordinates on the image taken by the camera.

[Second Embodiment]
As shown in FIG. 8, the speaker direction determination device 2 of the second embodiment has an arrival direction estimation unit 11, a measurement coordinate correction unit 12, an image recognition unit 13, and a speaker direction estimation unit 14, as in the first embodiment. And further, a voice recognition unit 21 is provided. The speaker direction determination method of the second embodiment is realized by the speaker direction determination device 2 performing the processing of each step illustrated in FIG. 9.

Hereinafter, the speaker direction determination method executed by the speaker direction determination device 2 of the second embodiment will be described with reference to FIG. 9.

In step S11, the arrival direction estimation unit 11 estimates the arrival direction and outputs the measurement coordinate correction unit 12 as in the first embodiment. At the same time, the M voice signals received from the M microphones are separated for each direction of arrival and output to the voice recognition unit 21.

In step S12, the measurement coordinate correction unit 12 calibrates the arrival direction estimation result received from the arrival direction estimation unit 11 into the same coordinate system as the camera, and outputs the result to the image recognition unit 13, as in the first embodiment. .. The transformation matrix used for calibration can be obtained by the same method as in the first embodiment.

In step S13, the image recognition unit 13 detects the face orientation and calculates the screen occupancy for each arrival direction received from the measurement coordinate correction unit 12 as in the first embodiment, and the face orientation detection result and the screen. The occupancy rate is output to the speaker direction estimation unit 19. As a face orientation detection method and a screen occupancy calculation method, the same method as in Reference 1 can be used as in the first embodiment.

In step S21, the voice recognition unit 21 performs voice recognition on the voice signals separated for each arrival direction received from the arrival direction estimation unit 11, and obtains the voice recognition result for each arrival direction. The voice recognition unit 21 outputs the obtained voice recognition result to the speaker direction estimation unit 14.

In step S14, the speaker direction estimation unit 14 is based on the voice recognition result for each arrival direction received from the voice recognition unit 21, the face orientation detection result for each arrival direction received from the image recognition unit 13, and the screen occupancy rate. To estimate the speaker direction, which is the target of directional sound collection. For example, as shown in FIG. 10 (A), the screen occupancy rate is high, the face is facing the front, and the score in the arrival direction in which a specific word is spoken is high, as shown in FIG. 10 (B). , It is conceivable to design so that the score in the direction of arrival that does not meet those conditions is low. At this time, the specific word is a word that is expected to be uttered by the speaker from a dialogue scenario or a speech recognition task, and is designed so that the speech recognition result is given a high score so that these words are included. It is good to do it. The speaker direction estimation unit 14 outputs the determined speaker direction to the directional sound collecting unit 8.

The main points of the present invention are 1. 2. Determine the speaker direction using image information and linguistic information based on the arrival direction estimation result using sound information. The image information uses the screen occupancy rate and the face orientation detection result by face recognition, and the language information uses the utterance detection result of a specific word. Based on the above points, the target speaker direction, which cannot be captured by sound alone, is erroneously estimated by the conventional direction estimation by performing speaker direction estimation using image information and linguistic information after the arrival direction estimation by sound. The situation that was detected can be avoided, and the robustness of the speaker direction estimation result is improved. When using a dialogue robot that uses voice recognition, even in an environment where there are noise sources such as surrounding speakers, only the utterances of the speaker to be spoken can be detected, so the usage scene can be expanded and the user can use it. Convenience is improved.

[Third Embodiment]
In the above embodiment, the arrival direction of the sound is converted into the coordinates of the camera by using the transformation matrix K prepared in advance. In the third embodiment, an optimum arrangement acquisition device for obtaining the optimum arrangement of the calibration speaker for acquiring the transformation matrix K will be described. By optimizing the arrangement of the calibration speaker, there is an effect that the influence of lens distortion, aberration, crossing deviation, etc. can be generally reduced on the camera side. For the microphone side, it is desirable to install the microphones at various positions and angles as much as possible in order to reduce the influence of the sensitivity error of each microphone. Therefore, such an arrangement is required as the optimum arrangement.

As shown in FIG. 11, the optimum arrangement acquisition device 3 of the third embodiment includes M audio signals collected by M (≧ 2) microphones and K captured by at least K (≧ 1) cameras. The optimum arrangement of the calibration speaker calculated from the audio signal and the image signal is displayed on the screen by inputting the individual image signals. The optimum arrangement acquisition device 3 includes a first angle difference acquisition unit 31, a second angle difference acquisition unit 32, a distance acquisition unit 33, an optimum arrangement calculation unit 34, and an optimum arrangement display unit 35. The optimum arrangement acquisition device 3 of the third embodiment is realized by performing the processing of each step described later by the optimum arrangement acquisition device 3.

The first angle difference acquisition unit 31 obtains an angle difference when each calibration speaker is viewed from a microphone array composed of M microphones, based on M audio signals collected by M microphones. The first angle difference acquisition unit 31 outputs the obtained angle difference of the calibration speaker to the optimum arrangement calculation unit 34. If the angle difference when each calibration speaker is viewed from the microphone array is known, it is not necessary to include the first angle difference acquisition unit 31, and the optimum arrangement acquisition device 3 is configured to input the known angle difference. do it.

The second angle difference acquisition unit 32 obtains the angle difference when each calibration speaker is viewed from the cameras based on the K image signals captured by the K cameras. The second angle difference acquisition unit 32 outputs the obtained angle difference of the calibration speaker to the optimum arrangement calculation unit 34. If the angle difference when each calibration speaker is viewed from the camera is known, it is not necessary to provide the second angle difference acquisition unit 32, and the optimum arrangement acquisition device 3 is configured to input the known angle difference. Just do it.

The distance acquisition unit 33 obtains the distance between the calibration speakers based on the K image signals captured by the K cameras. The distance acquisition unit 33 outputs the obtained distance between the calibration speakers to the optimum placement calculation unit 34. If the distance between the calibration speakers is known, it is not necessary to include the distance acquisition unit 33, and the optimum placement acquisition device 3 may be configured to input the known distance.

The optimum placement calculation unit 34 is based on the angle difference when each calibration speaker is viewed from the microphone array, the angle difference when each calibration speaker is viewed from the camera, and the distance between the calibration speakers. Calculate the optimal placement of. The optimum placement calculation unit 34 outputs the calculated optimum placement of the calibration speaker to the optimum placement display unit 35.

A method of calculating the optimum arrangement of the calibration speaker by the optimum arrangement calculation unit 34 will be described with reference to FIG. In the example of FIG. 12, there are three calibration speakers, and the optimum arrangement of the calibration speakers is calculated with reference to a microphone array composed of three microphones and one camera. In the figure, the distances between the calibration speakers are indicated by A-1 to A-3. The angle difference when each calibration speaker is viewed from the camera is shown by B-1 to B-3. The angle difference when each calibration speaker is viewed from the microphone array is shown by C-1 to C-3. At this time, the distances A-1 to A-3 between the calibration speakers, the angle difference B-1 to B-3 between the camera and the calibration speaker, and the angle difference C-1 to C-3 between the microphone array and the calibration speaker. By maximizing and, the optimum arrangement of the calibration speaker can be obtained. The angle difference is arg (B-1) -arg (B-2), for example, assuming that the angle difference between B-1 and B-2 is defined by B-1 and B-2 as vectors. Is.

The optimum arrangement display unit 35 outputs the optimum arrangement of the calibration speaker received from the optimum arrangement calculation unit 34 to an output unit (not shown) such as a screen. FIG. 13 is an example in which the optimum arrangement display unit 35 displays the optimum arrangement of each calibration speaker on the screen. FIG. 13 shows the positions of the calibration speakers actually installed (solid circles) on the image of the space in which the calibration speakers are installed from the camera, and each calibration calculated by the optimum placement calculation unit 34. This is an example of a screen displaying the optimum position of the speaker (dotted shaded circle). The actual position of the proofreading speaker and the optimum position of each proofreading speaker can be determined by plotting each position on the screen, for example, with the left and right as the x-axis, the top and bottom as the y-axis, and the depth as the z-axis. indicate. Here, an example in the case of using a Cartesian coordinate system is shown, but it may be displayed using an appropriate coordinate system for the space in which the calibration speaker is arranged, such as a cylindrical coordinate system or a spherical coordinate system.

Although the embodiments of the present invention have been described above, the specific configuration is not limited to these embodiments, and even if the design is appropriately changed without departing from the spirit of the present invention, the specific configuration is not limited to these embodiments. Needless to say, it is included in the present invention. The various processes described in the embodiments are not only executed in chronological order according to the order described, but may also be executed in parallel or individually as required by the processing capacity of the device that executes the processes.

[Program, recording medium]
When various processing functions in each device described in the above embodiment are realized by a computer, the processing contents of the functions that each device should have are described by a program. Then, by executing this program on the computer, various processing functions in each of the above devices are realized on the computer.

The program describing the processing content can be recorded on a computer-readable recording medium. The computer-readable recording medium may be, for example, a magnetic recording device, an optical disk, a photomagnetic recording medium, a semiconductor memory, or the like.

In addition, the distribution of this program is carried out, for example, by selling, transferring, renting, or the like a portable recording medium such as a DVD or CD-ROM on which the program is recorded. Further, the program may be stored in the storage device of the server computer, and the program may be distributed by transferring the program from the server computer to another computer via a network.

A computer that executes such a program first, for example, first stores a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. Then, when the process is executed, the computer reads the program stored in its own storage device and executes the process according to the read program. Further, as another execution form of this program, a computer may read the program directly from a portable recording medium and execute processing according to the program, and further, the program is transferred from the server computer to this computer. Each time, the processing according to the received program may be executed sequentially. In addition, the above processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition without transferring the program from the server computer to this computer. May be. The program in this embodiment includes information to be used for processing by a computer and equivalent to the program (data that is not a direct command to the computer but has a property of defining the processing of the computer, etc.).

Further, in this embodiment, the present device is configured by executing a predetermined program on the computer, but at least a part of these processing contents may be realized by hardware.

1, 2, 9 Speaker direction determination device 3 Optimal placement acquisition device 8 Directional sound collection unit 11 Arrival direction estimation unit 12 Measurement coordinate correction unit 13 Image recognition unit 14 Speaker direction estimation unit 21 Voice recognition unit 31 First angle difference Acquisition unit 32 Second angle difference acquisition unit 33 Distance acquisition unit 34 Optimal placement calculation unit 35 Optimal placement display unit 91 Direction-specific preprocessing unit 92 Direction-specific power calculation unit 93 Arrival direction selection unit

Claims (10)

A control device that controls a robot surrounded by a plurality of people to react only to a speaker who is trying to interact with the robot.
The microphone direction, which is the direction of a desired sound source with reference to the microphone array estimated from the acoustic signal picked up by the microphone array composed of a plurality of microphones, is the coordinates in the image taken by the camera according to the conversion rule. A conversion unit that converts to camera coordinates,
By recognizing the image, a directional sound collecting unit that emphasizes the direction of the image having the highest score obtained based on the screen occupancy and the orientation of the face and collects the sound.
Have,
The conversion rule is
An acoustic signal that is emitted from a sound emitting unit consisting of at least three speakers and is set so as to be able to identify which of the at least three speakers is emitted.
An image in which the sound emitting portion is taken so that the individual positions of the at least three speakers can be detected, and
It was obtained by associating
Control device. The control device according to claim 1.
The at least three speakers are used to convert the microphone direction to camera coordinates.
Control device. The control device according to claim 1 or 2.
The conversion rule is obtained by associating the two-dimensional angle spectrum of the arrival direction estimated from the acoustic signal emitted by the speaker with the coordinates on the image of the speaker taken by the camera.
Control device. The control device according to any one of claims 1 to 3.
i is the number of the speaker, (θ _i , φ _i ) is a point on the two-dimensional angle spectrum of the arrival direction estimated from the acoustic signal emitted by the i-th speaker, and (x _i , y _i ) is. The coordinates on the image of the i-th speaker taken by the camera, and a, b, c, d, e, and f are from the set of the point on the two-dimensional angle spectrum and the coordinates with 6 degrees of freedom. Use the obtained conversion parameters as
The conversion unit converts the direction of the microphone into the coordinates of the camera by calculating the following equation.

Control device. Converts the microphone direction, which is the direction of the desired sound source based on the microphone array estimated from the acoustic signal picked up by the microphone array consisting of a plurality of microphones, into the camera coordinates, which are the coordinates in the image taken by the camera. It is a conversion device that
A conversion unit that converts the microphone direction to the camera coordinates according to the conversion rule,
With the optimum placement calculation unit that obtains the distance between at least three speakers, the angle difference of each speaker as seen from the camera, and the position of each speaker that maximizes the angle difference of each speaker as seen from the microphone array as the optimum placement. ,
Including
The conversion rule is
The emitted consisting sound emitting portion from at least three loudspeakers, the acoustic signal is set to be able to identify or emitted from any speaker of the at least three loudspeakers,
An image in which the sound emitting portion is taken so that the individual positions of the at least three speakers can be detected, and
It was obtained by associating
Conversion device. The conversion device according to claim 5.
It further includes an optimum arrangement display unit that superimposes and displays the actual arrangement of each speaker and the optimum arrangement of each speaker on the image taken by the camera.
Conversion device. A control method executed by a control device that controls a robot surrounded by a plurality of people to react only to a speaker who is trying to interact with the robot.
An image taken by a camera according to a conversion rule, in which a conversion unit determines the direction of a desired sound source based on the microphone array estimated from an acoustic signal picked up by a microphone array composed of a plurality of microphones. Converted to camera coordinates, which are the coordinates in
By recognizing the image, the directional sound collecting unit emphasizes the direction of the image having the highest score obtained based on the screen occupancy and the orientation of the face, and collects the sound.
The conversion rule is
An acoustic signal that is emitted from a sound emitting unit consisting of at least three speakers and is set so as to be able to identify which of the at least three speakers is emitted.
An image in which the sound emitting portion is taken so that the individual positions of the at least three speakers can be detected, and
It was obtained by associating
Control method. Converts the microphone direction, which is the direction of the desired sound source based on the microphone array estimated from the acoustic signal picked up by the microphone array consisting of a plurality of microphones, into the camera coordinates, which are the coordinates in the image taken by the camera. This is the conversion method performed by the conversion device.
The conversion unit converts the microphone direction into the camera coordinates according to the conversion rule.
The optimum placement calculation unit sets the distance between at least three speakers, the angle difference of each speaker as seen from the camera, and the position of each speaker that maximizes the angle difference of each speaker as seen from the microphone array as the optimum placement. Ask,
The conversion rule is
The emitted consisting sound emitting portion from at least three loudspeakers, the acoustic signal is set to be able to identify or emitted from any speaker of the at least three loudspeakers,
An image in which the sound emitting portion is taken so that the individual positions of the at least three speakers can be detected, and
It was obtained by associating
Conversion method. A program for operating a computer as the control device according to any one of claims 1 to 4. A program for operating a computer as the conversion device according to claim 5 or 6.

Images