JP2023053670A - Information processing device, information processing method, and program - Google Patents

Abstract

To provide an information processing device, an information processing method, and a program, capable of collecting sound emitted by a sound source in high quality.SOLUTION: A information processing device includes an information acquisition unit and a sound collection control unit. The information acquisition unit acquires sound source information that indicates a position of a sound source and a direction in which the sound source emits sound. From multiple sound collection devices arranged around the sound source and having a configurable sound collection direction, the sound collection control unit selects, on the basis of the sound source information, at least one target device to be used to collect sound emitted by the sound source.SELECTED DRAWING: Figure 4

Description

The present technology relates to an information processing device, an information processing method, and a program applicable to a sound collection system or the like.

In recent years, techniques for separating sound sources and collecting sounds have been developed. For example, by selectively collecting sounds emitted from a specific direction, a desired sound can be separated from various sounds. As a method of collecting sound by designating a direction, for example, beamforming technology is known, which processes outputs of a plurality of microphones arranged in an array to separate a sound source in a specific direction.

Patent Literature 1 describes a speech recognition system using beamforming technology. In this system, the human body is detected from the image taken around the array microphone. The direction in which the human body exists as viewed from the array microphone is set as the sound collection direction, and the direction in which the human body does not exist is set as the noise direction. A beamforming process is also performed to separate the sound source in the sound collecting direction (target sound) and the sound source in the noise direction (noise sound) from the output of the array microphone. By canceling the noise sound from the target sound, highly accurate noise cancellation is possible (paragraphs [0017] [0018] [0023] [0024] FIG. 3 of Patent Document 1, etc.).

JP 2020-3724 A

Even if the noise sound can be canceled from the target sound as in Patent Document 1, the desired sound quality may not be obtained depending on the direction in which the target sound is emitted. Therefore, there is a demand for a technique for collecting the target sound itself with higher quality.

In view of the circumstances as described above, an object of the present technology is to provide an information processing device, an information processing method, and a program capable of collecting sound emitted by a sound source with high quality.

To achieve the above object, an information processing apparatus according to an aspect of the present technology includes an information acquisition unit and a sound collection control unit.
The information acquisition unit acquires sound source information indicating a position of a sound source and a direction in which the sound source emits sound.
Based on the sound source information, the sound collection control unit selects at least one target device that is used to collect sound emitted by the sound source from among a plurality of sound collection devices that are arranged around the sound source and whose sound collection direction can be set. to select.

In this information processing device, at least one target device for collecting the sound of the sound source is selected from a plurality of sound collectors arranged around the sound source. Each sound collecting device is a device capable of setting a sound collecting direction, and sound source information indicating the position of the sound source and the direction in which the sound source emits sound is used for selecting the target device. As a result, it becomes possible to use a sound collector adapted to, for example, the position of the sound source and the direction from which the sound is emitted, and it is possible to collect the sound emitted by the sound source with high quality.

The sound collection control unit may set a sound collection direction of the target device based on the sound source information.

The sound collection control unit may set a direction from the target device toward the sound source as a sound collection direction of the target device.

The sound collection control unit may determine the sound collection device capable of collecting the direct sound emitted by the sound source based on the direction in which the sound source emits sound, and select the sound collection device as the target device. .

The plurality of sound collecting devices may be configured such that the sound collecting direction can be set within an allocation range allocated according to each arrangement. In this case, the sound collection control unit may select, as the target device, the sound collection device whose direction in which the sound source emits sound corresponds to the central direction of the allocation range.

The sound collection control unit is capable of collecting sound along the direction in which the sound source emits sound when there is no sound collection device in which the direction in which the sound source emits sound corresponds to the center direction of the allocation range, The sound collecting device closest to the sound source may be selected as the target device.

The information acquisition unit may acquire the sound source information for each of a plurality of sound sources. In this case, the sound collection control unit may select the target device for each of the plurality of sound sources based on the sound source information for each of the plurality of sound sources.

The sound collection control unit is capable of setting the sound collection direction so as to collect direct sound emitted by a sound source to be processed and not to collect direct sound emitted by a sound source different from the sound source to be processed. may be selected as the target device.

The information processing device may further include a sound collection processing unit that generates sound data representing the sound emitted by the sound source based on the output of the at least one target device.

The plurality of sound collecting devices may include a plurality of candidate devices whose sound collecting directions are set in advance. In this case, the sound collection control unit may select the target device from the plurality of candidate devices. Further, the sound collection processing unit may make the candidate device that is not selected as the target device stand by in a sound collection state.

The sound collection control unit may select a plurality of target devices from the plurality of sound collection devices for the single sound source.

The sound collection processing unit may generate the sound data of the sound source by synthesizing data collected by the plurality of target devices.

The sound source may be a speaker. In this case, the direction in which the sound source emits sound may be the utterance direction of the speaker.

The information acquisition unit may estimate the speech direction of the speaker by performing bone detection on the speaker based on image data of the speaker.

The information acquisition unit may detect a gesture of the speaker.
The sound collection processing unit may control sound collection processing for collecting the voice of the speaker according to the gesture of the speaker.

The sound collection processing unit preferentially executes the sound collection processing for the speaker when a gesture of the speaker raising a hand is detected, and a gesture of the speaker covering the mouth with a hand is detected. case, the sound collection process for the speaker may be stopped.

The sound collection processing unit may separate the speech of the speaker and the gesture sound of the speaker from the data collected by the target device.

The sound collecting device may be a microphone array in which a plurality of microphones are arranged. In this case, the sound collection direction may be a beam direction set by beamforming processing for the microphone array.

An information processing method according to an embodiment of the present technology is an information processing method executed by a computer system, and includes acquiring sound source information indicating a position of a sound source and a direction in which the sound source emits sound.
Based on the sound source information, at least one target device used to collect the sound emitted by the sound source is selected from a plurality of sound collectors arranged around the sound source and capable of setting a sound collection direction.

A program according to an embodiment of the present technology causes a computer system to execute the following steps.
Obtaining sound source information indicating the position of a sound source and the direction in which the sound source emits sound.
Based on the sound source information, selecting at least one target device to be used for collecting the sound emitted by the sound source from a plurality of sound collecting devices arranged around the sound source and capable of setting a sound collection direction.

1 is a block diagram showing a configuration example of a sound collection system according to an embodiment of the present technology; FIG. It is a schematic diagram which shows the structural example of BF microphone. FIG. 4 is a schematic diagram showing an example of beams set in a BF microphone; FIG. 4 is a schematic diagram showing a basic sound collection operation of the sound collection system; 4 is a flowchart showing an operation example of the sound collection system; FIG. 4 is a schematic diagram showing an example of arrangement of BF microphones; FIG. 4 is a schematic diagram showing an example of a speaking direction of a speaker; FIG. 4 is a schematic diagram for explaining a sound collecting operation for a plurality of speakers; FIG. 4 is a schematic diagram showing an example of sound collection operation using a plurality of BF microphones; FIG. 4 is a schematic diagram showing an example of sound collection operation when a speaker moves. FIG. 4 is a schematic diagram for explaining a speech synthesizing process; FIG. 5 is a schematic diagram showing an example of sound collection operations when a plurality of speakers move; FIG. 10 is a schematic diagram showing an example of a sound collection operation assuming an utterance direction of a speaker; FIG. 10 is a schematic diagram showing an example of a sound collection operation in response to a gesture; FIG. 5 is a schematic diagram showing an example of a sound collection operation for collecting voice and operation sound;

Hereinafter, embodiments according to the present technology will be described with reference to the drawings.

[Configuration of sound collection system]
FIG. 1 is a block diagram showing a configuration example of a sound collection system according to an embodiment of the present technology. The sound collection system 100 is a system that collects the voice 5 of the speaker 1 in a space to be sound-collected and generates the voice data 6 of the speaker 1 . In this embodiment, speaker 1 is an example of a sound source, and speech 5 of speaker 1 is a sound to be collected (target sound).
As shown in FIG. 1 , the sound collection system 100 has multiple BF microphones M, a detection camera 10 , a storage unit 11 and a controller 20 .

A plurality of BF microphones M are sound collecting devices each capable of collecting sound in a specific direction using beam forming (BF) technology.
Four BF microphones M1 to M4 are schematically illustrated as the plurality of BF microphones M in FIG. Note that the number of BF microphones M is not limited.
Here, the beamforming technique is a technique of setting a beam extending in a specific direction from the BF microphone M and collecting sound waves arriving along the beam with high sensitivity. In this case, the direction in which the beam is set is the direction in which the BF microphone M collects sound.
Each BF microphone M is arranged at a predetermined position set in the space where the speaker 1 is present. An example of arrangement of the BF microphones M in the sound collection system 100 will be described later in detail.
In this way, each BF microphone M is a device that is arranged around the speaker 1 who is a sound source and can set the sound collecting direction. In this embodiment, the BF microphone M corresponds to a sound collector.

FIG. 2 is a schematic diagram showing a configuration example of the BF microphone M. As shown in FIG. FIG. 3 is a schematic diagram showing an example of the beam 7 set on the BF microphone M. As shown in FIG.
The BF microphone M shown in FIG. 2 has a flat board 15 and a plurality of microphones 16 arranged on the board 15 . That is, the BF microphone M is a microphone array in which a plurality of microphones 16 are arranged.
2A is a plan view of the BF microphone M seen from a direction orthogonal to the substrate 15, and FIG. 2B is a side view of the BF microphone M seen from a direction parallel to the substrate 15. FIG.

The substrate 15 is a plate member having a circular planar shape, and has a first surface 17a and a second surface 17b opposite to the first surface 17a. The first surface 17a is a surface on which a plurality of microphones 16 are arranged. 2A is a plan view of the first surface 17a of the BF microphone M. FIG. 2B, the upper surface of the substrate 15 in the drawing is the first surface 17a, and the lower surface of the substrate 15 in the drawing is the second surface 17b.
The multiple microphones 16 are elements that generate electrical signals (sound signals) corresponding to sound waves. Each microphone 16 is configured as an omnidirectional microphone, and detects sound waves with substantially constant sensitivity regardless of the arrival direction of the sound waves. As the microphone 16, for example, a dynamic microphone, a condenser microphone, or the like is used.

In the example shown in FIG. 2B , each microphone 16 is arranged with the sound receiving portion for receiving sound waves facing away from the substrate 15 . In this case, the first surface 17a side is the sound receiving side of the BF microphone M. FIG. In this configuration, for example, a cover or the like for protecting each microphone 16 may be provided on the first surface 17a side.
The configuration is not limited to this, and the BF microphone M may be configured such that the second surface 17a side is the sound receiving side of the BF microphone M. In this case, a microphone hole penetrating from the first surface 17a to the second surface 17b is provided at the placement position of each microphone 16 on the substrate 15 . Each microphone 16 is arranged with its sound receiving portion directed toward the microphone hole.

As shown in FIG. 2A, the BF microphone M is provided with eight microphones 16a-16h. The microphones 16a to 16h are arranged so as to be rotationally symmetric with respect to the center of the substrate 15 (substrate center C) on the first surface 17a. Therefore, the angle (angular interval) between the two line segments connecting the substrate center C and the two microphones 16 adjacent to each other is 45°.
In the following, the azimuth angle φ of the microphone 16a viewed from the substrate center C is assumed to be 0°. Also, in FIG. 2A, the azimuth angle φ increases in the clockwise direction (clockwise direction of rotation while viewing the substrate center C on the right side). Therefore, the azimuth angles at which the microphones 16a to 16h are arranged are 0°, 45°, 90°, 135°, 180°, 225°, 270° and 315°.

The BF microphone M is typically used with the substrate 15 (first surface 17a or second surface 17b) arranged horizontally. Therefore, the azimuth angles of the microphones 16a to 16h can be treated as azimuth angles in the horizontal plane. Note that the posture of the BF microphone M is not limited. For example, it is possible to arrange the BF microphone M tilted with respect to the horizontal plane.

The BF microphone M outputs each sound signal generated by the microphones 16a to 16h. That is, multi-channel sound signals generated by the plurality of microphones 16a to 16h are output from the BF microphone M. FIG.
A controller 20 (sound collection processing unit 23), which will be described later, performs beamforming processing on these sound signals.
In the beamforming process, a beam 7 directed in a specific direction is set, and a process of collecting sound waves arriving along the beam 7 is performed. For example, the propagation delay (difference in arrival time) of sound waves arriving along the beam 7 to each of the microphones 16a to 16h is corrected. Also, the signals whose propagation delays have been corrected are appropriately added to generate a signal in which the sound wave arriving along the beam 7 is emphasized. This makes it possible to selectively collect sound waves arriving along the beam 7 .
Thus, the sound collection direction 3 of the BF microphone M is the direction of the beam 7 set in the beamforming process for the BF microphone M. FIG.

FIG. 3 schematically shows the range of the beam 7 set on the BF microphone M using a gray area. In the BF microphone M, the range of the beam 7 is a fan-shaped range centered on the sound collecting direction 3 from the center C of the substrate. The range of this beam 7 is defined by the sound collection azimuth A and the beam width β.

A sound collection azimuth angle A is an azimuth angle representing the central angle of the sound collection direction 3 . For example, when the BF microphone M is regarded as a microphone having directivity in the sound collection direction 3, the sound collection azimuth angle A corresponds to the direction of the microphone having directivity.
In the BF microphone M, by arranging the eight microphones 16a to 16h in rotational symmetry, it is possible to set the sound collection azimuth angle A in all directions of 360°, that is, to extend the beam in all directions of 360°. It is possible. Therefore, it can be said that the BF microphone M shown in FIG. 2 is a beam forming microphone array corresponding to 360 degrees of sound source directions.

The beam width β is an angle representing the directivity of the BF microphone M with respect to the sound collection azimuth A. The smaller the beam width β, the higher the directivity. Also, the larger the beam width β, the wider the sound-collectable range. In this embodiment, the beam width β is set to a constant value.
The beam width β can be made variable by increasing the size of the BF microphone M, such as the number of microphones 16 and the diameter of the microphone array. In this case, for example, a process of changing the beam width β according to the situation or scene of the speaker 1 may be performed.

In this embodiment, the sound collection azimuth angle A is set so as to sequentially follow the speaker 1 based on information on the position of the speaker 1 detected using an external sensor (detection camera 10). . By controlling the azimuth angle range of the beam 7 to A±β with respect to the target speaker 1, it is possible to achieve high-quality sound collection of the voice 5 of the speaker 1, which is the target sound. It becomes possible.
A method for setting the sound collection azimuth angle A will be described later in detail.

Returning to FIG. 1, the detection camera 10 is a camera that captures an image of the speaker 1 who is the sound source. The detection camera 10 is arranged, for example, facing the space where the speaker 1 is present, and photographs the speaker 1 while the sound collection system 100 is operating.
As the detection camera 10, a digital camera equipped with an image sensor such as CMOS or CCD is used. Further, as the detection camera 10, a distance measuring camera capable of measuring depth, such as a stereo camera or a ToF camera, may be used.
One detection camera 10 may be used, or a plurality of detection cameras 10 may be used.

The storage unit 11 is a non-volatile storage device such as an SSD (Solid State Drive) or HDD (Hard Disk Drive). In addition, any non-transitory computer-readable recording medium may be used.
As shown in FIG. 1, the storage unit 11 stores a control program 12, microphone information 13, and a voice database (voice DB 14).

The control program 12 is a program that controls the operation of the sound collection system 100 as a whole.
The microphone information 13 is information about a plurality of BF microphones M. FIG. For example, the three-dimensional coordinates of the position where each BF microphone M is arranged, the posture of each BF microphone M, and the like are stored as microphone information. These pieces of microphone information are appropriately referred to when performing beam forming processing. In addition, the type, model number, etc. of the BF microphone M may be stored as the microphone information 13 .
The voice DB 14 is a database in which the voice data 6 of the speaker 1 is recorded. For example, the voice data 6 generated by the controller 20 are sequentially recorded together with the label of the speaker 1 . Further, for example, when there are a plurality of speakers 1, voice data 6 is recorded for each speaker 1. FIG.

The controller 20 controls the operation of each block included in the sound collection system 100 . The controller 20 has a hardware configuration necessary for a computer, such as a CPU and memory (RAM, ROM). Various processes are executed by the CPU loading the control program 12 stored in the storage unit 11 into the RAM and executing it.

The controller 20 is configured using a computer such as a PC, for example. As the controller 20, for example, a device such as a PLD (Programmable Logic Device) such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) may be used.

In this embodiment, the CPU of the controller 20 executes the control program 12 according to this embodiment, thereby realizing an image processing unit 21, a sound collection control unit 22, and a sound collection processing unit 23 as functional blocks. These functional blocks execute the information processing method according to the present embodiment. In order to implement each functional block, dedicated hardware such as an IC (integrated circuit) may be used as appropriate.

The image processing unit 21 performs various types of image processing on the image captured by the detection camera 10 to generate sound source information. Here, the sound source information is information about a sound source to be collected by the sound collection system 100 .
The sound source information includes information for identifying the sound source. For example, when a plurality of sound sources are targeted for sound collection, an ID or the like for identifying each sound source is generated as sound source information.
The sound source information includes information indicating the position of the sound source and information indicating the direction in which the sound source emits sound. That is, information indicating the position and direction in which the sound source emits sound is generated as the sound source information.
Thus, the image processing unit 21 acquires sound source information indicating the position of the sound source and the direction in which the sound source emits sound. In this embodiment, the image processing unit 21 corresponds to an information acquiring unit that acquires sound source information.

In this embodiment, sound source information is generated for the speaker 1 who is the sound source.
Therefore, the information identifying the sound source is the information identifying the speaker 1 (name, ID, etc. of the speaker 1). The image processing unit 21 identifies the speaker 1 from the image data of the speaker 1 captured by the detection camera 10 . To identify the speaker 1, processing such as individual identification using image recognition technology is used.

Information indicating the position of the sound source is information indicating the position of the speaker 1 .
The image processing unit 21 calculates the position of the speaker 1 from image data obtained by photographing the speaker 1 using the detection camera 10 . The information indicating the position of speaker 1 may be two-dimensional coordinates on the floor where speaker 1 is located, or may be three-dimensional coordinates of the head of speaker 1 .
A method for calculating the position of speaker 1 is not limited.

Also, the direction in which the sound source emits sound is the speaking direction of speaker 1 . The speaking direction is, for example, the direction in which the front of the head of the speaker 1 is directed. The sound source information includes such information indicating the speaking direction of the speaker 1 (for example, information indicating the orientation of the head of the speaker 1, etc.).
The image processing unit 21 performs bone detection (skeletal estimation) of the speaker 1 based on image data of the speaker 1 captured by the detection camera 10 to estimate the direction of speech of the speaker 1 . By using bone detection, it is possible to accurately estimate the direction of speech. Moreover, even when there are a plurality of speakers 1, the speaking direction of each speaker 1 can be easily estimated.
Note that the method of detecting the speech direction is not limited to the method using bone detection, and any method that can estimate the orientation of the head, for example, may be used.

For example, when speaker 1 can be identified, the position and speech direction of speaker 1 are sequentially calculated. Also, when there are a plurality of speakers 1, each speaker 1 is individually identified, and sound source information (position and speaking direction) is calculated for each speaker 1. FIG.
As described above, in the sound collection system 100, the detection camera 10 and the image processing unit 21 constitute a detection device that identifies the speaker 1 to be sound-collected and detects the position and speech direction of the speaker 1. be.

The sound collection control unit 22 controls the sound collection operation of the sound collection system 100 .
In the present embodiment, the sound collection control unit 22, based on the sound source information described above, selects a plurality of BF microphones M that are arranged around the sound source (speaker 1) and that can set the sound collection direction 3. Sound emitted by the sound source Select at least one target microphone 25 to be used for collecting (speech 5 of speaker 1).
Here, the target microphone 25 is the BF microphone M used to generate the voice data 6 of the speaker 1 who is the target of sound collection. That is, the output of the BF microphone M selected as the target microphone 25 is used as the original data of the voice data 6 .

The target microphone 25 is selected based on the position and speaking direction of the speaker 1 indicated by the sound source information.
In this process, for example, the BF microphone M that can detect the voice 5 of the speaker 1 with sufficient sensitivity is selected as the target microphone 25 . One or a plurality of BF microphones M may be selected. As a result, it is possible to select the BF microphone M suitable for the state of the speaker 1 as the target microphone 25 .
In the example shown in FIG. 1, the BF microphone M1 is selected as the target microphone 25. In the example shown in FIG.

Further, in this embodiment, the sound collection control unit 22 sets the sound collection direction 3 of the target microphone 25 based on the sound source information. That is, the direction of the beam 7 of the target microphone 25 is set based on the position and speaking direction of the speaker 1 indicated by the sound source information.
In this process, the sound collection direction 3 (the direction of the beam 7) is set so that the sound can be collected along the utterance direction of the speaker 1, for example. As a result, it is possible to set an appropriate sound collection direction that matches the utterance direction 2 .

When a plurality of speakers 1 are to be sound-collected, the target microphone 25 is selected for each speaker 1 based on the sound source information of each speaker 1, and the sound-collecting direction 3 is set. .

As shown in FIG. 1, in the sound collection control unit 22, a signal (sound selection signal) that designates the target microphone 25 among the plurality of BF microphones M and a signal that designates the sound collection direction 3 regarding the target microphone 25 (sound collection direction signals) are generated.
The audio selection signal is output to the sound collection processing unit 23 . For the BF microphone M selected as the target microphone 25, the sound collection direction 3 is set to the direction specified by the sound collection direction signal.
Note that FIG. 1 schematically shows how the sound collection direction signal is output to each BF microphone M. As shown in FIG. In practice, the sound collection direction signal is output to the sound collection processing unit 23 and used in the beamforming process for the target microphone 25 executed by the sound collection processing unit 23 .

The sound collection processing unit 23 generates sound data 6 representing the sound 5 uttered by the speaker 1 based on the output of at least one target microphone 25 .
As described above, the output of the target microphone 25 is the sound signal generated by the plurality of microphones 16a to 16h that constitute the target microphone 25. FIG. A beam forming process is performed on these sound signals to generate audio data 6 obtained by collecting the voice 5 of the speaker 1 . In this embodiment, the sound data 6 corresponds to sound data representing sound produced by a sound source.
As shown in FIG. 1 , the sound collection processing unit 23 has a microphone switching unit 27 and an audio data generation unit 28 .

The microphone switching unit 27 selects the target microphone 25 from the plurality of BF microphones M based on the voice selection signal. The microphone switching unit 27 can read the outputs of all the BF microphones M. Among them, the output of the BF microphone M designated as the target microphone 25 by the voice selection signal is read. Therefore, it can be said that the microphone switching unit 27 selects the target microphone 25 by reading the output of the target microphone 25 among the outputs of the plurality of BF microphones M.

Note that the microphone switching unit 27 shown in FIG. 1 is schematically illustrated as a switching switch that selects a single BF microphone M as the target microphone 25 from among the four BF microphones M1 to M4. Without being limited to this, the microphone switching unit 27 can also select a plurality of BF microphones M as the target microphones 25 from among the four BF microphones M1 to M4.

The audio data generation unit 28 generates audio data 6 by executing beamforming processing on the output of the target microphone 25 (sound signals of the microphones 16a to 16h) read by the microphone switching unit 27 .
In the beamforming process, the beam 7 is set in the sound collection direction 3 specified by the sound collection direction signal. Then, for sound waves arriving along the set beam 7, processing for correcting propagation delay, processing for adding sound signals after correction, and the like are executed.
In addition to the beamforming process, a process of adjusting the intensity of each sound signal, a process of removing noise, and the like may be performed.

The audio data 6 generated by the audio data generator 28 is output to a predetermined reproducing device 29 . Alternatively, the voice data 6 is stored in the voice DB 14 configured in the storage unit 11 .
Note that when a plurality of speakers 1 are to be sound-collected, voice data 6 is generated for each speaker 1 based on the output of the target microphone 25 selected for each speaker 1 .

FIG. 4 is a schematic diagram showing the basic sound collection operation of the sound collection system 100. As shown in FIG. FIG. 4 schematically shows a speaker 1, two BF microphones M1 and M2, and a detection camera 10. FIG.
In the following, the position of speaker 1 is denoted as Q, and the positions of BF microphones M1 and M2 are denoted as P1 and P2, respectively. Also, the description will be made assuming that the speaking direction 2 of the speaker 1 and the sound collecting direction 3 of the BF microphone M are in the horizontal plane. In FIG. 4, the utterance direction 2 and the sound collection direction 3 are schematically illustrated using solid white arrows and solid black arrows, respectively.
Also, the angle formed by the utterance direction 2 of the speaker 1 and the direction of the BF microphone M viewed from the speaker 1 is referred to as the sound collection angle of the BF microphone M.

In FIG. 4, speaker 1 faces to the right in the figure. Therefore, the utterance direction 2 of the speaker 1 is directed to the right side in the figure.
A BF microphone M1 is arranged at a position shifted to the left side from the front of the speaker 1, and a BF microphone M2 is arranged at the right side as seen from the speaker 1. - 特許庁Therefore, the sound collection angle of the BF microphone M1 is smaller than the sound collection angle of the BF microphone M2. Note that the position of the BF microphone M1 is farther from the position of the BF microphone M2 as viewed from the speaker 1. FIG.

For example, consider the case of selecting the BF microphone M for collecting the voice 5 of the speaker 1 using only the positional information of the speaker 1 detected by the detection camera 10 . If only the positional information is referred to, for example, the BF microphone M2 closest to the speaker 1 is selected.

By the way, in the scene shown in FIG. 4, the speaker 1 does not face the direction of the BF microphone M2. direction toward P2) exceeds 90°.
For example, if the voice 5 uttered at the utterance position (the mouth of the speaker 1) is a point sound source, the speaker 1 itself becomes an obstacle. Therefore, the BF microphone M2 collects the diffracted sound rather than the direct sound emitted from the mouth.

Here, the direct sound is the sound 5 that reaches the BF microphone M from the sound source without being blocked by obstacles or the like.
On the other hand, the sound 5 blocked by the obstacle and propagated around the obstacle (the sound 5 diffracted by the obstacle) becomes a diffracted sound. For example, when the sound collection angle is sufficiently large, the number of diffractions of the sound 5 increases, and the amount of attenuation of the sound 5 increases accordingly.

Further, as shown in FIG. 4, the BF microphone M2 directly picks up ambient noise 30 coming from the left side of the speaker 1 . Therefore, when the voice 5 of the speaker 1 is collected using the BF microphone M2, the volume level of the ambient noise 30 is higher than that of the voice 5, which is the target sound.

On the other hand, in the scene shown in FIG. 4, the BF microphone M1 is arranged near the front of the speaker 1 . Therefore, the sound collection angle of the BF microphone M1 with respect to the utterance direction 2 is less than 90°. Therefore, when the BF microphone M1 is used, the direct sound uttered by the speaker 1 can be collected, and the amount of attenuation of the sound 5 can be suppressed sufficiently compared to the case of collecting the diffracted sound.
Also, the BF microphone M1 does not directly pick up the environmental noise 30 . Thereby, the noise level of the speech 5 of the speaker 1 can be sufficiently suppressed.

Therefore, in the sound collection system 100, based on the video signal (image data) captured by the detection camera 10, the image processing unit 21 simultaneously detects the position of the speaker 1 and detects the bones of the speaker 1. Direction 2 is detected.
Based on the information (sound source information) of the position Q of the speaker 1 and the speaking direction 2 (sound source information) obtained in this way, the BF microphone M (target microphone 25) that collects the voice 5 of the speaker 1 by the sound collection control unit 22 is selected. selected. Also, the sound collection control unit 22 sets the sound collection direction 3 of the target microphone 25 .

In the process of selecting the target microphone 25, the BF microphone M capable of collecting the direct sound emitted by the speaker 1 is determined based on the speaking direction 2 in which the speaker 1, which is the sound source, emits the voice 5, and the BF microphone M is selected. It is selected as the target microphone 25 .
For example, by determining whether or not the sound collection direction 3 can be set within a predetermined range centered on the utterance direction 2, it is determined whether or not the direct sound can be collected. For example, when the sound source is the speaker 1, a range of ±90° centering on the speaking direction 2 is set as the predetermined range.
The method of determining whether direct sound can be collected is not limited, and determination may be made according to the presence or absence of an obstacle, for example.
In the example shown in FIG. 4, the BF microphone M1, which is displaced to the left from the speaking direction 2, is selected as the target microphone 25 because it can collect the direct sound.

In the process of setting the sound collection direction 3, the direction from the target microphone 25 toward the speaker 1 is set as the sound collection direction 3 of the target microphone 25. FIG. As a result, the direct sound uttered by the speaker 1 can be collected most efficiently.
In the example shown in FIG. 4, the direction from the position P1 of the BF microphone M1, which is the target microphone 25, toward the position Q of the speaker 1 is set as the sound collection direction 3 of the BF microphone M1. Also, the range of the beam 7 of the BF microphone M1 is a fan-shaped area that spreads at an angle of ±β centering on the sound collecting direction 3 toward the speaker 1 .

In this way, the sound collection system 100 includes a plurality of sound collection devices (BF microphones M) capable of collecting sound from a specific direction, and the position Q and speech direction 2 of the speaker 1 to be collected. A mechanism (the detection camera 10 and the image processing unit 21) is provided. Then, the sound collection control unit 22 selects the BF microphone M at the position Q and the speaking direction 2 of the speaker 1, and the sound collection processing unit 23 generates voice data 6 of the speaker 1. FIG. This makes it possible to collect the speech 5 of the speaker 1 with good quality.

For example, in a conference system that collects sound using a sound collecting microphone near speaker 1, if speaker 1 turns his back to the sound collecting microphone There was a possibility that the sound volume and sound quality would be greatly reduced. For example, a similar problem occurs when using a microphone array with beamforming technology.

On the other hand, in the sound collection system 100 according to the present embodiment, the sound collection operation is performed by selecting the BF microphone M that matches the position Q and the utterance direction 2 of the speaker 1 from a plurality of BF microphones M. .
For example, at a video content production site, etc., the position of the microphone is moved so as to collect sound from the front of the performer. Also, when collecting sound from the front of the performer, if noise coming from behind is expected to be mixed in, change the position and posture of the microphone so that the noise source does not enter the directional range of the microphone to achieve high sound quality. Sound collection is realized.
In the sound collection operation performed by the sound collection system 100, by selecting the BF microphone M that can collect sound from the front of the speaker 1, the same effect as the sound collection method at the production site described above is exhibited. .

Further, in the sound collecting system 100, while the sound collecting operation is being performed, the image processing unit 21 repeats the process of calculating the sound source information (the position Q and the speaking direction 2) of the speaker 1 at a predetermined frame rate. executed. Therefore, it can be said that the image processing unit 21 monitors sound source information.
In addition, the sound collection control unit 22 dynamically calculates a signal (voice selection signal and sound collection direction signal) specifying the target microphone 25 and the sound collection direction of the target microphone 25 according to the monitoring result of the sound source information. be. Then, the sound collection processing unit 23 generates the sound data 6 based on the sound selection signal and the sound collection direction signal.
As a result, it is possible to dynamically collect sound according to the position and speaking direction of the speaker 1 at each timing, and it is possible to always collect the voice 5 of the speaker 1 with high sensitivity. Become.

FIG. 5 is a flowchart showing an operation example of the sound collection system. FIG. 6 is a schematic diagram showing an arrangement example of the BF microphones M. As shown in FIG.
The processing shown in FIG. 5 is processing for selecting the target microphone 25 to be used for sound collection from the four BF microphones M1 to M4 arranged as shown in FIG. The process of setting the sound collection direction for the target microphone 25, the process of generating the audio data 6 from the output of the target microphone 25, and the like are appropriately executed after the target microphone 25 is selected.
The processing shown in FIG. 5 is loop processing that is repeatedly executed at a predetermined frame rate while the sound collection operation is being performed.

First, the arrangement of the BF microphones M shown in FIG. 6 will be described. Here, four BF microphones M1 to M4 are arranged at four vertices of a square area. This square area is the sound collection target area 40 of the sound collection system 100 . Here, at each point in the sound collection target area 40, the azimuth angle in the upward direction in the drawing is assumed to be 0°, and the azimuth angle increases in the clockwise direction.
The BF microphone M1 is placed at the upper right vertex in the figure, the BF microphone M2 is placed at the lower right vertex in the figure, the BF microphone M3 is placed at the lower left vertex in the figure, and the BF microphone M4 is placed at the upper left vertex in the figure. placed.

Further, in this embodiment, the plurality of BF microphones M are configured so that the sound collection direction 3 can be set in the allocation range 41 allocated according to the arrangement of each.
The allocation range 41 is, for example, an angle range in which each BF microphone M is in charge of sound collection, and is typically a range of azimuth angles on a horizontal plane. The allocation range 41 is appropriately set according to the position of each BF microphone M and the shape of the sound collection target area 40 .

FIG. 6 schematically illustrates the allocation range 41 of the BF microphone M1 using arc-shaped arrows. The allocation range 41 of the BF microphone M1 is a range from 180° to 270° with respect to the BF microphone M1. Similarly, the allocation range 41 of the BF microphone M2 ranges from 270° to 360°, the allocation range 41 of the BF microphone M3 ranges from 0° to 90°, and the allocation range 41 of the BF microphone M4 is It ranges from 90° to 180°.
Each BF microphone M can set the sound collection direction 3 at least within the allocation range 41 described above.

As shown in FIG. 5, the image processing unit 21 first detects the speaker 1 from the image data captured by the detection camera 10 (step 101). Any image processing for detecting a person, for example, is used to detect the speaker 1 . At this time, speaker 1 identification may be performed.

Also, in step 101, when the speaker 1 is detected, the position coordinates of the speaker 1 are detected. Here, the two-dimensional coordinates (xy coordinates) of the position Q of the speaker 1 in the sound collection target area 40 are detected.
Also, in step 101, bone detection is performed for speaker 1, and speech direction 2 of speaker 1 is detected. Here, the azimuth angle (frontal angle) of the speech direction 2 in the sound collection target area 40 is detected.

FIG. 7 is a schematic diagram showing an example of the speech direction 2 of the speaker 1. As shown in FIG.
As shown in FIG. 7, it is calculated based on the position Q of speaker 1 . Here, as viewed from position Q of speaker 1, the azimuth angle in the upward direction in the figure is assumed to be 0°. The azimuth angle in the right direction in the figure is 90°, the azimuth angle in the downward direction in the figure is 180°, and the azimuth angle in the left direction in the figure is 270°.
The utterance direction 2 of the speaker 1, that is, the frontal angle θ of the speaker 1 is calculated as an azimuth angle of 0° to 360°. For example, the angle θ of speech direction 2 shown in FIG. 7 is approximately 120°.

If the position Q and the speech direction 2 of the speaker 1 cannot be detected, information indicating that each parameter cannot be detected may be recorded.

Next, it is determined whether speech direction 2 is detectable (step 102).
For example, when the speech direction 2 is not detected by the image processing unit 21, it is determined that the speech direction 2 cannot be detected (No in step 102), and whether the position Q (xy coordinates) of the speaker 1 can be acquired. is determined (step 103).
For example, when the position Q of the speaker 1 is not detected by the image processing unit 21, it is determined that the position Q of the speaker 1 cannot be detected (No in step 103), and step 101 is executed again.

On the other hand, when the position Q of the speaker 1 is detected, it is determined that the position Q of the speaker 1 is detectable (Yes in step 103), and the BF microphone M closest to the position Q of the speaker 1 is installed. is selected as the target microphone 25 (step 104).
In this way, when the utterance direction 2 is unknown, but the position Q of the speaker 1 is known, the BF microphone M (in FIG. is selected). Note that N is an index representing the BF microphone M, where N=1, 2, 3, 4.
At step 104, when the target microphone 25 is selected, the following loop processing is executed.

Returning to step 102, when the speech direction 2 is detected by the image processing unit 21, it is determined that the speech direction 2 is detectable (Yes in step 102). The presence or absence is determined (step 105).

Here, the BF microphone M most suitable for the utterance direction 2 is the BF microphone M for which the utterance direction 2 and the center direction of the allocation range 41 correspond.
By using such a BF microphone M, it is possible to collect the sound 5 arriving along the center of the allocation range 41 . As a result, processing such as effectively emphasizing the voice 5 and suppressing other noise becomes possible, and high-quality voice data 6 can be generated.
Specifically, it is determined whether or not the angle θ of the speaking direction 2 satisfies the following relationship.
θ=90°×N-45° (1)

From the equation (1), when N=1, θ=45°. This utterance direction 2 of θ=45° is a direction obtained by rotating the central direction (225°) of the allocation range 41 (180° to 270°) of the BF microphone M1 by 180°. direction. That is, utterance direction 2 at θ=45° corresponds to the central direction of allocation range 41 of BF microphone M1. In this case, the BF microphone M1 is the most suitable BF microphone M for the speaking direction 2. FIG.
Similarly, for N=2, 3, and 4, BF microphones M2, M3, and M4 are the most suitable BF microphones M for speech direction 2, respectively, if equation (1) is satisfied.

Note that in step 105, processing may be performed in which a certain width α is given to the determination of θ by equation (1). For example, it is determined for each N whether the angle θ of the speaking direction 2 satisfies (90°×N−45°−α)≦θ≦(90°×N−45°+α). As described above, even if the direction of speech 2 is slightly deviated from the central direction of the allocation range 41, it is possible to generate high-quality voice data 6. FIG.

If there is N that satisfies the expression (1) (Yes in step 105), the BF microphone (N) that satisfies the expression (1) is selected as the target microphone 25 as the BF microphone M most suitable for the utterance direction 2 ( step 106).
Thus, in this embodiment, the BF microphone M whose center direction of the allocation range 41 corresponds to the speech direction 2 is selected as the target microphone 25 . This makes it possible to collect the voice 5 of the speaker 1 with sufficiently high quality.
At step 106, when the target microphone 25 is selected, the following loop processing is executed.

Returning to step 105, if there is no N satisfying the formula (1) (No in step 105), the BF microphone M closest to speaker 1 is detected from the xy coordinates of position Q of speaker 1 (step 107 ).
For example, in the example shown in FIG. 6, it is determined that there is no N that satisfies the expression (1) for the speech direction 2 of the speaker 1, and the BF microphone M4 (N=4) closest to the speaker 1 is detected.

For the BF microphone M detected in step 107, it is determined whether or not sound can be collected along the speaking direction 2 (step 108). Here, the sound collection along the utterance direction 2 is a sound collection operation performed in a state where the utterance direction 2 is included in the direction range of the beam 7 .
As explained with reference to FIG. 6, here each BF microphone M can set the sound collection direction 3 within the allocation range 41 of 90°. Therefore, the range of azimuth angles that can be set by the N-th BF microphone M is from 90°×(N−1)−β to 90°×N+β.
At step 108, for the BF microphone (N) closest to the speaker 1, it is determined whether or not the angle .theta. This is a process of determining whether or not the following relationship is satisfied.
90×(N−1)−β≦θ≦90°×N+β (2)

The determination of expression (2) will be described with reference to FIG. Here, since the BF microphone M4 (N=4) is detected as the nearest BF microphone M, the formula (2) is 270-β≤θ≤360°+β. This corresponds to the range of the beam 7 that can be set under the condition that the sound collection direction 3 is set in the allocation range 41 of the BF microphone M4. It is determined whether or not the angle θ of the speech direction 2 is included in this range.
Thus, it can be determined whether or not the BF microphone M closest to the speaker 1 can collect sound along the speaking direction 2 .

If the expression (2) is satisfied (Yes in step 108), the nearest BF microphone (N) detected in step 107 is selected as the target microphone 25 (step 109). This makes it possible to collect the voice 5 from the position closest to the speaker 1 with sufficient sensitivity.
At step 109, when the target microphone 25 is selected, the following loop processing is executed.

Also, if the formula (2) is not satisfied (No in step 108), the nearest BF microphone (N) detected in step 107 is not selected as the target microphone 25. FIG. In this case, it is determined whether or not the next BF microphone (N+1) can collect sound along the speech direction 2 (step 110).
In this process, it is determined whether or not the angle θ of the speaking direction 2 satisfies the following relationship.
90×N+β<θ≦90×(N+1)+β (3)

Expression (3) is such that the BF microphone (N+1) adjacent to the BF microphone (N) closest to the speaker 1 is in the range of the settable beam 7 that does not overlap with the BF microphone (N). is a conditional expression for determining whether the angle θ of is included.
In the example shown in FIG. 6, it was the nearest BF microphone M4. In this case, in step 110, determination processing is executed for the range of the beam 7 in which the next BF microphone M1 (N=1) can be set separately from the BF microphone M4.

If the expression (3) is satisfied (Yes in step 110), the BF microphone (N+1) adjacent to the nearest BF microphone (N) is selected as the target microphone 25 (step 111). This makes it possible to collect the voice 5 from a position second (or third) closest to the speaker 1 with sufficient sensitivity.
At step 111, when the target microphone 25 is selected, the following loop processing is executed.

Further, if the formula (3) is not satisfied (No in step 110), the BF microphone (N−1) adjacent to the nearest BF microphone (N) on the opposite side of the BF microphone (N+1) is selected as the target microphone 25. (step 112). As a result, as in the case where the BF microphone (N+1) is selected, it is possible to collect the voice 5 from a position sufficiently close to the speaker 1 with sufficient sensitivity.
At step 112, when the target microphone 25 is selected, the following loop processing is executed.

The process performed in steps 107 to 112 is a process of retrieving BF microphones M capable of collecting sound along the speaking direction 2 in order of proximity and setting them as the target microphone 25 . As described above, in the present embodiment, when there is no BF microphone M whose speech direction 2 corresponds to the central direction of the allocation range 41, sound can be collected along the speech direction 2, and the distance to the sound source is the closest. BF microphone M is selected as the target microphone.
This makes it possible to set the BF microphone M capable of collecting the voice 5 with the highest possible sensitivity as the target microphone 25 . As a result, the sound quality of the audio data 6 can be sufficiently improved.

FIG. 8 is a schematic diagram for explaining the sound collection operation for a plurality of speakers 1. FIG. A sound collection operation when a plurality of speakers 1 are present in the sound collection target area 40 will be described below.
Here, it is assumed that four speakers 1A, 1B, 1C, and 1D sitting around a desk 43 placed in the center of a square-shaped sound-collection target area 40 are subjected to sound-collection operations. . Speakers 1A, 1B, 1C, and 1D are positioned at the upper left, upper right, lower right, and lower left in the drawing when viewed from the center of the target sound collection area 40, and are having a conversation facing each other.
BF microphones M1 to M4 are arranged at the four vertices of the sound collection target area 40, respectively, as in FIG.

When a plurality of speakers 1 are targeted for sound collection, the image processing unit 21 acquires sound source information for each of the plurality of speakers 1 (sound sources).
Specifically, from the image data captured by the detection camera 10 (not shown) of the sound collection target area 40, the position and the speaking direction 2 of each speaker 1 are obtained for each of the speakers 1A, 1B, 1C, and 1D. detected.

When the sound source information of each speaker 1 is obtained, the sound collection control unit 22 selects the target microphone 25 for each of the speakers 1 based on the sound source information of each speaker 1 . The sound collection control unit 22 also sets the sound collection direction 3 for each of the target microphones 25 selected for each of the plurality of speakers 1 .
In the example shown in FIG. 8, the BF microphone M1 arranged at the upper right of the sound collection target area 40 is selected as the target microphone 25 of the speaker 1A. Also, the BF microphone M4 arranged at the upper left of the sound collection target area 40 is selected as the target microphone 25 of the speaker 1B. Also, the BF microphone M3 arranged at the lower left of the sound collection target area 40 is selected as the target microphone 25 of the speaker 1C. Also, the BF microphone M2 arranged at the lower right of the sound collection target area 40 is selected as the target microphone 25 of the speaker 1D.

For example, suppose that the BF microphone M4 arranged in the immediate vicinity of the speaker 1A is used to collect the voice 5 of the speaker 1A. Here, speaker 1A is having a conversation while facing speaker 1B and speaker 1C facing each other across the desk. Therefore, the sound collection angle of the BF microphone M4 with respect to the speaking direction 2 of the speaker 1A is 90° or more. Furthermore, when collecting the voice 5 of the speaker 1A using the BF microphone M4, the sound collection direction 3 for beam forming is set within 90° of the speech direction 2 of the speakers 1B and 1C.
As a result, the BF microphone M4 collects the diffracted sound of the speaker 1A and the direct sounds of the speakers 1B and 1C, making it difficult to selectively collect the voice 5 of the speaker 1A. .

On the other hand, the BF microphone M1 can be selected as the target microphone 25 for collecting the voice of the speaker 1A by considering the information on the speaking direction 2, as in the processing described with reference to FIG. 5, for example. is possible. By using the BF microphone M1, it is possible to collect the direct sound of the speaker 1A. Also, the sound collection direction 3 set from the BF microphone M1 toward the speaker 1A hardly collects the voices 5 of the speakers 1B and 1C. In this way, since the speakers 1B and 1C can be placed outside the sound collection range of beamforming, it is possible to sufficiently suppress the influence of the speaker 1 who is not the target of sound collection.

The same effects as described above can be exhibited for the target microphones 25 set for the speakers 1B to 1D. As a result, even when there are a plurality of speakers 1, it is possible to collect the voice 5 of each speaker 1 individually and with good sound quality.

FIG. 9 is a schematic diagram showing an example of sound collection operation using a plurality of BF microphones M. FIG.
In FIG. 9, an example of collecting the voice of one speaker 1 using a plurality of BF microphones M will be described. In this case, the sound collection control unit 22 selects a plurality of target microphones 25 from a plurality of BF microphones M for a single sound source (single speaker 1).
Here, as in FIGS. 6 and 8, four BF microphones M1 to M4 are arranged in a square sound collection target area 40. FIG.

A speaker 1 shown in FIG. 9 is positioned above the center of the sound collection target area 40 in the figure, and is facing downward in the figure and uttering a voice 5 . For this reason, it is difficult for the BF microphones M1 and M4, which are close to the speaker 1, to collect the direct sound of the speaker 1. FIG.
In such a case, the sound collection control unit 22 selects both the BF microphones M2 and M3 located on the front side of the speaker 1 (the side toward which the speech direction 2 is directed) in the sound collection target area 40 as the target microphones of the speaker 1. selected. The sound collection processing unit 23 simultaneously collects the voice 5 of the speaker 1 using the BF microphones M2 and M3, and adds (synthesizes) the collected sound results to generate voice data 6. FIG.
By using the two BF microphones M2 and M3 in this way, it is possible to improve the sound collection level at the time of long-distance sound collection, and it is possible to collect the speech 5 of the speaker 1 without deteriorating the quality. It becomes possible.

FIG. 10 is a schematic diagram showing an example of the sound collection operation when the speaker 1 moves. FIG. 11 is a schematic diagram for explaining the process of synthesizing voice 5. As shown in FIG. Here, the operation of selecting the target microphone 25 when the speaker 1 moves within the sound collection target area 40 will be described with reference to FIGS. 10 and 11. FIG.
It is assumed that the speaker 1 moves from the upper left of the target sound collection area 40 to the lower left through the right side of the center. FIG. 10 schematically shows the position and speech direction 2 of speaker 1 at times T1, T2, T3, and T4. The gray color representing the range of the beam 7 corresponds to each time, and the darker the color, the later the beam 7 is set.

For example, at time T1, the speaker 1 is positioned at the upper left of the sound collection target area 40, and the speech direction 2 is directed to the right side in the figure. In this case, the BF microphone M1 becomes the target microphone 25, and the beam 7 is set toward the speaker 1. FIG.
At time T2, speaker 1 is approaching BF microphone M1 and speaking direction 2 is directed to the lower right in the figure. In this case, the BF microphone M2 is selected as the target microphone 25 along with the BF microphone M1.
At time T3, the speaker 1 is positioned on the right side of the center of the sound collection target area 40, and the speaking direction 2 is directed downward in the figure. In this case, the BF microphone M1 is removed from the target microphone 25, and the BF microphone M2 is selected as the target microphone 25. FIG.
At time T4, the speaker 1 is approaching the BF microphone M2, and the speaking direction 2 is directed toward the BF microphone M3 at the lower left in the figure. In this case, the BF microphone M3 is selected as the target microphone 25 together with the BF microphone M2.

As described above, in the present embodiment, the target microphone 25 is set by appropriately switching the plurality of BF microphones M as the speaker 1 moves.
Also, when sound can be collected by two BF microphones M, such as times T2 and T4, both BF microphones M are set as the target microphones 25, and the audio data 6 is synthesized using the data. That is, the sound collection processing unit 23 synthesizes the data collected by the plurality of target microphones 25 to generate the speech data 6 of the speaker 1 .
A method of synthesizing the audio data 6 using the two BF microphones M1 and M2 selected as the target microphones 25 will be described below, taking the case of time T2 as an example.

FIG. 11 schematically shows the positional relationship between speaker 1 and BF microphones M1 and M2 at time T2.
The angle between the direction from speaker 1 to BF microphone M1 (the direction from Q to P1) and the utterance direction 2 is denoted as _γ1 , and the direction from speaker 1 to BF microphone M2 (the direction from Q to P2) ) and the speaking direction 2 is denoted as γ ₂ . Also, the distance between speaker 1 and BF microphone M1 (distance between Q and P1) is indicated as _L1 , and the distance between speaker 1 and BF microphone M2 (distance between Q and P2) is indicated as _L2. do.
(γ ₁ , γ ₂ , L ₁ , L ₂ ) are calculated using bone detection and human position detection processing by the image processing unit 21, for example.

Here, a reference sound collection distance L is the distance at which the necessary speech level A can be collected when the sound is collected in front of the speaker 1 .
For example, the sound collection level A1 of the BF microphone M1 that collects sound at a position separated from the speaker 1 by a distance _L1 with respect to the reference sound collection distance L is expressed by the following equation.
A1=A×(L/ _L1 ) ² (4)
Similarly, the sound collection level A2 of the BF microphone M2 that collects sound at a position separated from the speaker 1 by a distance _L2 with respect to the reference sound collection distance L is expressed by the following equation.
A2=A×(L/L ₂ ) ² (5)

Also, each output of the BF microphones M1 and M2 is synthesized according to the following formula.
A _mix =sqrt{(A1×(L ₁ /L) ² ×cosγ) ² +(A1×(L ₁ /L) ² ×cosγ) ² }
... (6)
Here, A _mix is a synthesis level obtained by synthesizing the outputs of the BF microphones M1 and M2.
Also, sqrt{} means the square root of the value in {}.
γ is either one of (γ ₁ , γ ₂ ) described above.

From the equations (4) and (5), the required speech level A is expressed as follows.
A=A1×( _L1 /L) ² =A2×( _L2 /L) ² (7)
Therefore, the synthesis level A _mix synthesized according to the formula (6) is A _mix =A.
Thus, by using the equation (6), the synthesis level A _mix can always be kept at the same level as the speech level A.

In addition, γ in equation (6) is, for example, the sound collection angle of the BF microphone M (main microphone array) that mainly collects sound, out of the two BF microphones M (here, M1 and M2), with respect to the utterance direction 2. .
For example, based on the position Q of the speaker 1 and the speaking direction 2, the sound collection angle γ is −90°≦γ≦90°, and two BF microphones M close to the speaker 1 are selected as the target microphones 25. be. Of the two selected BF microphones M, the one closer to the speaker 1 is set as the BF microphone M that mainly collects sound, and its sound collection angle is used as γ in equation (6).

For example, in the situation shown in FIG. 11, the BF microphone M1 close to the speaker 1 is set as the BF microphone M that mainly collects sound, and its sound collection angle _γ1 is used as γ in equation (6).
Further, when the speaker 1 moves after time T2 and becomes γ ₁ =90° (or γ ₁ =−90°), the BF microphone M that mainly collects sound is switched to the BF microphone M2, γ in equation (6) is switched to the sound collection angle γ ₁ .
This makes it possible to realize continuous switching of adjacent BF microphones M. FIG. As a result, it is possible to continuously perform high-quality sound collection at a high sound collection level without causing unnatural sound interruptions or the like.

FIG. 12 is a schematic diagram showing an example of a sound collection operation when a plurality of speakers 1 move.
FIG. 12 illustrates a case where a plurality of speakers 1 move and sound collection operations for each speaker 1 interfere.
Here, it is assumed that two speakers 1A and 1B move within the sound collection target area 40 along the thick arrows in the drawing. 12A and 12B schematically show the arrangement of speakers 1A and 1B at time T1 and time T2.
The range of the beam 7 of the target microphone 25 of the speaker 1A is indicated by a light gray area, and the range of the beam 7 of the target microphone 25 of the speaker 1B is indicated by a dark gray area. Also, the dot area represents the range of the virtual beam 7 shown for comparison.

In FIG. 12A, the speaker 1A is positioned near the upper left outer periphery of the sound collection target area 40, and the speaking direction 2 of the speaker 1A faces the right side in the figure. Also, the speaker 1B is positioned near the outer periphery of the lower center of the sound collection target area 40, and the speech direction 2 of the speaker 1B is directed to the upper left in the figure.

In the situation shown in FIG. 12A, even if the voice 5 of the speaker 1A is collected by the BF microphone M1 closest to the front side of the speaker 1A, the sound collection direction 3 (the direction of the beam 7a) Person 1B) does not overlap. Therefore, the BF microphone M1 is selected as the target microphone 25 for the speaker 1A, and the beam 7a is set toward the speaker 1A.
Similarly, even if the voice 5 of the speaker 1B is collected with the BF microphone M3 that is closest to the front side of the speaker 1B, the other person (the speaker 1A) is in the sound collection direction 3 (direction of the beam 7c). Do not overlap. Therefore, the BF microphone M3 is selected as the target microphone 25 of the speaker 1B, and the beam 7b is set toward the speaker 1B.

Even if the beam 7d is directed toward the speaker 1A, the BF microphone M4 located closest to the speaker 1A picks up sound from behind the speaker 1A. Similarly, the BF microphone M2 located closest to the speaker 1B picks up the sound of the speaker 1B from behind even if the beam 7b is directed toward the speaker 1B. Therefore, since the beam 7d of the BF microphone M4 and the beam 7b of the BF microphone M2 cannot collect the direct sound of the speaker 1, the sound quality may deteriorate.

In FIG. 12B, the speaker 1A is positioned at the upper right of the center of the sound collection target area 40, and the speech direction 2 of the speaker 1A is directed to the lower right in the figure. Also, the speaker 1B is positioned at the lower left of the center of the sound collection target area 40, and the speech direction 2 of the speaker 1B is directed upward in the figure.

In the situation shown in FIG. 12B, when the speaker 1A is picked up using the BF microphone M1 as in FIG. 12A, the other person (speaker 1B) overlaps the beam 7a' of the BF microphone M1. Also, since the sound collection angle of the BF microphone M1 with respect to the speaking direction 2 of the speaker 1B is 90° or less, the direct sound emitted by the speaker 1B may be collected when the beam 7a' is used.
On the other hand, when the other BF microphone M2 in front of the speaker 1A is used to collect the sound of the speaker 1A, the other person (speaker 1B) does not overlap the beam 7b' of the BF microphone M2. Therefore, in FIG. 12B, the BF microphone M2 is selected as the target microphone 25 of the speaker 1A, and the beam 7b' is set toward the speaker 1A. This makes it possible to collect only the voice 5 of the speaker 1A with high quality.

The target microphone 25 is similarly switched for the speaker 1B shown in FIG. 12B. for example,
When the speaker 1B is collected using the BF microphone M3 as in FIG. 12A, the beam 7c' of the BF microphone M3 overlaps the other person (speaker 1A), and the direct sound emitted by the speaker 1A is Sound may be collected.
On the other hand, when the BF microphone M4 in front of the speaker 1B is used to collect the sound of the speaker 1B, the other person (speaker 1A) does not overlap the beam 7d' of the BF microphone M4. Therefore, in FIG. 12B, the BF microphone M4 is selected as the target microphone 25 of the speaker 1B, and the beam 7d' is set toward the speaker 1B. This makes it possible to collect only the voice 5 of the speaker 1B with high quality.

As described above, in this embodiment, the sound collection direction is changed so as to collect the direct sound uttered by the speaker 1 to be processed (sound collection target) and not to collect the direct sound uttered by another speaker 1 different from the processing target. 3 can be set is selected as the target microphone 25 .
As a result, for example, it is possible to generate voice data 6 by selectively collecting the voice 5 uttered by the speaker 1 to be processed.

13A and 13B are schematic diagrams showing an example of the sound collection operation assuming the speaking direction 2 of the speaker 1. FIG.
FIG. 13 illustrates a sound collection operation in a situation where it is possible to assume speech directed toward a plurality of speech directions 2 and the speech direction 2 switches relatively frequently.
Here, as an example, it is assumed that a remote conference is being held. Speakers 1A and 1B are sitting on the right and left sides of the sound collection target area 40 . A speaker 1</b>C, who is a participant in the remote conference, is displayed on the monitor 44 provided in the upper center of the sound collection target area 40 .

When a plurality of utterance directions 2 are assumed, a sound collection direction 3 corresponding to the assumed utterance direction 2 is set in advance for the corresponding BF microphone M. The BF microphone M for which the sound collection direction 3 is set in advance becomes a candidate microphone 26 that is a candidate for the target microphone 25 .
Thus, the multiple BF microphones M include multiple candidate microphones 26 for which the sound collection directions 3 are set in advance. In this embodiment, the candidate microphones 26 correspond to candidate devices.

Focusing on speaker 1A, in the situation shown in FIG. A case of speaking (speech direction 2 directed to the right) is assumed.
In this case, the BF microphones M4 and M1 are set as candidate microphones 26 for collecting the speech 5 of the speaker 1A.
For example, a sound collection direction 3a is set for the BF microphone M4 corresponding to an upward speaking direction 2a when the speaker 1A speaks to the speaker 1C. Similarly, a sound collection direction 3b is set for the BF microphone M1 in correspondence with the speech direction 2b directed to the right when the speaker 1A speaks to the speaker 1B.

In this way, the sound collecting operation for the speaker 1 is performed with the candidate microphones 26 set. Specifically, the target microphone 25 is selected from the plurality of candidate microphones 26 by the sound collection control unit 22 . For example, the actual speaking direction 2 of the speaker 1 is monitored, and the target microphone 25 is selected from each candidate microphone 26 according to the monitoring result.
In FIG. 13, it is assumed that speaker 1A is speaking to speaker 1C. In this case, the BF microphone M4 for which the sound collection direction 3a corresponding to the speaking direction 2a is set is selected as the target microphone 25. FIG. Then, the voice 5 of the speaker 1A is collected along the sound collection direction 3a by the BF microphone M4.

In addition, the sound collection processing unit 23 causes the candidate microphones 26 not selected as the target microphones 25 to stand by in the sound collection state. Here, the standby in the sound collecting state is, for example, a process of continuing the sound collecting process (beam forming process) in the background of the sound collecting operation by the target microphone 25 . Note that the voice data 6 generated during standby is deleted as appropriate.
In FIG. 13, since the BF microphone M4 is selected as the target microphone 25, the other candidate microphone 26, the BF microphone M1, is on standby in a sound collecting state. At this time, the BF microphone M1 continues the sound collection operation in the sound collection direction 3b.
As a result, even when the speech direction 2 suddenly changes, it is possible to continuously collect high-quality sound by switching to the sound collection by the candidate microphone 26 that is on standby.

For example, in FIG. 13, since speaker 1B is in the seat next to speaker 1A, even if speaker 1A faces the main direction (speech direction 2a) and talks to speaker 1C, the conversation with speaker 1B suddenly occurs. Conversation can start. Therefore, as described above, if the BF microphone M1 is on standby in the sound-collecting state in advance in the adjacent seat direction (speech direction 2b), the speaker 1A can frequently and quickly turn around and converse with the adjacent speaker 1B. , it is possible to collect the voice 5 of the speaker 1A without truncating.

FIG. 14 is a schematic diagram showing an example of a sound collection operation according to a gesture.
In FIG. 14, a method of controlling sound collection processing for speaker 1 according to gestures (specific actions) of speaker 1 will be described.
Here, the gesture of speaker 1 is detected by the image processing unit 21 . In this embodiment, the gesture of speaker 1 is detected from information on the skeleton of speaker 1 using a bone detection function that detects the speaking direction 2 of speaker 1 . The gesture of speaker 1 may be a static gesture (pose) or a dynamic gesture (movement).

14A to 14C schematically show the posture of speaker 1 using the skeleton of speaker 1. FIG. The skeleton of speaker 1 is represented by a plurality of coordinate points 45. For example, the head of speaker 1 is represented by head coordinate point 45a and neck coordinate point 45b. The right hand of speaker 1 is represented by a pair 46R of coordinate points 45 representing the right wrist and right palm, and the left hand of speaker 1 is represented by a pair 46L of coordinate points 45 representing the left wrist and left palm. ing.
It is not limited to this, and for example, coordinate points 45 representing other parts such as eyes, nose, ears, etc. may be used.

In this embodiment, the sound collection processing unit 23 controls sound collection processing for collecting the voice 5 of the speaker 1 according to the gesture of the speaker 1 .
Here, the sound collection process is a series of processes necessary for collecting the voice 5 of the speaker 1, for example. In the sound collection processing, in addition to the beamforming processing for generating the voice data 6, the image processing unit 21 detects the position Q and the speech direction 2 of the speaker 1, and the sound collection control unit 22 selects the target microphone 25. processing and processing for setting the sound collection direction 3 are included.
These processes are controlled according to the gesture of speaker 1 .

FIG. 14( a ) shows the general posture of speaker 1 . The general posture is, for example, the normal posture of the speaker 1, in which the speaker stands upright with his left and right hands down. A general posture may be set when the left and right hands (pair 46L and 46R) are positioned lower than the shoulder coordinate point 45, for example.
When the general posture is detected, normal sound collection processing is performed for speaker 1 .

FIG. 14(b) shows a stop gesture for stopping sound collection. A stop gesture is a posture of holding a hand in front of the mouth. In this way, when the stop gesture of speaker 1 covering his mouth with his hand is detected, the sound collection process for the speaker is stopped.
Here, the right hand (pair 46R) of speaker 1 is detected at a position overlapping between head coordinate point 45a and neck coordinate point 45b. When such a gesture is detected, it is assumed that speaker 1 has covered his mouth, and sound collection processing for speaker 1 is stopped. As a result, for example, it is possible to avoid a situation in which a conversation or the like, which the speaker 1 does not want to collect, is collected.
Note that the sound collection processing that is being executed for the other speaker 1 is continued as it is.

FIG. 14(c) shows a priority gesture that prioritizes sound collection. A priority gesture is a posture in which either the left or right hand is held above the head. In this way, when the priority gesture of raising the hand of speaker 1 is detected, the sound collection process for speaker 1 is preferentially executed.
Here, speaker 1's left hand (pair 46L) is detected at a position higher than head coordinate point 45a. When such a gesture is detected, it is assumed that speaker 1 has raised his/her hand to speak, and sound collection processing (prioritized sound collection) for preferentially collecting sound for speaker 1 is performed.
In the priority sound collection, for example, the accuracy of beam forming processing for collecting the voice of speaker 1 is raised. Alternatively, the detection accuracy of the utterance direction 2 and the like of the speaker 1 is raised. Conversely, the accuracy of the sound collection process being executed for the other speaker 1 may be lowered. Further, a process of independently collecting the voice 5 of the speaker 1 may be executed. As a result, for example, the voice of speaker 1 who wishes to speak can be collected with high quality.

FIG. 15 is a schematic diagram showing an example of sound collection operation for collecting voice and operation sound.
With reference to FIG. 15, a method of separating and collecting a gesture sound 8 accompanying an action such as movement of the speaker 1 will be described. In the following description, footsteps generated when the speaker 1 moves will be described as an example of the gesture sound. This processing is executed when movement of the speaker 1 is detected by bone detection or position detection, for example. Note that the process of separating the gesture sound 8 (footsteps) may be executed regardless of whether or not the speaker 1 moves.

FIG. 15A is a schematic diagram showing the spread of the beam 7 directed toward the speaker 1 from the target microphone 25 (BF microphone M) in the vertical direction. For example, the beam 7 set on the target microphone 25 spreads vertically as shown in FIG. 15A. Therefore, the target microphone 25 can collect not only the voice 5 of the speaker 1 but also the sound of footsteps (sound 8) generated at the feet of the speaker 1 .
Therefore, the voice data 6 generated based on the output of the target microphone 25 contains the voice 5 of the speaker 1 and the gesture sound 8 .

In this embodiment, the sound collection processing unit 23 separates the sound 5 of the speaker 1 and the gesture sound 8 of the speaker 1 from the sound data 6 collected by the target microphone 25 .
For example, by separating the utterance component from the voice data 6, it is possible to generate gesture sound data or the like in which the gesture sound 8 (footsteps) of the speaker 1 is collected.

FIG. 15B is a block diagram showing a configuration example of the sound collection processing unit 23 that separates the gesture sound 8. As shown in FIG. The sound collection processing unit 23 is provided with a sound source separation unit 35 after the audio data generation unit 28 described with reference to FIG.
The sound source separation unit 35 removes the voice 5 of the speaker 1 from the voice data 6 generated using the target microphone 25 and extracts the gesture sound 8 . An adaptive sound source separation process that changes parameters such as a separation frequency according to the content of data, the sound collection environment, and the like is used for extracting the motion sound 8 . Alternatively, a fixed bandpass filter (BPF) or the like may be used according to the characteristics of the sound 8 .

FIG. 15C is a schematic graph showing the frequency distribution of collected sound levels for voice 5 and gesture sound 8. FIG. The horizontal axis of the graph is frequency, and the vertical axis is sound collection level. Sound collection levels of voice 5 and gesture sound 8 are indicated using a solid line graph and a dashed line graph, respectively.
For example, the sound 5 is distributed in a relatively sharp peak shape centered at 1 kHz, and does not have frequency components in a region sufficiently higher (or lower) in frequency than 1 kHz. On the other hand, the gesture sound 8 exhibits a relatively broad distribution in a frequency range wider than that of the voice 5 . That is, the frequency components of the gesture sound 8 are distributed even in areas where the voice 5 does not have frequency components.

Thus, the frequency components of voice 5 are concentrated around 1 kHz. Therefore, the sound source separation unit 35 performs a process of removing frequency components around 1 kHz from the audio data 6 . In this way, the sound source separation unit 35 regards the data from which the frequency component around 1 kHz is removed as the gesture sound 8 (footsteps) and collects the sound.
FIG. 15C shows the frequency characteristics of a BPF that removes frequency components around 1 kHz using a dashed line graph. By applying such a BPF to the voice data 6, gesture sound data in which the voice 5 is removed and the gesture sound 8 is extracted is generated.
In addition, the method of extracting the gesture sound 8 is not limited, and for example, a sound source separation technique using machine learning or the like may be used as appropriate.

The action sound 8 (action sound data) separated from the voice 5 is output to the reproducing device 29 or the storage unit 11 as sound data of a track different from that of the voice 5, for example.
For example, in an application that reproduces the behavior of the speaker 1 at a remote location (remote conference, remote presentation, etc.), it is possible to improve the sense of presence by reproducing the voice 5 and the gesture sound 8 separately. is.
Also, for example, when recording video content, it is possible to record the gesture sound 8 on a separate track from the audio 5, thereby improving the quality of the content.

As described above, in the controller 20 according to the present embodiment, at least one target microphone 25 for collecting the voice 5 of the speaker 1 is selected from the plurality of BF microphones M arranged around the speaker 1 which is the sound source. be done. Each BF microphone M is a device that can set the sound collection direction 3, and sound source information indicating the position Q of the speaker 1 and the speech direction 2 in which the speaker 1 emits voice is used to select the target microphone 25. This makes it possible to use, for example, the BF microphone M adapted to the position of the speaker 1 and the direction from which the voice 5 is emitted, so that the voice 5 emitted by the speaker 1 can be collected with high quality.

As a method for collecting sound from a sound source, for example, a method using noise cancellation for removing sounds other than the target sound is conceivable. For example, Patent Literature 1 describes a method of noise cancellation by beam forming technology using one microphone array. In this method, an image processing device separate from the microphone array is used to detect the placement of a person to be sound-collected, and the noise direction is set based on the placement of the sound-collection target. Noise is canceled by subtracting the sound in the noise direction from the sound in the direction in which the sound collection target exists.

However, for example, when the target person turns his/her back to the microphone array, the person's voice is collected from the opposite side of the speaking direction. difficult to make a sound Also, depending on the positional relationship between the sound collection target and the noise source, the noise may be louder than the target sound. In this case, since the utterance information, which is the target sound, is extracted from the noise information, there is a possibility that the quality of the speech will be degraded.

In this embodiment, the position Q and the speech direction 2 of the sound source (speaker 1) to be collected is detected as the sound source information. Based on this sound source information, the sound 5 of the speaker 1 is collected by controlling a plurality of sound collectors capable of setting the sound collection direction 3 in an arbitrary direction. This makes it possible to individually and simultaneously collect sounds 5 emitted from a plurality of speakers 1 facing various directions.
Even if a plurality of speakers 1 speak at the same time, it is possible to collect the voice data 6 of each speaker 1 as separate objects for the number of utterances. This facilitates handling of the audio data 6 .

The method of selecting the target microphone 25 from a plurality of BF microphones M and setting the sound collection direction 3 aims at creating a situation in which the voice 5 of the speaker 1 can be collected with good sound quality. It can be said that this is a method of improving the sound quality of the original data before canceling the noise.
Thus, since the sound collection method performed by the sound collection system 100 is not noise removal, it is possible to provide audio data 6 that can be heard clearly when reproduced.

<Other embodiments>
The present technology is not limited to the embodiments described above, and various other embodiments can be implemented.

A method of setting one beam 7 for each BF microphone M to collect sound has been described above. It is not limited to this, and it is also possible to set a plurality of beams 7 (sound collection direction 3) for one BF microphone M, for example. As a result, even when the number of speakers 1 is greater than the number of BF microphones M, for example, it is possible to achieve high-quality sound collection for each speaker 1 .

In the configuration described with reference to FIG. 1, the sound collection processing unit 23 executes beam forming processing. For example, each BF microphone M may be configured to be able to perform beam forming processing. In this case, each BF microphone M performs beamforming processing for collecting sound waves in the sound collection direction 3 specified by the sound collection direction signal, and each BF microphone M outputs sound data 6 in the sound collection direction 3. be done. Even with such a configuration, it is possible to collect the speech 5 of the speaker 1 with high quality.

A unidirectional microphone or the like may be used instead of the BF microphone M as a sound collecting device capable of setting the sound collecting direction 3 . In this case, for example, many unidirectional microphones are arranged around speaker 1 . A unidirectional microphone having a sound collecting direction 3 that matches the speaking direction 2 of the speaker 1 is selected and used as the target microphone 25 . Even with such a configuration, it is possible to collect the speech 5 of the speaker 1 with high quality.

A case has been described above in which the computer (controller) of the sound collection system executes the information processing method according to the present technology. However, the computer of the sound collection system and another computer that can communicate via a network or the like may execute the information processing method and the program according to the present technology.

That is, the information processing method and program according to the present technology can be executed not only in a computer system configured by a single computer, but also in a computer system in which a plurality of computers work together. In the present disclosure, a system means a set of multiple components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a single device housing a plurality of modules within a single housing, are both systems.

The computer system executes the information processing method and program according to the present technology, for example, when the process of acquiring sound source information and the process of selecting a target microphone are executed by a single computer, and each process is executed by a different computer. includes both cases where Execution of each process by a predetermined computer includes causing another computer to execute part or all of the process and obtaining the result.

That is, the information processing method and program according to the present technology can also be applied to a configuration of cloud computing in which a plurality of devices share and jointly process one function via a network.

It is also possible to combine at least two characteristic portions among the characteristic portions according to the present technology described above. That is, various characteristic portions described in each embodiment may be combined arbitrarily without distinguishing between each embodiment. Moreover, the various effects described above are only examples and are not limited, and other effects may be exhibited.

In the present disclosure, the terms “same”, “equal”, “orthogonal”, etc. are concepts including “substantially the same”, “substantially equal”, “substantially orthogonal”, and the like. For example, states included in a predetermined range (for example, a range of ±10%) based on "exactly the same", "exactly equal", "perfectly orthogonal", etc. are also included.

Note that the present technology can also adopt the following configuration.
(1) an information acquisition unit that acquires sound source information indicating the position of a sound source and the direction in which the sound source emits sound;
A sound collection control unit that selects, based on the sound source information, at least one target device used for collecting sound emitted by the sound source from a plurality of sound collection devices arranged around the sound source and capable of setting a sound collection direction. An information processing device comprising and.
(2) The information processing device according to (1),
The information processing apparatus, wherein the sound collection control unit sets a sound collection direction of the target device based on the sound source information.
(3) The information processing device according to (2),
The information processing apparatus, wherein the sound collection control unit sets a direction from the target device toward the sound source as a sound collection direction of the target device.
(4) The information processing device according to any one of (1) to (3),
The sound collection control unit determines the sound collection device capable of collecting the direct sound emitted by the sound source based on the direction in which the sound source emits sound, and selects the sound collection device as the target device. .
(5) The information processing device according to (4),
The plurality of sound collecting devices are configured to be able to set the sound collecting direction in an allocation range allocated according to each arrangement,
The information processing device, wherein the sound collection control unit selects, as the target device, the sound collection device whose direction in which the sound source emits sound corresponds to the center direction of the allocation range.
(6) The information processing device according to (5),
The sound collection control unit is capable of collecting sound along the direction in which the sound source emits sound when there is no sound collection device in which the direction in which the sound source emits sound corresponds to the center direction of the allocation range, An information processing device that selects the sound collecting device closest to the sound source as the target device.
(7) The information processing device according to any one of (1) to (6),
The information acquisition unit acquires the sound source information for each of a plurality of sound sources,
The information processing apparatus, wherein the sound collection control unit selects the target device for each of the plurality of sound sources based on the sound source information for each of the plurality of sound sources.
(8) The information processing device according to (7),
The sound collection control unit is capable of setting the sound collection direction so as to collect direct sound emitted by a sound source to be processed and not to collect direct sound emitted by a sound source different from the sound source to be processed. as the target device. Information processing device.
(9) The information processing device according to any one of (1) to (8), further comprising:
An information processing apparatus comprising a sound collection processing unit that generates sound data representing the sound emitted by the sound source based on the output of the at least one target device.
(10) The information processing device according to (9),
The plurality of sound collecting devices includes a plurality of candidate devices whose sound collecting directions are set in advance,
The sound collection control unit selects the target device from the plurality of candidate devices,
The information processing device, wherein the sound collection processing unit makes a candidate device that is not selected as the target device stand by in a sound collection state.
(11) The information processing device according to (9) or (10),
The information processing device, wherein the sound collection control unit selects a plurality of target devices from the plurality of sound collectors for the single sound source.
(12) The information processing device according to (11),
The information processing device, wherein the sound collection processing unit synthesizes data collected by the plurality of target devices to generate the sound data of the sound source.
(13) The information processing device according to any one of (9) to (12),
The sound source is a speaker,
The information processing apparatus, wherein the direction in which the sound source emits sound is the utterance direction of the speaker.
(14) The information processing device according to (13),
The information processing apparatus, wherein the information acquisition unit estimates a speech direction of the speaker by performing bone detection on the speaker based on image data of the speaker.
(15) The information processing device according to (13) or (14),
The information acquisition unit detects a gesture of the speaker,
The information processing apparatus, wherein the sound collection processing unit controls sound collection processing for collecting the voice of the speaker according to the gesture of the speaker.
(16) The information processing device according to (15),
The sound collection processing unit preferentially executes the sound collection processing for the speaker when a gesture of the speaker raising a hand is detected, and a gesture of the speaker covering the mouth with a hand is detected. information processing device that stops the sound collection process for the speaker if the
(17) The information processing device according to any one of (13) to (16),
The information processing device, wherein the sound collection processing unit separates the speech of the speaker and the gesture sound of the speaker from the data collected by the target device.
(18) The information processing device according to any one of (1) to (17),
The sound collecting device is a microphone array in which a plurality of microphones are arranged,
The information processing apparatus, wherein the sound collection direction is a beam direction set in beamforming processing for the microphone array.
(19) obtaining sound source information indicating the position of a sound source and the direction in which the sound source emits sound;
selecting at least one target device used for collecting sound emitted by the sound source from a plurality of sound collecting devices arranged around the sound source and capable of setting a sound collection direction, based on the sound source information. Information processing methods performed by
(20) obtaining sound source information indicating the position of a sound source and the direction in which the sound source emits sound;
selecting, based on the sound source information, at least one target device used to collect the sound emitted by the sound source from among a plurality of sound collecting devices arranged around the sound source and capable of setting a sound collection direction; A program that you want the system to run.

M, M1 to M4... BF microphone 1, 1A to 1D... Speaker 2... Speech direction 3... Sound collection direction 5... Sound 10... Detection camera 11... Storage unit 12... Control program 16... Microphone 20... Controller 21... Image processing Unit 22 Sound collection control unit 23 Sound collection processing unit 25 Target microphone 26 Candidate microphone 35 Sound source separation unit 41 Allocation range 100 Sound collection system

Claims (20)

an information acquisition unit that acquires sound source information indicating the position of a sound source and the direction in which the sound source emits sound;
A sound collection control unit that selects, based on the sound source information, at least one target device used for collecting sound emitted by the sound source from a plurality of sound collection devices arranged around the sound source and capable of setting a sound collection direction. An information processing device comprising and. The information processing device according to claim 1,
The information processing apparatus, wherein the sound collection control unit sets a sound collection direction of the target device based on the sound source information. The information processing device according to claim 2,
The information processing apparatus, wherein the sound collection control unit sets a direction from the target device toward the sound source as a sound collection direction of the target device. The information processing device according to claim 1,
The sound collection control unit determines the sound collection device capable of collecting the direct sound emitted by the sound source based on the direction in which the sound source emits sound, and selects the sound collection device as the target device. . The information processing device according to claim 4,
The plurality of sound collecting devices are configured to be able to set the sound collecting direction in an allocation range allocated according to each arrangement,
The information processing device, wherein the sound collection control unit selects, as the target device, the sound collection device whose direction in which the sound source emits sound corresponds to the center direction of the allocation range. The information processing device according to claim 5,
The sound collection control unit is capable of collecting sound along the direction in which the sound source emits sound when there is no sound collection device in which the direction in which the sound source emits sound corresponds to the center direction of the allocation range, An information processing device that selects the sound collecting device closest to the sound source as the target device. The information processing device according to claim 1,
The information acquisition unit acquires the sound source information for each of a plurality of sound sources,
The information processing apparatus, wherein the sound collection control unit selects the target device for each of the plurality of sound sources based on the sound source information for each of the plurality of sound sources. The information processing device according to claim 7,
The sound collection control unit is capable of setting the sound collection direction so as to collect direct sound emitted by a sound source to be processed and not to collect direct sound emitted by a sound source different from the sound source to be processed. as the target device. Information processing device. The information processing apparatus according to claim 1, further comprising:
An information processing apparatus comprising a sound collection processing unit that generates sound data representing the sound emitted by the sound source based on the output of the at least one target device. The information processing device according to claim 9,
The plurality of sound collecting devices includes a plurality of candidate devices whose sound collecting directions are set in advance,
The sound collection control unit selects the target device from the plurality of candidate devices,
The information processing device, wherein the sound collection processing unit makes a candidate device that is not selected as the target device stand by in a sound collection state. The information processing device according to claim 9,
The information processing device, wherein the sound collection control unit selects a plurality of target devices from the plurality of sound collectors for the single sound source. The information processing device according to claim 11,
The information processing device, wherein the sound collection processing unit synthesizes data collected by the plurality of target devices to generate the sound data of the sound source. The information processing device according to claim 9,
The sound source is a speaker,
The information processing apparatus, wherein the direction in which the sound source emits sound is the utterance direction of the speaker. The information processing device according to claim 13,
The information processing apparatus, wherein the information acquisition unit estimates a speech direction of the speaker by performing bone detection on the speaker based on image data of the speaker. The information processing device according to claim 13,
The information acquisition unit detects a gesture of the speaker,
The information processing apparatus, wherein the sound collection processing unit controls sound collection processing for collecting the voice of the speaker according to the gesture of the speaker. The information processing device according to claim 15,
The sound collection processing unit preferentially executes the sound collection processing for the speaker when a gesture of the speaker raising a hand is detected, and a gesture of the speaker covering the mouth with a hand is detected. information processing device that stops the sound collection process for the speaker if the The information processing device according to claim 13,
The information processing device, wherein the sound collection processing unit separates the speech of the speaker and the gesture sound of the speaker from the data collected by the target device. The information processing device according to claim 1,
The sound collecting device is a microphone array in which a plurality of microphones are arranged,
The information processing apparatus, wherein the sound collection direction is a beam direction set by beamforming processing for the microphone array. Acquiring sound source information indicating the position of a sound source and the direction in which the sound source emits sound;
selecting at least one target device used for collecting sound emitted by the sound source from a plurality of sound collecting devices arranged around the sound source and capable of setting a sound collection direction, based on the sound source information. Information processing methods performed by obtaining sound source information indicating the position of a sound source and the direction in which the sound source emits sound;
selecting, based on the sound source information, at least one target device used to collect the sound emitted by the sound source from among a plurality of sound collecting devices arranged around the sound source and capable of setting a sound collection direction; A program that you want the system to run.

Images