JP7111202B2 - SOUND COLLECTION CONTROL SYSTEM AND CONTROL METHOD OF SOUND COLLECTION CONTROL SYSTEM - Google Patents

Description

The present invention relates to a sound collection control system and a control method for the sound collection control system.

Patent Document 1 below describes a camera that captures an image of a target space, a microphone array that collects sound using a plurality of microphones, and a sound source analysis unit that calculates the sound source position based on the collected sound data. , an image display processing unit that displays image data including an image of the captured target space and a sound source position display that indicates the position of the sound source on the display; A sound collection system is disclosed that includes a user input device that accepts designation of a desired location.

JP 2016-146547 A

In the conventional sound collection system, the sound source analysis unit calculates the sound source position based only on the collected sound data, so very high directivity was required.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and an object of the present invention is to realize a configuration for specifying the position of a sound source without requiring high directivity for the sound pickup function.

A sound collection control system according to the present disclosure includes a sound collection unit that receives sound input and generates a plurality of sound signals, the plurality of sound signals, and images of participants in which a plurality of participants are displayed. and using comparative information between a plurality of said audio signals to identify the location of a first sound source, and performing facial recognition on an image of said participant to a range including said location of said first sound source. a position information generation unit that identifies the position of the second sound source by using the above-described position information, and generates position information representing the position of the second sound source; and an image generation unit that generates an image of the speaker using the position information. , a display processing unit for displaying at least one image regarding the speaker on a display device; and a selection receiving unit for receiving selection of one image from the at least one image regarding the speaker.

A control method for a sound collection control system according to the present disclosure receives an audio input, generates a plurality of audio signals, and obtains the plurality of audio signals and an image of a participant in which a plurality of participants are displayed. and identifying the location of a first sound source using comparative information between a plurality of said audio signals, and performing facial recognition on an image of said participant to a range including said location of said first sound source. to identify the position of a second sound source, generate position information representing the position of the second sound source, generate an image of the speaker using the position information, and generate at least one image of the speaker The selection of one image is received from among the images relating to the at least one speaker, which are displayed on a display device.

FIG. 1 is a block diagram of a sound collection control system according to the first embodiment. FIG. 2 is a block diagram of the photographing and sound collecting apparatus according to the first embodiment. FIG. 3 is a schematic diagram showing an example of the positional relationship between the photographing/sound collecting device and the participants according to the first embodiment. FIG. 4 is a schematic diagram showing an example of face recognition using participant images according to the first embodiment. FIG. 5 is a schematic diagram showing an example of a clipped image generated from a participant image according to the first embodiment. FIG. 6 is a schematic diagram showing an example of a clipped image generated from a participant image according to the first embodiment. FIG. 7 is a block diagram of the operating device according to the first embodiment. FIG. 8 is a block diagram showing another example of the operating device according to the first embodiment. FIG. 9 is a schematic diagram showing an example of the display screen of the display device according to the first embodiment. FIG. 10 is a flow chart showing a control method of the photographing/sound collecting apparatus according to the first embodiment. FIG. 11 is a flow chart showing a control method for the operating device according to the first embodiment. FIG. 12 is a flow chart showing a control method of the sound collection control system according to the first embodiment.

[First embodiment]
A first embodiment will be described below with reference to the drawings.

[Sound pickup control system 100]
FIG. 1 is a block diagram of a sound collection control system 100 according to this embodiment. As shown in FIG. 1, the sound collection control system 100 includes a photographing and sound collecting device 10, an operation device 30 connected to the photographing and sound collection device 10 via a network 20, and reproduction according to instructions from the operation device 30. and a playback device 40 for performing. Note that the playback device 40 may be externally connected to the operating device 30 or may be built in the operating device 30 .

The photographing/sound collecting device 10 receives input of a speaker's voice and generates a voice signal 51 . The photographing sound collecting device 10 photographs a participant image in which a plurality of participants participating in a conference, lecture, or the like are displayed. The photographing and sound collecting device 10 identifies the range including the sound source position from the phase difference of the plurality of audio signals 51, and performs face recognition on the participant image to identify the face position of the estimated speaker, that is, the sound source position. do. The photographing/sound collecting device 10 generates an estimated speaker image 52 based on the identified face position. The photographing sound collecting device 10 transmits the estimated speaker image 52 and the audio signal 51 to the operating device 30 via the network 20 .

The operation device 30 is used by an operator who is a speaker in a conference or lecture, for example, and receives an audio signal 51 and an estimated speaker image 52 via the network 20 . The operation device 30 has an internal display device or is externally connected to the display device, and displays at least one estimated speaker image 52 received from the photographing sound pickup device 10 on the display device. When the operator selects one of at least one estimated speaker image 52 displayed on the display device, the operating device 30 generates a directional control signal 53 for the selected estimated speaker image 52. , to the photographing and sound collecting device 10 via the network 20 . The directivity control signal 53 is a signal for controlling the output of the sound signal 51 by the photographing sound collecting device 10 based on the sound source position information associated with the estimated speaker image 52 .

Upon receiving the directivity control signal 53, the photographing/sound collecting device 10 emphasizes the output of the sound signal 51 at the sound source position associated with the estimated speaker image 52 over the other sound signals 51 based on the directivity control signal 53. and transmits it to the operation device 30 .

The operating device 30 receives an audio signal 51 in which the voice around the speaker is emphasized, and instructs a reproducing device 40 such as a speaker to reproduce the audio signal 51 .

With such a configuration, the display device displays the estimated speaker image 52 in which the face of the estimated speaker is enlarged, so that the operator can easily recognize the speaker.

In addition, when the venue is large and there are many participants, in order to specify the position of the speaker's face using only the phase difference of the audio signal 51, the sound collection function included in the photographing sound collection device 10 requires very high directivity. However, the photographing and sound collecting device 10 generates sound source position information using face recognition technology in combination, so that even if the directivity of the sound collecting function is not so high, the position of the speaker's face can be specified. becomes possible.

More specific configurations of the photographing/sound collecting device 10 and the operation device 30 will be described below.

[Photographing sound collection device 10]
FIG. 2 is a block diagram of the photographing and sound collecting device 10 according to this embodiment. As shown in FIG. 2, the photographing/sound-collecting device 10 includes a sound-collecting unit 11 that receives voice input and generates a plurality of sound signals 51, and a photographing unit that captures participant images in which a plurality of participants are displayed. 12, a sound source position information generating unit 13 that generates sound source position information for specifying the sound source position using the phase difference of the plurality of audio signals 51 and face recognition for the participant image, and an estimated speech using the sound source position information. and an estimated speaker image generation unit 14 that generates an estimated speaker image 52 related to the person and transmits it to the operation device 30 .

Further, the photographing/sound collecting device 10 acquires the directivity control signal 53 from the operation device input unit 15 that receives the directivity control signal 53 transmitted from the operation device 30, and obtains the directivity control signal 53 from the operation device input unit 15. and a directivity calculation unit 16 for controlling and outputting the audio signal 51 based on the directivity control signal 53 .

The sound source position information generator 13 , the estimated speaker image generator 14 , the operating device input section 15 and the directivity calculator 16 can be configured by one controller 17 . The control unit 17 includes a CPU and a memory, controls the entire shooting and sound collecting device 10 based on a program stored in the storage unit 18 included in the shooting and sound collecting device 10, and performs an audio signal generation step S101 described later, a participant An image capturing step S102, a sound source position information generating step S103, an estimated speaker image generating step S104, an estimated speaker image transmitting step S105, a directivity control signal receiving step S106, a directivity calculating step S107, and the like are executed. The program may be installed from various recording media regardless of the type, such as optical or magnetic, and may be downloaded via the Internet.

In the present embodiment, the photographing/sound collecting device 100 includes the storage unit 18 , but the storage unit 18 is not an essential component, and the control unit 17 is a program connected to the photographing/sound collecting device 100 . A recording medium or the like may be used to execute various control steps.

FIG. 3 is a schematic diagram showing an example of the arrangement relationship between the photographing/sound collecting device 10 and the participants according to the present embodiment. As shown in FIG. 3, the photographing/sound-collecting device 10 is placed on the ceiling in front of the venue, and the photographing unit 12 included in the photographing/sound-collecting device 10 photographs the entire front of the venue. Note that the installation location of the photographing and sound collecting device 10 is not limited to the ceiling of the hall, and the photographing and sound collecting device 10 may be placed in front of the hall.

[Sound pickup unit 11]
In this embodiment, the sound pickup unit 11 is configured by an array microphone including a plurality of microphones. The sound pickup unit 11 picks up the sound of the venue where the photographing sound pickup device 10 is arranged. The sound pickup unit 11 may be arranged in the same housing including the imaging unit 12 shown in FIG. 3, or may be arranged in another housing installed on the ceiling in front of the hall.

A plurality of microphones included in the array microphone, which is the sound pickup unit 11, each has a sound pickup position in charge, picks up sounds around the sound pickup position in charge, and outputs a plurality of sound signals 51 corresponding to the number of microphones. to generate The sound pickup unit 11 includes, for example, an amplifier, an A/D converter, and an encoding unit. An amplifier amplifies audio data picked up by the microphone. The A/D converter converts audio data output from the amplifier into digital audio data. The encoder encodes digital audio data to generate an audio signal.

The sound pickup unit 11 transmits the plurality of sound signals 51 to the sound source position information generation unit 13 and the directivity calculation unit 16, as shown in FIG.

[Photographing unit 12]
The photographing unit 12 has at least one camera and photographs a participant image 60 in which a plurality of participants are displayed. In this embodiment, the photographing unit 12 has one camera, and this camera photographs an image of all the participants in the venue from the front.

The photographing unit 12 transmits the photographed participant image 60 to the sound source position information generating unit 13 and the estimated speaker image generating unit 14 .

[Sound source position information generator 13]
As described above, the sound pickup unit 11 generates sound signals 51 picked up by a plurality of microphones according to the number of microphones, and transmits the sound signals 51 to the sound source position information generation unit 13 . The sound source position information generating unit 13 that has received the plurality of sound signals 51 generates a first range 71, a second range 71, and a second Identify the range 72 of .

In the present embodiment, the sound source position information generation unit 13 has described a configuration in which the sound source position is specified from the phase difference as the comparison information of the plurality of audio signals 51. However, the phase difference of the plurality of audio signals 51 Instead, a configuration may be adopted in which the sound source position is specified from the power ratio of the plurality of audio signals 51 .

In this embodiment, the sound source position information generator 13 performs frequency analysis on the audio signal 51 to distinguish between mere noise and human voice, and selects human voice from among a plurality of sound source positions. Acquire the position information where the voice is emitted as the sound source position information. As a specific example, the sound source position information generation unit 13 converts the audio signal 51 from a time domain with signal amplitude on the vertical axis and time on the horizontal axis to a frequency domain with signal amplitude on the vertical axis and frequency on the horizontal axis. A speech signal 51 included in a predetermined range (for example, 0.2 kHz to 4 kHz) appropriate for the frequency of human speech is determined to be human speech, and a speech signal 51 not included in the predetermined range is regarded as noise. judge there is. The sound source position information generating unit 13 specifies a range that includes the sound source position only for the sound signal 51 that is included in a predetermined range as the frequency of the person's voice.

Furthermore, the sound source position information generation unit 13 receives participant images from the imaging unit 12 . The sound source position information generation unit 13 performs face recognition on the participant image acquired from the imaging unit 12 within a range that includes the sound source position specified from the phase difference of the audio signal 51 described above. The position of the person's face, that is, the position of the sound source is specified. The information specifying the sound source position is the sound source position information.

FIG. 4 is a schematic diagram showing an example of face recognition using participant images in this embodiment. As shown in FIG. 4, the participant image 60 includes an image of all participants present at the venue. Here, it is assumed that the first participant 61 on the rear left side of the participant image 60 and the second participant 62 on the front right side utter some sound. Based on the sound signal 51 from the sound pickup unit 11 that picked up this sound, the sound source position information generation unit 13 generates a first range 71 that includes the first sound source position and a second range 71 that includes the second sound source position. Identify the range 72 of . A first range 71 includes a first participant 61 and a second range 72 includes a second participant 62 .

The sound source position information generator 13 identifies the face position 81 of the first participant 61 as the first sound source position by performing face recognition in the first range 71 . Further, the sound source position information generation unit 13 identifies the face position 82 of the second participant 62 as the second sound source position by performing face recognition in the second range 72 .

In this embodiment, the sound source position information generation unit 13 generates an estimated speaker image using the face position 81 of the first participant 61 and the face position 82 of the second participant 62 as sound source position information. Send to unit 14 . At that time, the sound source position information generation unit 13 associates the first range 71 with the face position 81 of the first participant 61, and associates the second range 72 with the face position 82 of the second participant 62. are associated with each other and transmitted to the estimated speaker image generation unit 14 .

[Estimated speaker image generator 14]
The estimated speaker image generating unit 14 receives the sound source position information transmitted from the sound source position information generating unit 13 and the participant image 60 transmitted from the imaging unit 12, and generates an estimated speaker image 52 related to the estimated speaker. do.

In this embodiment, the sound source position information generation unit 13 generates information on the face position 81 of the first participant 61 and the face position 82 of the second participant 62 as shown in FIG. is receiving The estimated speaker image generation unit 14 generates a clipped image from the participant image 60 transmitted from the imaging unit 12 based on this face position information.

5 and 6 are schematic diagrams showing examples of clipped images generated from the participant images 60 in this embodiment. In this embodiment, the estimated speaker image generation unit 14 generates the first participant's face position 81 as shown in FIG. A first clipped image 91 in which the face of 61 is enlarged is generated. In addition, the estimated speaker image generation unit 14, based on the position 82 of the face of the second participant 62 received from the sound source position information generation unit 13, the face of the second participant 62 as shown in FIG. An enlarged second cropped image 92 is generated.

The estimated speaker image generation unit 14 transmits the first clipped image 91 and the second clipped image 92 to the operation device 30 as the estimated speaker image 52 . The first clipped image 91 and the second clipped image 92 are the sound source position information from the sound source position information generation unit 13, that is, the face position 81 of the first participant 61 and the face position 81 of the second participant 62. It is generated based on the information of the position 82 of the face. Therefore, the estimated speaker image generation unit 14 transmits the sound source position information and the estimated speaker image 52 to the operation device 30 in a state of being associated with each other.

Furthermore, in the present embodiment, the estimated speaker image generator 14 associates position information corresponding to the estimated speaker image 52 in the participant image 60 with the estimated speaker image 52 and transmits the information to the operation device 30 . With such a configuration, the participant image 60 and the estimated speaker image 52 can be displayed in association with each other on the operation device 30, which will be described later.

In this embodiment, the estimated speaker image generation unit 14 generates a first clipped image 91 and a second clipped image 92 as the estimated speaker image 52 and transmits them to the operation device 30. As described above, another image may be sent as the estimated speaker image 52 . For example, the photographing unit 12 described above has a plurality of cameras, the first camera photographs the participant image 60, and based on the sound source position information generated by the sound source position information generating unit 13, the second camera detects the position of the sound source. An image in the direction of is captured, and the image in the direction of the sound source position is used as the estimated speaker image 52 and acquired by the estimated speaker image generation unit 14 . In the case of this configuration, in order to photograph a plurality of speakers and generate a plurality of estimated speaker images 52, it is necessary to provide a plurality of second cameras according to the number of speakers.

In this embodiment, the estimated speaker image generation unit 14 generates a plurality of estimated speaker images related to a plurality of estimated speakers and transmits them to the operation device 30. A configuration may be adopted in which the person image generation unit 14 generates an estimated speaker image for one estimated speaker and transmits the image to the operation device 30 . However, by configuring the estimated speaker image generation unit 14 to generate a plurality of estimated speaker images and transmit them to the operation device 30, a plurality of estimated speaker images 52 can be displayed side by side on the display device 33, which will be described later. It can be configured to display. As a result, there is an advantage that the operator can operate the operation device 30 while comparing a plurality of estimated speaker images 52 .

[Device input unit 15 for operation]
The operation device input unit 15 receives the directivity control signal transmitted from the operation device 30 and transmits it to the directivity calculation unit 16 . A directivity control signal is information for controlling the output of the sound signal 51 from the sound pickup unit 11 based on sound source position information associated with the estimated speaker image 52 .

In this embodiment, the directivity control signal includes sound source location information associated with the first clipped image 91 or the second clipped image 92, that is, the face position 81 of the first participant 61, or the 2 participants 62 face positions 82 are included.

The operation device input unit 15 transmits the directivity control signal containing the sound source position information to the directivity calculation unit 16 .

[Directivity calculator 16]
The directivity calculation unit 16 acquires the directivity control signal 53 from the operating device input unit 15 and acquires the audio signal 51 from the sound pickup unit 11 . Based on this directivity control signal 53, the directivity calculation unit 16 controls and outputs the audio signal 51 from the sound pickup unit 11. FIG.

Although details will be described later, when the operator who operates the operation device 30 selects the first clipped image 91 in which the face of the first participant 61 is displayed, the image is associated with the first clipped image 91 . The directivity control signal 53 includes first sound source location information, ie, information about the face location 81 of the first participant 61 . Based on this directivity control signal 53, the directivity calculation unit 16 selects the sound signal 51 from the face position 81 of the first participant 61 from among the plurality of sound signals 51 received from the sound pickup unit 11, The audio signal 51 is emphasized more than other audio signals 51 and transmitted to the operating device 30 .

[Device for operation 30]
FIG. 7 is a block diagram of the operating device 30 according to this embodiment. As shown in FIG. 7, the operation device 30 includes a communication unit 31 that receives at least one estimated speaker image 52 generated by the photographing sound pickup device 10, and a display device 33 that displays at least one estimated speaker image 52. a selection reception unit 34 for receiving selection of one estimated speaker image 52 from at least one estimated speaker image 52; and a directivity control signal for the selected estimated speaker image 52. 53, and a reproduction processing unit 36 that receives the audio signal from the photographing/sound pickup device 10 and instructs the reproduction device 40 to reproduce the audio signal 51. FIG.

The display processing unit 32, the directivity control signal generation unit 35, and the reproduction processing unit 36 can also be configured by one control unit 38. FIG. The control unit 38 includes a CPU and a memory, controls the operation device 30 as a whole based on a program stored in the storage unit 39 included in the operation device 30, and performs an estimated speaker image receiving step S201 and display processing, which will be described later. Step S202, selection reception step S203, directivity control signal generation step S204, audio signal reception step S205, reproduction processing step S206, etc. are executed. The program may be installed from various recording media regardless of the type, such as optical or magnetic, and may be downloaded via the Internet.

The operating device 30 is used by an operator who gives a lecture, for example, in front of the venue.

In this embodiment, the operation device 30 includes the touch panel 37 , and the touch panel 37 includes the display device 33 and the selection reception section 34 .

Note that as shown in FIG. 8, which is a block diagram of the operating device 30 showing another embodiment, the operating device 30 does not include the display device 33, and the display processing section 32 is externally connected to the operating device 30. Alternatively, the display device 33 may be configured to instruct the display of the estimated speaker image 52 .

[Communication section 31]
The communication unit 31 is compatible with wireless LAN, wired LAN, WAN, infrared rays, radio waves, Bluetooth (registered trademark), etc., and performs signal transmission with the photographing and sound collecting device 10 via the network 20 .

In this embodiment, the communication unit 31 receives the first cut-out image 91 and the second cut-out image 92 generated by the photographing sound pickup device 10 as the estimated speaker image 52 . The communication unit 31 transmits the first clipped image 91 and the second clipped image 92 to the display processing unit 32 . Note that the first clipped image 91 is associated with information about the face position 81 of the first participant 61 as the first sound source position information, and the second clipped image 92 is associated with the second clipped image 92 . Information about the face position 82 of the second participant 62 is associated as the sound source position information.

In addition, the communication unit 31 transmits the directivity control signal 53 generated by the directivity control signal generation unit 35 to the photographing/sound pickup device 10 .

[Display processing unit 32]
The display processing unit 32 causes the display device 33 to display at least one estimated speaker image 52 . That is, when a plurality of estimated speaker images 52 have been acquired, the plurality of estimated speaker images 52 are displayed on the display device 33, and when a single estimated speaker image 52 has been acquired, that The only estimated speaker image 52 is displayed on the display device 33 .

In the present embodiment, the first clipped image 91 shown in FIG. 5 and the second clipped image 92 shown in FIG. 6 are displayed on the display device 33 . FIG. 9 is a schematic diagram showing an example of the display screen of the display device 33 in this embodiment. As shown in FIG. 9, the display processing unit 32 causes the display device 33 to display the first clipped image 91 and the second clipped image 92 side by side. Note that the number of clipped images displayed on the display device 33 may be configured to match the number of clipped images received from the communication unit 31 .

In this way, by displaying a plurality of estimated speaker images 52 on the display device 33, the operator can operate the operating device 30 while comparing the plurality of estimated speaker images 52. , you can select the speaker you want to hear with particular emphasis.

By displaying the plurality of estimated speaker images 52, the operator can operate the operation device 30 while comparing the plurality of estimated speaker images 52. The advantage of this is that the sound source position information generation unit 13 can It can be obtained even when the sound source position information is generated without using face recognition. Note that when the sound source position information generation unit 13 generates sound source position information without using face recognition, the estimated speaker image 52 is the first range 71 including the sound source position shown in FIG. Based on the second range 72, for example, clipped images of the first range 71 and the second range 72 are generated.

Furthermore, in this embodiment, the display processing unit 32 is configured to cause the display device 33 to display the participant image 60 together. The display processing unit 32 causes the display device 33 to display the participant images 60 together, so that the operator can confirm the state of the entire venue together.

Furthermore, in the present embodiment, the operation device 30 receives the position information corresponding to the estimated speaker image 52 in the participant image 60 from the photographing sound pickup device 10 in a state associated with the estimated speaker image 52. is doing. Therefore, the display processing unit 32 can cause the display device 33 to display the participant image 60 and the estimated speaker image 52 in association with each other.

Specifically, as shown in FIG. 9, the display processing unit 32 causes the display device 33 to display a display frame 81A at the position of the face of the first participant 61 in the participant image 60, A display frame 82A is displayed at the position of the face of the participant 62. FIG. Furthermore, the display processing unit 32 displays the display frames 81A and 82A and at least a part of the estimated speaker image 52 corresponding to the display frames 81A and 82A in the same color on the display device 33. Let As a specific example, the display frame 81A of the face position 81 of the first participant 61 is displayed in red, the frame of the first clipped image 91 is displayed in red, and the face of the second participant 62 is displayed in red. A configuration may be adopted in which the display frame 82A of the position 82 is displayed in green and the frame of the second clipped image 92 is displayed in green. With such a configuration, the operator can easily determine where the first participant 61 and the second participant 62 displayed as clipped images are positioned in the participant image 60. I can understand.

In addition, the coloring portion of the first cutout image 91 and the second cutout image 92 is not limited to the frame. Any configuration may be used as long as the operator can associate the first clipped image 91, the second clipped image 92, and the display frames 81A and 81B included in the participant image 60 with each other.

With such a configuration, the display device 33 displays the first clipped image 91 in which the face of the first participant 61 who is the estimated speaker is enlarged, and the face of the second participant 62 who is the estimated speaker. Since the enlarged second clipped image 92 is displayed, the operator can easily recognize the speaker.

In addition, when the venue is large and there are many participants, very high directivity is required to specify the position of the speaker's face using only the phase difference of the audio signal 51 of the sound pickup unit 11. Desired. However, as shown in the present embodiment, the sound source position information generation unit 13 uses face recognition technology in combination, so that even if the directivity of the sound pickup unit 11 is not so high, the sound source position information can be identified by the phase difference of the audio signal. It is possible to specify the sound source position by face recognition from within the range including the sound source position obtained. Moreover, since the range including the sound source position can be narrowed down using the phase difference of the audio signal 51, the face recognition range can be narrowed down, and the sound source position can be specified efficiently.

In this embodiment, the communication unit 31 of the operation device 30 receives the first clipped image 91 and the second clipped image 92 as the estimated speaker image 52, and the first clipped image 91 and the second clipped image 92 are received. 2 has been described above. However, when the communication unit 31 of the operation device 30 receives, as the estimated speaker image 52, images in the direction of each sound source position captured by a plurality of cameras, the image in the direction of each sound source position is A configuration may be adopted in which the estimated speaker image 52 is displayed on the display device 33 .

[Display device 33]
The display device 33 is a display device built in or externally connected to the operating device 30, and displays the estimated speaker image 52 and the like based on instructions from the display processing section 32. FIG.

As an example in which the display device 33 is externally connected to the operating device 30 , a general liquid crystal display, EL display, plasma display, projector, or the like may be used as the display device 33 . Alternatively, when the operator uses a notebook computer for the lecture, the notebook computer is connected to the operation device 30, and the notebook computer responds to instructions from the display processing unit 32 included in the operation device 30. The estimated speaker image 52 may be displayed on the included display device.

[Selection accepting unit 34]
The selection accepting unit 34 accepts selection of one estimated speaker image 52 out of at least one estimated speaker image 52 . That is, when the selection accepting unit 34 has obtained a plurality of estimated speaker images 52, it accepts selection of one estimated speaker image 52 from the plurality of estimated speaker images 52 by the operation of the operator. When the selection receiving unit 34 has acquired the only estimated speaker image 52, the operator selects whether or not to emphasize the voice of the speaker included in this unique estimated speaker image 52. , the selection accepting unit 34 accepts the selection by the operator.

In the present embodiment, as shown in FIG. 7, the operation device 30 includes a touch panel 37, and a touch sensor included in this touch panel 37 functions as the selection reception section 34. FIG.

Alternatively, as shown in FIG. 8, the operation device 30 may not include the touch panel 37 and the display device 33, but may have a selection receiving section 34 as an input device separately. This input device may be a general input device such as a keyboard or mouse. Alternatively, the selection accepting unit 34 may be a gesture recognition device that includes a camera and an image recognition device and accepts the operator's selection by recognizing the operator's gesture.

When the selection reception unit 34 receives the operator's selection, the selection reception unit 34 transmits the selection result to the directivity control signal generation unit 35 . In the present embodiment, it is assumed that the operator has determined that he or she wishes to emphasize the voice of the first participant 61 and selects the first clipped image 91 . The selection reception unit 34 selects the sound source position information associated with the first clipped image 91, that is, the face position 81 of the first participant 61 shown in FIG. 4 as the operator's selection result. The result is transmitted to the directivity control signal generator 35 .

[Directivity control signal generator 35]
The directivity control signal generator 35 generates a directivity control signal 53 for the selected estimated speaker image 52 .

In the present embodiment, the directivity control signal generator 35 receives information about the face position 81 of the first participant 61 as sound source position information associated with the first clipped image 91 from the selection receiver 34 . Therefore, the directivity control signal 53 is generated to instruct to emphasize the output of the audio signal 51 generated from the position 81 of the face of the first participant 61 .

The directivity control signal generation unit 35 transmits the directivity control signal 53 to the photographing sound pickup device 10 via the communication unit 31 .

[Reproduction processing unit 36]
The reproduction processing unit 36 receives the audio signal 51 from the photographing/sound pickup device 10 via the communication unit 31 and instructs the reproduction device 40 to reproduce the audio signal 51 .

In this embodiment, the directivity control signal 53 instructing to emphasize the output of the audio signal 51 generated from the position 81 of the face of the first participant 61 is captured from the directivity control signal generation unit 35. The directivity control signal 53 is transmitted to the sound collecting device 10, and the directivity control signal 53 is received by the directivity calculating section 16 included in the photographing sound collecting device 10 via the operation device input section 15 described above.

Based on the directivity control signal 53, the directivity calculation unit 16 selects the audio signal 51 from the face position 81 of the first participant 61 from among the plurality of audio signals 51 received from the sound pickup unit 11. The audio signal 51 is emphasized more than the other audio signals 51 and transmitted to the operating device 30 .

Therefore, the reproduction processing unit 36 instructs the reproduction device 40 to reproduce the audio signal 51 in which the voice from the face position 81 of the first participant 61 is emphasized.

As a result, the operator can emphasize and listen to the voice from the first participant 61 selected by him/herself.

As described above, the operator uses the sound pickup control system 100 including the photographing sound pickup device 10 and the operating device 30 according to the present embodiment to obtain the estimated speaker image 52 in which the face of the participant is enlarged. can be used to easily select a speaker to be heard with emphasis.

In addition, when the venue is large and there are many participants, extremely high directivity is required in order to specify the position of the speaker's face using only the audio signal 51 of the sound pickup unit 11 . However, as shown in the present embodiment, the sound source position information generating unit 13 also uses face recognition technology, so that even if the directivity of the sound collecting unit 11 is not so high, the phase difference of the audio signal 51 It is possible to specify the sound source position, that is, the position of the speaker's face, by face recognition from within the range that includes the specified sound source position. As a result, it becomes possible for the operator to easily select the speaker to be emphasized and listened to by using the estimated speaker image 52 in which the face of the speaker is displayed in an enlarged manner.

[Method for Controlling Shooting and Sound Collecting Device 10]
FIG. 10 is a flow chart showing a control method of the photographing/sound collecting device 10 according to this embodiment. A control method of the photographing/sound collecting apparatus 10 according to the present embodiment will be described below with reference to FIG. 10 .

[Audio signal generation step S101]
The sound pickup unit 11 of the photographing sound pickup device 10 performs an audio signal generation step S101 for generating a plurality of audio signals.

In the present embodiment, the sound pickup unit 11 is an array microphone including a plurality of microphones, and the plurality of microphones pick up sound around the sound pickup positions they are in charge of. Thereby, the sound pickup unit 11 generates a plurality of sound signals corresponding to the number of microphones.

The sound pickup unit 11 transmits a plurality of sound signals 51 to the sound source position information generation unit 13 and the directivity calculation unit 16 .

[Participant image capturing step S102]
The photographing unit 12 of the photographing/sound collecting device 10 performs a participant image photographing step S102 for photographing the participant images 60 in which a plurality of participants are displayed.

In this embodiment, the photographing unit 12 has one camera, and this camera photographs a participant image 60 of all the participants in the hall from the front.

The photographing unit 12 transmits the photographed participant image 60 to the sound source position information generating unit 13 and the estimated speaker image generating unit 14 .

Note that this S102 may be performed in parallel with the audio signal generation step S101, or may be performed after the audio signal generation step S101 and before the sound source position information generation step S103 described later.

[Sound source position information generation step S103]
The sound source position information generating unit 13 of the photographing sound collecting device 10 performs a sound source position information generating step S103 of generating sound source position information for estimating the sound source position from the phase difference of the plurality of audio signals 51 .

The sound source position information generation unit 13 uses face recognition technology for the participant image 60 acquired from the imaging unit 12 in addition to the phase differences of the plurality of audio signals 51 acquired from the sound collection unit 11 to determine the estimated speaker. Identify the position of the face as the sound source position.

In this embodiment, the sound source position information generation unit 13 identifies a first range 71 and a second range 72 containing the sound source position from the phase difference of the audio signal 51, and determines the first range 71 and the second range 72. 2, the facial position 81 of the first participant 61 and the facial position 82 of the second participant 62 are identified using face recognition technology for the participant image 60 . The position 81 of the face of the first participant 61 and the position 82 of the face of the second participant 62 are sent to the estimated speaker image generation unit 14 as first sound source position information and second sound source position information, respectively. Send. Also, at that time, the sound source position information generating unit 13 associates the first range 71 with the face position 81 of the first participant 61, and the second range 72 with the face position 81 of the second participant 62. 82 and transmitted to the estimated speaker image generator 14 .

If there is one or more estimated speakers here, the step moves to the estimated speaker image generation step S104, which will be described later, but if there is no estimated speaker, the step returns to the speech signal generation step S101 described above. .

[Estimated speaker image generation step S104]
The estimated speaker image generation unit 14 of the photographing sound pickup device 10 performs an estimated speaker image generation step S104 for generating an estimated speaker image related to the estimated speaker.

In the present embodiment, the photographing and sound collecting device 10 uses the first sound source position information and the second sound source position information obtained from the sound source position information generating unit 13 to extract the participant image 60 from the participant image 60 obtained from the photographing unit 12. Generate a clipped image of the estimated speaker. In this embodiment, a first clipped image 91 regarding the first participant 61 and a second clipped image 92 regarding the second participant 62 are generated.

The first clipped image 91 and the second clipped image 92 contain the sound source position information from the sound source position information generation unit 13, that is, the face position 81 of the first participant 61 and the face position 81 of the second participant 62, respectively. Information about the position 82 of the face is associated. Furthermore, in the present embodiment, the estimated speaker image generator 14 associates position information corresponding to the estimated speaker image 52 in the participant image 60 with the estimated speaker image 52 .

In this embodiment, in the estimated speaker image generation step S104, the estimated speaker image generation unit 14 generates the first clipped image 91 and the second clipped image 92 as the estimated speaker image 52. As explained above, based on the sound source position information from the sound source position information generating unit 13, the image capturing unit 12 captures an image in the direction of the sound source position, and the estimated speaker image generating unit 14 generates this sound source position as the estimated speaker image 52. A method of acquiring an image in the direction of .

[Estimated speaker image transmission step S105]
The estimated speaker image generation unit 14 of the photographing sound pickup device 10 carries out an estimated speaker image transmission step S<b>105 for transmitting the generated estimated speaker image 52 to the operation device 30 .

In the present embodiment, the estimated speaker image generator 14 transmits the clipped image associated with the sound source position information to the operation device 30 as the estimated speaker image 52 . Furthermore, in the present embodiment, the estimated speaker image generation unit 14 generates the estimated speaker image 52 while associating position information corresponding to the estimated speaker image 52 in the participant image 60 with the estimated speaker image 52. It transmits to the operation device 30 .

[Directivity control signal reception step S106]
The directionality control signal reception step S106 is performed after the estimated speaker image reception step S201, the display processing step S202, the selection acceptance step S203, and the directionality control signal generation step S204 in the control method of the operation device 30 described later. , are the steps performed in the photographing and sound collecting device 10. FIG.

In directivity control signal reception step S106 , the operating device input unit 15 of the photographing/sound pickup device 10 receives the directivity control signal 53 transmitted from the operating device 30 .

In this embodiment, the operating device input unit 15 outputs sound source position information associated with the first clipped image 91, that is, the sound signal 51 generated from the face position 81 of the first participant 61. A directional control signal 53 is received from the operating device 30 to instruct to emphasize the .

[Directivity calculation step S107]
The directivity calculation unit 16 of the photographing/sound collecting device 10 performs a directivity calculation step S107 of controlling and outputting the audio signal 51 based on the directivity control signal 53 .

In this embodiment, the directivity calculation unit 16 receives the directivity control signal 53 instructing to emphasize the output of the audio signal 51 generated from the face position 81 of the first participant 61. there is Based on the directivity control signal 53, the directivity calculation unit 16 selects the audio signal 51 from the face position 81 of the first participant 61 from among the plurality of audio signals 51 received from the sound pickup unit 11. The audio signal 51 is emphasized more than other audio signals 51 and transmitted to the operating device 30 .

After that, in the control method of the operation device 30 described later, an audio signal reception step S205 and a reproduction processing step S206 are performed, the voice from the participant specified by the operator is emphasized in the operation device 30, and the playback device 40 output from

[Method for controlling operation device 30]
FIG. 11 is a flow chart showing a control method for the operating device 30 according to this embodiment. A method of controlling the operating device 30 according to this embodiment will be described below with reference to FIG. 11 .

[Estimated speaker image reception step S201]
Estimated speaker image receiving step S201 includes voice signal generation step S101, participant image capturing step S102, sound source position information generation step S103, estimated speaker image generation step S104, estimation This step is performed in the operation device 30 after the speaker image transmission step S105 is performed.

The communication unit 31 of the operation device 30 carries out an estimated speaker image reception step S201 of receiving at least one estimated speaker image 52 generated by the photographing sound pickup device 10 .

In this embodiment, the communication unit 31 receives the first cutout image 91 and the second cutout image 92 generated by the photographing and sound collecting device 10 as the estimated speaker image 52 from the photographing and sound collecting device 10 . . The communication unit 31 transmits the first clipped image 91 and the second clipped image 92 to the display processing unit 32 .

[Display processing step S202]
The display processing unit 32 of the operation device 30 performs a display processing step S202 of causing the display device 33 to display at least one estimated speaker image 52 . That is, when the display processing unit 32 acquires a plurality of estimated speaker images 52, the display processing unit 32 displays the plurality of estimated speaker images 52 on the display device 33, and acquires a single estimated speaker image 52. In this case, the only estimated speaker image 52 is displayed on the display device 33 .

In this embodiment, the display processing unit 32 causes the display device 33 to display the first clipped image 91 and the second clipped image 92, and displays the participant image 60 together.

Furthermore, in the present embodiment, a display frame 81A is displayed at the position of the face of the first participant 61 in the participant image 60, and a display frame 82A is displayed at the position of the face of the second participant 62.

In this embodiment, since the operation device 30 acquires the first clipped image 91 and the second clipped image 92 as the estimated speaker image 52, these clipped images are obtained in the display processing step S202. It is displayed on the display device 33 . However, if the operating device 30 acquires, as the estimated speaker image 52, an image in the direction of each sound source position captured by a plurality of cameras as the estimated speaker image 52, in this display processing step S202, , an image in the direction of each sound source position may be displayed on the display device 33 .

[Selection acceptance step S203]
The selection reception unit 34 of the operation device 30 performs selection reception step S<b>203 for receiving selection of one estimated speaker image 52 from at least one estimated speaker image 52 . That is, when the selection reception unit 34 has obtained a plurality of estimated speaker images 52, one estimated speaker image 52 is selected from the plurality of estimated speaker images 52 by the operation of the operator. accept. When the selection receiving unit 34 has obtained the only estimated speaker image 52, the operator determines whether or not to emphasize the voice of the speaker included in the only estimated speaker image 52. , the selection accepting unit 34 accepts a selection operation based on the operator's judgment.

In the present embodiment, it is assumed that the operator has determined that he or she wishes to emphasize the voice of the first participant 61 and selects the first clipped image 91 . The selection reception unit 34, as the selection result of the operator, emphasizes the sound source position information associated with the first clipped image 91, that is, the voice output from the face position 81 of the first participant 61. It is transmitted to the directivity control signal generator 35 to instruct.

[Directivity control signal generation step S204]
The directivity control signal generator 35 of the operation device 30 performs the directivity control signal generation step S204 to generate the directivity control signal 53 for the selected estimated speaker image 52 .

In the present embodiment, the directivity control signal generator 35 receives the face position 81 of the first participant 61 as sound source position information associated with the first clipped image 91 from the selection receiver 34 . Therefore, the directivity control signal 53 is generated to instruct to emphasize the output of the audio signal 51 generated from the position 81 of the face of the first participant 61 .

The directivity control signal generation unit 35 transmits the directivity control signal 53 to the photographing sound pickup device 10 via the communication unit 31 .

[Audio signal reception step S205]
The audio signal reception step S205 is a step performed in the operation device 30 after the directionality control signal reception step S106 and the directionality calculation step S107 are performed in the above-described control method for the photographing/sound pickup device 10 .

In voice signal reception step S205, the communication unit 31 of the operation device 30 receives the voice signal 51. FIG.

In this embodiment, the communication unit 31 receives the voice signal 51 in which the voice is emphasized from the face position 81 of the first participant 61 by the directivity calculation unit 16 .

[Reproduction processing step S206]
The reproduction processing unit 36 of the operation device 30 acquires the audio signal 51 from the communication unit 31 and performs reproduction processing step S<b>206 of instructing the reproduction device 40 to reproduce the audio signal 51 .

In this embodiment, the reproduction processing unit 36 acquires the voice signal 51 in which the voice is emphasized from the position 81 of the face of the first participant 61, and instructs the reproduction device 40 to reproduce the voice signal 51. .

With such a control method, the operator can easily select the speaker whom the operator wants to emphasize and listen to by using the estimated speaker image 52 in which the participant's face is enlarged.

Further, according to the control method of the present embodiment, in the sound source position information generation step S103, the sound source position information generation unit 13 also uses face recognition technology, so that even when the directivity of the sound pickup unit 11 is not so high, However, it is possible to specify the position of the speaker's face included in the first range 71 and the second range 72 . In addition, since the sound source position information generation unit 13 can perform face recognition within the first range 71 and the second range 72 specified in advance by the phase difference of the audio signal 51, the speaker's face can be efficiently detected. can be located.

[Control Method of Sound Collection Control System 100]
FIG. 12 is a flowchart showing a control method of the sound collection control system 100 according to this embodiment. Since each step is the same as described above in the control method of the photographing/sound collecting device 10 and the control method of the operation device 30, the description thereof will be omitted.

As shown in FIG. 12, in the control method of the sound collection control system 100, steps performed by the photographing sound collection device 10 and steps performed by the operation device 30 are mixed. After the voice signal generation step S101, the participant image capturing step S102, the sound source position information generation step S103, the estimated speaker image generation step S104, and the estimated speaker image transmission step S105 are carried out by the image capturing and sound collecting device 10, the operating The device 30 performs an estimated speaker image reception step S201, a display processing step S202, a selection reception step S203, and a directivity control signal generation step S204. Thereafter, the flow returns to the photographing/sound-collecting device 10 again, and the photographing/sound-collecting device 10 executes the directionality control signal reception step S106 and the directionality calculation step S107. , and the reproduction processing step S206 are performed.

10 photographing sound collecting device 11 sound collecting unit 12 photographing unit 13 sound source position information generating unit 14 estimated speaker image generating unit 15 operation device input unit 16 directivity calculation unit 17 control unit 18 storage unit , 20 network, 30 operation device, 31 communication unit, 32 display processing unit, 33 display device, 34 selection reception unit, 35 directivity control signal generation unit, 36 reproduction processing unit, 37 touch panel, 38 control unit, 39 storage unit , 40 playback device, 51 audio signal, 52 estimated speaker image, 53 directional control signal, 60 participant image, 61 first participant, 62 second participant, 71 first range, 72 second range 81 position of face 82 position of face 81A display frame 82A display frame 91 first clipped image 92 second clipped image 100 sound collection control system S101 audio signal generation step S102 participant image Photographing step S103 Sound source position information generation step S104 Estimated speaker image generation step S105 Estimated speaker image transmission step S106 Directivity control signal reception step S107 Directivity calculation step S201 Estimated speaker image reception step S202 Display Processing step S203 selection acceptance step S204 directivity control signal generation step S205 audio signal reception step S206 reproduction processing step.

Claims (11)

a sound pickup unit that receives voice input and generates a plurality of voice signals;
obtaining the plurality of audio signals and a full image of the participants with the plurality of participants displayed; using comparative information between the plurality of audio signals to locate a first sound source; Position information generation for specifying the position of a second sound source by performing face recognition on the entire image in a range including the position of the first sound source, and generating position information representing the position of the second sound source. Department and
an image generating unit that uses the position information to generate individual images for at least two speakers by extracting from the overall image ;
a display processing unit that causes a display device to display the enlarged individual images in an arrangement layout corresponding to the positional relationship based on the positional relationship of the individual images in the overall image, side by side with the overall image ;
a selection reception unit that receives selection of one image from the enlarged individual images ;
including
wherein the whole image has a display frame colored with a different color at each clipped position of each individual image;
The sound collection control system , wherein each of the enlarged individual images is at least partially colored with the same color as that of the corresponding display frame . wherein the comparison information is a phase difference between the plurality of audio signals;
The sound collection control system according to claim 1. wherein the comparison information is a power ratio of the plurality of audio signals;
The sound collection control system according to claim 1. further comprising a photographing unit for photographing an image in the direction of the position of the second sound source based on the position information generated by the position information generating unit;
The image generation unit generates the individual image from an image in a direction of the position of the second sound source.
The sound collection control system according to any one of claims 1 to 3 . The position information generation unit Fourier-transforms the audio signal from the time domain to the frequency domain, and extracts the audio signal included in the frequency domain related to human speech from the audio signal, thereby determining the position of the first sound source. locate,
The sound collection control system according to any one of claims 1 to 4 . The position information generating unit identifies the position of the first sound source by extracting the audio signal included in the frequency range from 0.2 kHz to 4 kHz.
The sound collection control system according to claim 5 . The image generation unit associates the position information with the individual image and transmits the information to an operation device.
The sound collection control system according to any one of claims 1 to 6 . The sound pickup unit includes an array microphone having a plurality of microphones,
The sound collection control system according to any one of claims 1 to 7 . Further comprising a photographing unit for photographing the entire image,
The sound collection control system according to claim 1. The image generation unit associates position information corresponding to the individual image in the overall image with the individual image , and transmits the information to the operation device.
The sound collection control system according to claim 1. accepts voice input, generates multiple voice signals,
obtaining the plurality of audio signals and a full image of the participants with the plurality of participants displayed; using comparative information between the plurality of audio signals to locate a first sound source; Identifying the position of a second sound source by performing face recognition on the entire image in a range including the position of the first sound source, and generating position information representing the position of the second sound source;
using the location information to generate individual images for at least two speakers cut from the overall image ;
causing a display device to display each enlarged individual image in an arrangement layout corresponding to the positional relationship, based on the positional relationship of the individual images in the overall image, side by side with the overall image ;
accepting selection of one image from each of the enlarged individual images ;
A control method for a sound pickup control system , comprising:
wherein the whole image has a display frame colored with a different color at each clipped position of each individual image;
A control method for a sound collection control system, wherein at least a part of each of the enlarged individual images is colored with the same color as that of the corresponding display frame .

Images