JP2021196433A - Sound pickup device, system, program, and method for transmitting environmental sound signal picked up by plurality of microphones to reproduction device - Google Patents

Abstract

To provide a sound pickup device etc. for transmitting an environmental sound signal picked up by a plurality of microphones so that a reproduction device can reproduce it to enable a sound to be heard from a predetermined arrival direction.SOLUTION: A sound pickup device that transmits environmental sound signals picked up by a plurality of microphones to a reproduction device which reproduces sounds through a plurality of speakers has: a first acoustic database which stores an acoustic signal for each acoustic tag; sound source separation means which detects one or more acoustic signals present in the environmental sound signals from the environmental sound signals, and also estimates an arrival direction of each acoustic signal; acoustic tag estimation means which uses the first acoustic database to estimate an acoustic tag of an acoustic signal; and acoustic tag transmission means which transmits acoustic tags and the arrival direction to the reproduction device. Arrival method estimation means estimates arrival directions of acoustic signals through a spline sound source separation system or beam forming using the plurality of microphones.SELECTED DRAWING: Figure 2

Description

The present invention relates to a technique for collecting and reproducing an acoustic signal inherent in an environmental sound. In particular, it is suitable for telepresence (video conference) systems via networks.

Not limited to work and work, there are many cases where members live apart from each other, such as parents and children who live apart from each other. In this case, the situation of the other party is recognized by the telephone and communication functions of a smartphone or a personal computer. At this time, it is necessary to communicate voluntarily, and problems such as feeling alienated by members who work remotely have been pointed out (see, for example, Non-Patent Document 1).

On the other hand, a telepresence system that constantly flows video and audio between a plurality of bases has come to be used (see, for example, Non-Patent Documents 2 and 3). This is the same as a video conference system, but it is not only connected during a call or conference, but is always connected. According to this system, even members (parents and children, family members, employees) who stay at different remote locations can share the situation with each other by constantly transmitting environmental sounds and images around the other party, as if they were. It is possible to provide an environment that makes you feel like you are in the same place. The telepresence system can be easily connected via a network not only between domestic and overseas bases but also between a company and a home or shared office.
For example, according to Skype (registered trademark), which is a typical videophone, the status of "enrollment / absence" can be displayed on the display of the other party's terminal in order to share the trends of members at remote locations. ..

As another conventional technique, when the image is enlarged and displayed, the direction and width of the sound field of the acoustic signal picked up by a plurality of microphones are adjusted to obtain a sound field that matches the range of the image specified by the user. There is also a technique for reproducing (see, for example, Patent Document 1). According to this technology, it is possible to adjust and reproduce the direction and width of the sound field of the acoustic signal picked up by a plurality of microphones.
In addition, there is also a technique for transmitting the situation of the other party by synthetic sound in order to know the situation of the member of the remote base without looking at the display during the work (see, for example, Non-Patent Document 4).
Further, there is also a technology of an environmental sound recognition device that recognizes various environmental sounds from each other between remote bases (see, for example, Patent Document 2).

Japanese Unexamined Patent Publication No. 2019-066210 Japanese Patent No. 6085538

Ministry of Internal Affairs and Communications, "Research Report on Trends and Productivity of Telework, International Strategy Bureau for Information and Communication, Ministry of Internal Affairs and Communications (2010)", [online], [Search on March 10, 2nd year of Reiwa], Internet <URL: https //www.soumu.go.jp/johotsusintokei/linkdata/h22_06_houkoku.pdf ＞ Telepresence: Integrating shared task and person spaces, W Buxton --Proceedings of graphics interface, 1992, [online], [Search on March 10, 2nd year of Reiwa], Internet <URL: https://www.billbuxton.com/TelepShrdSpce .pdf ＞ Proceedings of the Japan Ergonomics Society Conference 406-407, 2009: Always-on audio conferencing system for telework HRI 2018: Fribo: A Social Networking Robot for Increasing Social Connectedness through Sharing Daily Home Activities from Living Noise Data., [Online], [Search on March 10, 2nd year of Reiwa], Internet <URL: https: // yonsei. pure.elsevier.com/en/publications/fribo-a-social-networking-robot-for-increasing-social-connectedne ＞ Institute of Electronics, Information and Communication Engineers "Knowledge Forest", 2 groups (image / sound / language) -6 volumes (acoustic signal processing) -Chapter 2 (sound source separation), [online], [Search on April 20, 2nd year of Reiwa] , Internet <URL: http://www.ieice-hbkb.org/files/02/02gun_06hen_02.pdf> Kazuho Ono, "Multi-channel audio", [online], [Search on March 10, 2nd year of Reiwa], Internet <URL: https://www.jstage.jst.go.jp/article/itej/68/8 / 68_604 / _pdf / -char / ja ＞

According to the existing telepresence system, the acoustic signal picked up by the sound pick-up device of the first base can be reproduced by the reproduction device of the second base. At this time, the environmental sound of the first base is simply recorded, and the environmental sound of the second base is simply reproduced as it is.

On the other hand, in the second base, the inventors of the present application reproduce the environmental sound of the first base according to the sound source position of the first base, and the user of the second base is the second. I thought that I could feel the existence of the user of one base in an atmosphere. For example, if the faucet sound of the water supply is picked up at the first base, can it be reproduced so that the faucet sound of the water supply of the first base arrives from the same sound source position at the second base? Thought.

Therefore, an object of the present invention is to provide a sound collecting device, a system, a program, and a method for transmitting an environmental sound signal collected by a plurality of microphones so that the sound can be reproduced by a reproducing device so as to be heard from a predetermined arrival direction. do.

According to the present invention, in a sound collecting device that transmits an environmental sound signal picked up by a plurality of microphones to a playback device that is reproduced by a plurality of speakers.
A first acoustic database that stores acoustic signals for each acoustic tag,
A sound source separation means that detects one or more acoustic signals inherent in the environmental sound signal from the environmental sound signal and estimates the arrival direction of each acoustic signal.
An acoustic tag estimation means for estimating an acoustic tag of an acoustic signal using the first acoustic database,
It is characterized by having an acoustic tag transmitting means for transmitting an acoustic tag and an arrival direction to a reproduction device.

According to another embodiment of the sound collecting device of the present invention.
As the arrival method estimation means, it is also preferable to estimate the arrival direction of the acoustic signal by a brine sound source separation method using a plurality of microphones or beamforming.

According to another embodiment of the sound collecting device of the present invention.
It is connected to the sound tag and the environment sensor associated with the direction of arrival.
When the acoustic tag transmitting means receives a predetermined signal from the environment sensor, it is also preferable to transmit the acoustic tag and the arrival direction of the environment sensor to the reproduction device.

According to another embodiment of the sound collecting device of the present invention.
It is connected to the camera and
An image database that stores image objects associated with acoustic tags, and
It further has an image object detection engine that detects one or more image objects inherent in the image captured by the camera using an image database and identifies the acoustic tag of the image object.
It is also preferable that the acoustic tag transmitting means transmits the arrival direction in the acoustic signal associated with the acoustic tag identified by the image object detection engine.

According to the present invention, in a system in which the above-mentioned sound collecting device and a playback device equipped with a plurality of speakers are connected via a network.
The playback device is
A second acoustic database that stores acoustic signals for each acoustic tag,
An acoustic tag receiving means for receiving an acoustic tag and an arrival direction from a sound collecting device,
Using the second acoustic database, the acoustic signal associated with the acoustic tag is characterized by having an environmental sound reproducing means for outputting an environmental sound synthesized so as to be heard from the direction of arrival of the acoustic tag from a plurality of speakers. And.

According to other embodiments in the system of the present invention.
The acoustic tag transmitting means in the sound collecting device further transmits an acoustic signal together with the acoustic tag and the direction of arrival.
It is also preferable that the environmental sound reproducing means in the reproducing device reproduces the acoustic signal received from the sound collecting device instead of the acoustic signal associated with the acoustic tag.

According to other embodiments in the system of the present invention.
The acoustic tag and the acoustic signal stored in the second acoustic database of the reproduction device are a part or all of the acoustic tag and the acoustic signal stored in the first acoustic database of the sound pickup device.
Even if the acoustic tag stored in the second acoustic database of the reproducing device and the acoustic tag stored in the first acoustic database of the sound collecting device are the same, they may be acoustic signals based on different acoustic signals. preferable.

According to the present invention, in a program for operating a computer mounted on a sound collecting device, which transmits an environmental sound signal picked up by a plurality of microphones to a playback device reproduced by a plurality of speakers.
A first acoustic database that stores acoustic signals for each acoustic tag,
A sound source separation means that detects one or more acoustic signals inherent in the environmental sound signal from the environmental sound signal and estimates the arrival direction of each acoustic signal.
An acoustic tag estimation means for estimating an acoustic tag of an acoustic signal using the first acoustic database,
It is characterized in that a computer functions as an acoustic tag transmitting means for transmitting an acoustic tag and an arrival direction to a playback device.

According to the present invention, in a program that receives an acoustic tag and an arrival direction from the above-mentioned sound collecting device and makes a computer mounted on a playback device equipped with a plurality of speakers function.
A second acoustic database that stores acoustic signals for each acoustic tag,
An acoustic tag receiving means for receiving an acoustic tag and an arrival direction from a sound collecting device,
Using the second acoustic database, the computer functions as an environmental sound reproduction means for outputting the environmental sound synthesized so that the acoustic signal associated with the acoustic tag can be heard from the direction of arrival of the acoustic tag from a plurality of speakers. It is characterized by.

According to the present invention, in a sound collecting method of a sound collecting device, which transmits an environmental sound signal picked up by a plurality of microphones to a playback device reproduced by a plurality of speakers.
The sound collecting device is
Each acoustic tag has a first acoustic database that stores acoustic signals.
From the environmental sound signal, the first step of detecting one or more acoustic signals inherent in the environmental sound signal and estimating the arrival direction of each acoustic signal, and
The second step of estimating the acoustic tag of the acoustic signal using the first acoustic database,
It is characterized by performing a third step of transmitting the acoustic tag and the arrival direction to the reproduction device.

According to the present invention, in a reproduction method of a reproduction device equipped with a plurality of speakers by receiving an acoustic tag and an arrival direction from the sound collecting device described above.
The playback device is
Each acoustic tag has a second acoustic database that stores acoustic signals.
The first step of receiving the acoustic tag and the direction of arrival from the sound collector,
Using the second acoustic database, the second step of outputting the environmental sound synthesized so that the acoustic signal associated with the acoustic tag can be heard from the direction of arrival of the acoustic tag from the plurality of speakers is executed. It is a feature.

According to the sound collecting device, the system, the program and the method of the present invention, the environmental sound signal picked up by a plurality of microphones can be transmitted so as to be reproduced by the reproducing device so as to be heard from a predetermined arrival direction.
According to the present invention, specifically, the acoustic signal of each sound source on the sound collecting side can be reproduced according to the position of each sound source on the reproducing side. Even when the sound collecting device and the reproducing device are located at different bases, the members staying remotely can share each other's environmental sounds.

It is an external view which shows the environmental sound of the base A where the sound collecting device is arranged. It is a functional block diagram of the sound collecting device in this invention. It is explanatory drawing of the sound source separation part and the acoustic tag estimation part in a sound collecting apparatus. It is explanatory drawing which shows the detection of the arrival direction using the blind sound source separation method. It is explanatory drawing which shows the detection of the arrival direction using the beamforming method. It is a functional block diagram of the reproduction apparatus in this invention. It is explanatory drawing which shows the data which is transmitted from a sound collecting apparatus to a reproduction apparatus. It is explanatory drawing of the environmental sound reproduction part of the reproduction apparatus in pattern 1. FIG. It is an external view which shows the acoustic signal reproduced by the pattern 1. FIG. It is explanatory drawing of the environmental sound reproduction part in pattern 2. It is an external view which shows the acoustic signal reproduced by the pattern 2. It is explanatory drawing of the sound collecting device which transmits an acoustic tag corresponding to an environment sensor. It is explanatory drawing of the sound collecting device which transmits the acoustic tag estimated from the image of a camera.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Sound collecting device 1>
FIG. 1 is an external view showing the environmental sound of the base A in which the sound collecting device is arranged.

According to the present invention, it is composed of at least a sound collecting device 1 arranged in the first base.
The sound collecting device 1 detects the arrival direction of a plurality of acoustic signals from the environmental sound heard by the user in the first base, and transmits the acoustic tag and the arrival direction to the reproduction device (pattern 1). Further, the collected acoustic signal itself may be transmitted to the reproduction device (Pattern 2).

According to FIG. 1, the appearance of the surrounding environment of the users a1 and a2 is shown, and a water supply, a window, and a washing machine are arranged. At this time, the users a1 and a2 hear the following acoustic signals as mixed environmental sounds.
Water faucet sound "jar"
Window opening and closing sound "fluttering"
Washing machine drive sound "Grungun"
The sound collecting device 1 is equipped with a plurality of microphones for collecting environmental sounds, and communicates with a remote reproducing device 2 via a network.

FIG. 2 is a functional configuration diagram of the sound collecting device according to the present invention.

According to FIG. 2, the sound collecting device 1 includes a plurality of microphones 101, a first acoustic database 11, a sound source separation unit 12, an acoustic tag estimation unit 13, an acoustic tag transmission unit 14, and a video transmission unit 15. And have. These functional components can be realized by executing a program that makes the computer mounted on the device function. Further, the processing flow of these functional components can be understood as a sound collection transmission method.

[Microphone 101]
The microphone 101 is composed of a plurality of microphones that collect environmental sounds. For example, it may be something like a microphone array. The microphone array can acquire spatial information of sound by signal processing the environmental sound picked up by a plurality of microphones.

[First acoustic database 11]
The first acoustic database 11 accumulates acoustic signals for each acoustic tag.
Acoustic tag <-> Acoustic signal An "acoustic tag" is an identifier for identifying an acoustic signal.
The "acoustic signal" is not limited to the acoustic signal itself, and may be something like a standard pattern of acoustic features such as a time-series frequency spectrum.

[Sound source separation unit 12]
The sound source separation unit 12 detects one or more acoustic signals inherent in the environmental sound and estimates the arrival direction of each acoustic signal.

FIG. 3 is an explanatory diagram of a sound source separation unit and an acoustic tag estimation unit in the sound collecting device.
According to FIG. 3, the environmental sound picked up by the microphone 101 is input to the sound source separation unit 12. Various acoustic signals such as the following are inherent in this environmental sound.
"Battery"
"Grun Grun"
"Jar"
The sound source separation unit 12 outputs an acoustic signal separately detected for each sound source and its arrival direction.

A blind sound source separation method or a beamforming method can be adopted for the sound source separation unit 12 in order to estimate the arrival direction of the acoustic signal. According to these methods, acoustic signals mixed with environmental sounds can be detected, and the direction of arrival of each acoustic signal can also be detected.

(Blind sound source separation method)
FIG. 4 is an explanatory diagram showing the detection of the arrival direction using the blind sound source separation method.
In the blind sound source separation method, for example, when based on independent component analysis (see Non-Patent Document 5), the separated signals are independent of each other under the assumption that even if a plurality of sound sources are unknown, they are statistically independent of each other. Configure such a filter. Up to (number of microphones-1) acoustic signals can be detected. In principle, the blind sound source separation method does not require information such as knowledge of the type and spatial position of the sound source, cutting out the section of the target sound, and synthesis conditions, and does not use the assumption of the tuning structure of the sound source signal.

(Beamforming method)
FIG. 5 is an explanatory diagram showing detection of an arrival direction using a beamforming method.
The beamforming method refers to a method in which each microphone detects an acoustic signal of a sound source in a target direction (see, for example, Non-Patent Document 5). The phase and amplitude are controlled by delays and filters based on the different sound wave propagation from the sound source to each microphone. As a result, the sensitivity of the acoustic signal other than the target direction is lowered, and the sensitivity (S / N ratio) of the acoustic signal in the target direction is secured.
Specifically, the angle from the microphone array is divided into a plurality of parts (8 divisions according to FIG. 5), and the acoustic signal is picked up as the target direction for each angle range.

[Acoustic tag estimation unit 13]
The acoustic tag estimation unit 13 estimates the acoustic tag of the acoustic signal by using the first acoustic database 11.

The acoustic tag estimation unit 13 extracts the mel frequency cepstrum coefficient (MFCC) as a feature quantity, and identifies the acoustic signal using a neural network based on deep learning (see, for example, Non-Patent Documents 3 and 4). It builds a multi-layered hierarchical network by stacking hidden layers pre-learned by a self-encoder based on a constrained Boltzmann machine (RBM), adding an identification network using the output of the final layer, and the whole. The acoustic tag is detected by supervised learning.
As a learning step, the acoustic tag estimation unit 13 learns from the teacher data associated with the acoustic tags and acoustic signals stored in the first acoustic database 11. As an estimation step, an acoustic signal from the sound source separation unit 12 is input, and an acoustic tag corresponding to the acoustic signal is output.

According to FIG. 3, the acoustic tag estimation unit 13 estimates the acoustic tag as follows, for example.
Acoustic tag 101 (water faucet sound)
Acoustic tag 167 (washing machine drive sound)
Acoustic tag 143 (window opening and closing sound)

[Acoustic tag transmitter 14]
The acoustic tag transmission unit 14 transmits the acoustic tag and the arrival direction to the reproduction device 2 (pattern 1).
Further, as another embodiment, when the acoustic signal picked up by the sound collecting device 1 is reproduced as it is by the reproducing device 2, the acoustic tag transmitting unit 14 also transmits the "acoustic signal" itself to the reproducing device 2 (pattern). 2).

[Video transmitter 15]
The video transmission unit 15 transmits the video captured by the camera to the playback device 2. As a telepresence system, the image of the base A is transmitted to the base B.

<Reproduction device 2>
The reproduction device 2 is arranged in the second base and receives the acoustic tag and the arrival direction from the sound collecting device. Then, a plurality of acoustic signals received from the sound collecting device of the first base are synthesized and reproduced as the environmental sound in the second base. At this time, each acoustic signal can be reproduced so as to be heard from a predetermined arrival direction. Multi-channel acoustic techniques can be used for this (see, eg, Non-Patent Document 6).

FIG. 6 is a functional configuration diagram of the reproduction device according to the present invention.
According to FIG. 6, the reproduction device 2 includes a speaker 201, a display 202, a second acoustic database 21, an acoustic tag receiving unit 22, an environmental sound reproduction unit 23, and a video reproduction unit 24. These functional components can be realized by executing a program that makes the computer mounted on the device function. In addition, the processing flow of these functional components can be understood as an environmental sound reproduction method.

[Second acoustic database 21]
The second acoustic database 21 accumulates acoustic signals for each acoustic tag. Basically, the first acoustic database 11 of the sound collecting device 1 has the same function as the second acoustic database of the reproducing device 2. The acoustic signal associated with the acoustic tag is preferably an acoustic signal as close to the original sound as possible.

Here, the acoustic tag and the acoustic signal stored in the second acoustic database 21 of the reproducing device 2 are a part or all of the acoustic tag and the acoustic signal stored in the first acoustic database 11 of the sound collecting device 1. There may be.
For example, if the acoustic tag and the acoustic signal stored in the first acoustic database 11 of the sound collecting device 1 are not stored in the second acoustic database 21 of the reproducing device 2, the acoustic signal is not reproduced. ..
On the other hand, when the acoustic tag and the acoustic signal stored in the first acoustic database 11 of the sound collecting device 1 are also stored in the second acoustic database 21 of the reproducing device 2, the second acoustic database in the reproducing device 2 is used. The environmental sound synthesized by the acoustic signal of the acoustic database 21 is reproduced. That is, when the acoustic tag stored in the second acoustic database 21 of the reproducing device 2 and the acoustic tag stored in the first acoustic database 11 of the sound collecting device 1 are different acoustic signals, the second The environmental sound converted by the acoustic signal of the acoustic database 21 will be reproduced.

[Acoustic tag receiver 22]
The acoustic tag receiving unit 22 receives the acoustic tag and the arrival direction (and the acoustic signal) from the sound collecting device 1. The received acoustic tag and the arrival direction (and the acoustic signal) are output to the environmental sound reproduction unit 23.

[Environmental sound reproduction unit 23]
The environmental sound reproduction unit 23 synthesizes the acoustic signal associated with the received acoustic tag so that it can be heard from the received arrival direction, and reproduces the environmental sound. The ambient sound is output to the plurality of speakers 201.
The speaker 201 is composed of a plurality of speakers, and reproduces the environmental sound so that the user can hear it from the direction of arrival of the sound source position at the base A where the sound collecting device 1 is arranged.

[Video playback unit 24]
The video reproduction unit receives the video from the sound collecting device 1 and outputs the video to the display 202.
The display 202 reproduces the image and makes it visible to the user.

FIG. 7 is an explanatory diagram showing data transmitted from the sound collecting device to the reproducing device.

According to FIG. 7, for example, there are examples of the following two patterns.
[Pattern 1]
The acoustic tag transmitting unit 14 of the sound collecting device 1 transmits the “acoustic tag” and the “arrival direction” to the acoustic tag receiving unit 25 of the reproducing device 2.
(Sound collecting device)
Acoustic tag 101 (water faucet sound): Arrival direction 1->
Acoustic tag 167 (washing machine drive sound): Arrival direction 3->
Acoustic tag 143 (window opening / closing sound): Arrival direction 8->
[Pattern 2]
The acoustic tag transmitting unit 14 of the sound collecting device 1 transmits the “acoustic tag”, “arrival direction”, and “acoustic signal” to the acoustic tag receiving unit 25 of the reproducing device 2.
(Sound collecting device)
Acoustic tag 101 (water faucet sound): Arrival direction 1: Acoustic signal "jar"->
Acoustic tag 167 (driving sound of washing machine): Arrival direction 3: Acoustic signal "Grungun"->
Acoustic tag 143 (window opening / closing sound): Arrival direction 8: Acoustic signal "flutter"->

FIG. 8 is an explanatory diagram of the environmental sound reproduction unit of the reproduction device in the pattern 1.

According to FIG. 8, the environmental sound reproduction unit 23 inputs an acoustic tag and an arrival direction from the acoustic tag receiving unit 22 for the acoustic signal inherent in the environmental sound.
Acoustic tag 101 (water faucet sound): Arrival direction 1
Acoustic tag 167 (washing machine drive sound): Arrival direction 3
Acoustic tag 143 (window opening / closing sound): Arrival direction 8
Further, according to the second acoustic database 21, the following acoustic signals are associated with each acoustic tag.
Acoustic tag 101 (water faucet sound): Acoustic signal "Shah"
Acoustic tag 167 (washing machine drive sound): Acoustic signal "Krun Krun"
Acoustic tag 143 (window opening / closing sound): Acoustic signal "key"
The environmental sound reproduction unit 23 synthesizes the acoustic signals "Shah", "Krunkrun", and "Kee" registered in the second acoustic database 21 so as to be heard from the sound sources in each arrival direction at the base A, and reproduces the environmental sound. ..
Acoustic signal "Shah": Arrival direction 1
Acoustic signal "Krun Krun": Arrival direction 3
Acoustic signal "key": arrival direction 8

FIG. 9 is an external view showing the acoustic signal reproduced by the pattern 1.

According to FIG. 9, the user b at the base B where the reproduction device 2 is arranged can hear each acoustic signal from the arrangement position of the water supply, the window, and the washing machine which are the sound sources of the base A.
For example, as the environmental sound of the base B, the acoustic signal "key" is reproduced from the direction of the window of the base A. This is because, when the window opening / closing sound "butter" is detected as the environmental sound of the base A, the acoustic signal "key" to be reproduced is reproduced so as to come from the same direction as the window of the base A. It is preferable to reproduce a sound as close to the original sound as possible.
In this way, the acoustic signal registered in the second acoustic database 21 can be reproduced as an environmental sound that can be heard from a sound source at a position corresponding to the arrival direction of the acoustic signal picked up by the sound collecting device 1. can.

FIG. 10 is an explanatory diagram of the environmental sound reproduction unit in the pattern 2.

According to FIG. 10, the environmental sound reproduction unit 23 inputs the acoustic tag, the arrival direction, and the acoustic signal from the acoustic tag receiving unit 22 for the acoustic signal inherent in the environmental sound.
Acoustic tag 101 (water faucet sound): Arrival direction 1: Acoustic signal "jar"
Acoustic tag 167 (driving sound of washing machine): Arrival direction 3: Acoustic signal "Grungun"
Acoustic tag 143 (window opening / closing sound): Arrival direction 8: Acoustic signal "flutter"
The environmental sound reproduction unit 23 synthesizes the received acoustic signals "jar", "grungrun", and "batat" so as to be heard from the sound sources in each arrival direction at the base A, and reproduces the environmental sound.
Acoustic signal "jar": arrival direction 1
Acoustic signal "Grungun": Arrival direction 3
Acoustic signal "flutter": Arrival direction 8

FIG. 11 is an external view showing the acoustic signal reproduced by the pattern 2.

According to FIG. 11, the user b at the base B where the reproduction device 2 is arranged can hear each acoustic signal from the arrangement position of the water supply, the window, and the washing machine which are the sound sources of the base A.
Further, according to FIG. 11, a display 202 is arranged at the base B, and the image of the base A taken by the sound collecting device 1 is reproduced. At this time, a "window" is reflected in the image. For example, as the environmental sound of the base B, the acoustic signal "bat" is reproduced from the direction of the window of the base A. This means that the acoustic signal "butter" to be reproduced is reproduced so as to come from the same direction as the window in the image of the base A.
In this way, the acoustic signal picked up by the sound collecting device 1 of the base A can be reproduced at the base B as an environmental sound that can be heard from the sound source at the position corresponding to the direction of arrival.

FIG. 12 is an explanatory diagram of a sound collecting device that transmits an acoustic tag corresponding to an environment sensor.

According to FIG. 12, the sound collecting device 1 is connected to the environment sensor 17 and receives an ON / OFF signal. The environment sensor may be, for example, a window open / close sensor. The environment sensor is associated with any one acoustic tag. The ON / OFF signal of the environment sensor is input to the acoustic tag transmission unit 14. When the acoustic tag transmission unit 14 receives a predetermined signal from the environment sensor 17, the acoustic tag transmission unit 14 transmits the acoustic tag corresponding to the environmental sound signal and the arrival direction to the reproduction device 2. Thereby, for example, only the window opening / closing sound can be transmitted to the reproduction device 2.

FIG. 13 is an explanatory diagram of a sound collecting device that transmits an acoustic tag estimated from a camera image.

According to FIG. 13, the sound collecting device 1 inputs an image taken by the camera 102.
Further, according to FIG. 13, the sound collecting device 1 further includes an image database 180 and an image object detection engine 181.
The image database 180 stores image objects associated with acoustic tags.
The image object detection engine 181 detects one or more image objects inherent in the image captured by the camera 102 using the image database 180, and identifies the acoustic tag of the image object. The specified acoustic tag is output to the acoustic tag transmission unit 14.

Specifically, the image object detection engine 181 surrounds an object (image object) with a frame (bounding box) from the input image or video, and identifies the type (category) of the object. This may be something like an SSD (Single Shot Multibox Detector), for example. The SSD divides the image by a grid and detects the bounding box at that position from the fit of a plurality of bounding boxes fixed to each grid. One image object fits in the bounding box.
Further, the image object detection engine 181 may be a neural network such as a CNN (Convolutional Neural Network) based on RGB recognition, and may be a neural network such as YOLO (You Only Look Once) (registered trademark). ..

As described in detail above, according to the sound collecting device, system, program and method of the present invention, the environmental sound signal collected by a plurality of microphones can be reproduced by the reproducing device so as to be heard from a predetermined arrival direction. Can be sent to.
According to the present invention, specifically, the acoustic signal of each sound source on the sound collecting side can be reproduced on the reproducing side according to the sound source position on the sound collecting side. Even when the sound collecting device and the reproducing device are located at different bases, the members staying remotely can share each other's environmental sounds.

Various modifications, modifications and omissions of the above-mentioned various embodiments of the present invention within the scope of the technical idea and viewpoint of the present invention can be easily made by those skilled in the art. The above explanation is just an example and does not attempt to limit anything. The present invention is limited only to the scope of claims and their equivalents.

1 Sound pickup device 101 Microphone 102 Camera 11 First sound database 12 Sound source separation unit 13 Sound tag estimation unit 14 Sound tag transmission unit 15 Video transmission unit 17 Environment sensor 180 Image database 181 Image object detection engine 2 Playback device 201 Speaker 202 Display 21 Second acoustic database 22 Acoustic tag receiver 23 Environmental sound reproduction unit 24 Video reproduction unit

Claims (11)

In a sound collecting device that transmits an environmental sound signal collected by a plurality of microphones to a playback device that is reproduced by a plurality of speakers.
A first acoustic database that stores acoustic signals for each acoustic tag,
A sound source separation means that detects one or more acoustic signals inherent in the environmental sound signal from the environmental sound signal and estimates the arrival direction of each acoustic signal.
An acoustic tag estimation means for estimating an acoustic tag of an acoustic signal using the first acoustic database,
A sound collecting device comprising an acoustic tag transmitting means for transmitting an acoustic tag and an arrival direction to a reproduction device. The sound collecting device according to claim 1, wherein the arrival method estimation means estimates the arrival direction of an acoustic signal by a brine sound source separation method using a plurality of microphones or beamforming. It is connected to the sound tag and the environment sensor associated with the direction of arrival.
The sound collecting device according to claim 1 or 2, wherein the sound tag transmitting means transmits the sound tag and the arrival direction of the environment sensor to the reproduction device when a predetermined signal is received from the environment sensor. It is connected to the camera and
An image database that stores image objects associated with acoustic tags, and
It further has an image object detection engine that detects one or more image objects inherent in the image captured by the camera using an image database and identifies the acoustic tag of the image object.
The sound collecting device according to any one of claims 1 to 3, wherein the acoustic tag transmitting means transmits an arrival direction in an acoustic signal associated with an acoustic tag identified by an image object detection engine. In a system in which the sound collecting device according to any one of claims 1 to 4 and a playback device equipped with a plurality of speakers are connected via a network.
The playback device is
A second acoustic database that stores acoustic signals for each acoustic tag,
An acoustic tag receiving means for receiving an acoustic tag and an arrival direction from a sound collecting device,
Using the second acoustic database, the acoustic signal associated with the acoustic tag is characterized by having an environmental sound reproducing means for outputting an environmental sound synthesized so as to be heard from the direction of arrival of the acoustic tag from a plurality of speakers. System. The acoustic tag transmitting means in the sound collecting device further transmits an acoustic signal together with the acoustic tag and the direction of arrival.
The system according to claim 5, wherein the environmental sound reproducing means in the reproducing device reproduces an acoustic signal received from the sound collecting device instead of the acoustic signal associated with the acoustic tag. The acoustic tag and the acoustic signal stored in the second acoustic database of the reproduction device are a part or all of the acoustic tag and the acoustic signal stored in the first acoustic database of the sound pickup device.
Even if the acoustic tag stored in the second acoustic database of the reproducing device and the acoustic tag stored in the first acoustic database of the sound collecting device are the same, the acoustic signal is based on a different acoustic signal. The system according to claim 5. In a program that functions a computer installed in a sound collecting device that transmits an environmental sound signal picked up by multiple microphones to a playback device that is played back by multiple speakers.
A first acoustic database that stores acoustic signals for each acoustic tag,
A sound source separation means that detects one or more acoustic signals inherent in the environmental sound signal from the environmental sound signal and estimates the arrival direction of each acoustic signal.
An acoustic tag estimation means for estimating an acoustic tag of an acoustic signal using the first acoustic database,
A program of a sound collecting device, characterized in that a computer functions as an acoustic tag transmitting means for transmitting an acoustic tag and an arrival direction to a reproduction device. In a program for receiving an acoustic tag and an arrival direction from the sound collecting device according to any one of claims 1 to 4 and operating a computer mounted on a playback device equipped with a plurality of speakers.
A second acoustic database that stores acoustic signals for each acoustic tag,
An acoustic tag receiving means for receiving an acoustic tag and an arrival direction from a sound collecting device,
Using the second acoustic database, the computer functions as an environmental sound reproduction means for outputting the environmental sound synthesized so that the acoustic signal associated with the acoustic tag can be heard from the direction of arrival of the acoustic tag from a plurality of speakers. A program of a playback device characterized by. In the sound collecting method of a sound collecting device, which transmits an environmental sound signal picked up by a plurality of microphones to a playback device reproduced by a plurality of speakers.
The sound collecting device is
Each acoustic tag has a first acoustic database that stores acoustic signals.
From the environmental sound signal, the first step of detecting one or more acoustic signals inherent in the environmental sound signal and estimating the arrival direction of each acoustic signal, and
The second step of estimating the acoustic tag of the acoustic signal using the first acoustic database,
A sound collecting method comprising performing a third step of transmitting an acoustic tag and an arrival direction to a reproduction device. In a method for reproducing an acoustic tag and a reproduction device equipped with a plurality of speakers by receiving an acoustic tag and an arrival direction from the sound collecting device according to any one of claims 1 to 4.
The playback device is
Each acoustic tag has a second acoustic database that stores acoustic signals.
The first step of receiving the acoustic tag and the direction of arrival from the sound collector,
Using the second acoustic database, the second step of outputting the environmental sound synthesized so that the acoustic signal associated with the acoustic tag can be heard from the direction of arrival of the acoustic tag from the plurality of speakers is executed. Characteristic playback method.

Images