JP7403392B2 - Sound collection device, system, program, and method for transmitting environmental sound signals collected by multiple microphones to a playback device - Google Patents

Description

The present invention relates to technology for collecting and reproducing acoustic signals inherent in environmental sounds. It is particularly suitable for telepresence (video conference) systems via networks.

In many cases, members engage in activities and lives apart from each other in different locations, not just for work or business, but also for parents and children who live far apart. In this case, each party tries to recognize the other party's situation using the telephone and communication functions of smartphones, computers, etc. At this time, it is necessary to communicate spontaneously, and it has been pointed out that members working remotely may feel left out (for example, see Non-Patent Document 1).

In contrast, telepresence systems that constantly stream video and audio between multiple locations have come into use (for example, see Non-Patent Documents 2 and 3). This is the same as a video conferencing system, but instead of being connected only during calls or meetings, it is always connected. According to this system, even if members (parents and children, families, employees) staying in different remote locations can share each other's situation by constantly transmitting environmental sounds and images of the other party's surroundings, they can We can provide an environment where you feel like you are living in the same place. A telepresence system can easily connect not only between domestic and overseas bases, but also between a company and a home or shared office via a network.
For example, according to Skype (registered trademark), a typical videophone service, in order to share the movements of members at remote locations, it is possible to display the status of "attended/away" on the display of the other party's terminal. .

Another conventional technique is to adjust the direction and width of the sound field of acoustic signals picked up by multiple microphones to create a sound field that matches the range of the video specified by the user when displaying an enlarged video. There is also a technique for reproducing this (for example, see Patent Document 1). According to this technology, the direction and width of the sound field of acoustic signals collected by multiple microphones can be adjusted and reproduced.
In addition, there is also a technology in which the situation of a member at a remote site is communicated using a synthesized voice in order to know the situation of a member at a remote site even if the user is not looking at the display while working (for example, see Non-Patent Document 4).
Furthermore, there is a technology for an environmental sound recognition device that mutually recognizes various environmental sounds between remote locations (for example, see Patent Document 2).

JP2019-068210A Patent No. 6085538

Edited by the Ministry of Internal Affairs and Communications, “Research Report on Telework Trends and Productivity, International Strategy Bureau of Information and Communications, Ministry of Internal Affairs and Communications (2010)”, [online], [Retrieved March 10, 2020], Internet <URL: https //www.soumu.go.jp/johotsusintokei/linkdata/h22_06_houkoku.pdf＞ Telepresence: Integrating shared task and person spaces, W Buxton - Proceedings of graphics interface, 1992, [online], [Retrieved March 10, 2020], Internet <URL: https://www.billbuxton.com/TelepShrdSpce .pdf＞ Japan Ergonomics Society Conference Proceedings 406-407, 2009: Always-on audio conferencing system for telework HRI 2018: Fribo: A Social Networking Robot for Increasing Social Connectedness through Sharing Daily Home Activities from Living Noise Data., [online], [Retrieved March 10, 2020], Internet <URL: https://yonsei. pure.elsevier.com/en/publications/fribo-a-social-networking-robot-for-increasing-social-connectedne＞ Institute of Electronics, Information and Communication Engineers "Forest of Knowledge", Group 2 (Image/Sound/Language) - Volume 6 (Acoustic Signal Processing) - Chapter 2 (Sound Source Separation), [online], [Retrieved April 20, 2020] , Internet <URL:http://www.ieice-hbkb.org/files/02/02gun_06hen_02.pdf> Kazuho Ono, "Multichannel Audio", [online], [Retrieved March 10, 2020], Internet <URL: https://www.jstage.jst.go.jp/article/itej/68/8 /68_604/_pdf/-char/ja>

According to the existing telepresence system, a sound signal picked up by a sound pickup device at a first base can be reproduced by a playback device at a second base. At this time, the environmental sounds at the first base are simply recorded, and the second base simply plays back the environmental sounds as they are.

In contrast, the inventors of the present application have found that if the environmental sound of the first base is played back at the second base according to the sound source position of the first base, the user of the second base will We thought that the presence of users at base 1 could be felt in the atmosphere. For example, if the sound of a water faucet is collected at a first base, is it possible to reproduce it at a second base so that the sound of the water faucet at the first base comes from the same sound source position? Thought.

SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide a sound collection device, a system, a program, and a method for transmitting environmental sound signals collected by a plurality of microphones so that they can be reproduced by a reproduction device so as to be heard from a predetermined direction of arrival. do.

According to the present invention, in a sound collection device that transmits environmental sound signals collected by a plurality of microphones to a playback device that reproduces them using a plurality of speakers,
It is connected to an acoustic tag and an environmental sensor linked to the direction of arrival.
a first acoustic database that accumulates acoustic signals for each acoustic tag;
Sound source separation means for detecting one or more acoustic signals inherent in the environmental sound signal from the environmental sound signal and estimating the direction of arrival of each acoustic signal;
acoustic tag estimating means for estimating an acoustic tag of an acoustic signal using a first acoustic database;
The reproduction device reproduces the acoustic tag estimated by the acoustic tag estimating means, the arrival direction estimated by the sound source separation means, and the acoustic tag and arrival direction associated with the environmental sensor when receiving a predetermined signal from the environmental sensor. and an acoustic tag transmitting means for transmitting to the acoustic tag, and causes the reproduction device to reproduce the acoustic signal associated with the acoustic tag from the direction of arrival.

According to another embodiment of the sound collection device of the present invention,
It is also preferable that the sound source separation means estimates the arrival direction of the acoustic signal by a blind sound source separation method using a plurality of microphones or beamforming.

According to another embodiment of the sound collection device of the present invention,
connected to the camera,
an image database that stores image objects associated with acoustic tags;
further comprising an image object detection engine that uses the image database to detect one or more image objects within the video captured by the camera and identifies an acoustic tag of the image object;
It is also preferable that the acoustic tag transmitting means transmits the arrival direction of the acoustic signal associated with the acoustic tag identified by the image object detection engine.

According to the present invention, in a system in which the aforementioned sound collection device and a playback device equipped with a plurality of speakers are connected via a network,
The playback device is
a second acoustic database that accumulates acoustic signals for each acoustic tag;
acoustic tag receiving means for receiving the acoustic tag and the direction of arrival from the sound collection device;
It is characterized by having an environmental sound reproducing means for outputting, from a plurality of speakers, an environmental sound synthesized using a second acoustic database so that the acoustic signal linked to the acoustic tag can be heard from the direction of arrival of the acoustic tag. shall be.

According to other embodiments of the system of the invention:
The acoustic tag transmitting means in the sound collection device further transmits the acoustic signal together with the acoustic tag and the direction of arrival,
It is also preferable that the environmental sound reproducing means in the reproducing device reproduces the acoustic signal received from the sound collection device instead of the acoustic signal tied to the acoustic tag.

According to other embodiments of the system of the invention:
The acoustic tags and acoustic signals accumulated in the second acoustic database of the playback device are part or all of the acoustic tags and acoustic signals accumulated in the first acoustic database of the sound collection device,
Even if the acoustic tag stored in the second acoustic database of the playback device and the acoustic tag accumulated in the first acoustic database of the sound collection device are the same, the acoustic signals may be based on different acoustic signals. preferable.

According to the present invention, in a program that causes a computer installed in a sound collection device to transmit environmental sound signals picked up by a plurality of microphones to a playback device that reproduces them through a plurality of speakers,
The sound collection device is connected to an environmental sensor that is associated with an acoustic tag and a direction of arrival,
a first acoustic database that accumulates acoustic signals for each acoustic tag;
Sound source separation means for detecting one or more acoustic signals inherent in the environmental sound signal from the environmental sound signal and estimating the direction of arrival of each acoustic signal;
acoustic tag estimating means for estimating an acoustic tag of an acoustic signal using a first acoustic database;
The reproduction device reproduces the acoustic tag estimated by the acoustic tag estimating means, the arrival direction estimated by the sound source separation means, and the acoustic tag and arrival direction associated with the environmental sensor when receiving a predetermined signal from the environmental sensor. The present invention is characterized in that a computer functions as an acoustic tag transmitting means for transmitting an acoustic tag to a user, and causes a reproducing device to reproduce an acoustic signal associated with the acoustic tag from the direction of arrival.

According to the present invention, in a program that receives an acoustic tag and a direction of arrival from the above-described sound collection device and causes a computer installed in a playback device equipped with a plurality of speakers to function,
a second acoustic database that accumulates acoustic signals for each acoustic tag;
acoustic tag receiving means for receiving the acoustic tag and the direction of arrival from the sound collection device;
Using a second acoustic database, the computer functions as an environmental sound reproducing means for outputting from a plurality of speakers an environmental sound synthesized so that the acoustic signal linked to the acoustic tag can be heard from the direction of arrival of the acoustic tag. It is characterized by

According to the present invention, in a sound collection method for a sound collection device that transmits environmental sound signals collected by a plurality of microphones to a playback device that reproduces them using a plurality of speakers,
The sound collection device is
It is connected to an acoustic tag and an environmental sensor linked to the direction of arrival.
a first acoustic database for accumulating acoustic signals for each acoustic tag;
a first step of detecting one or more acoustic signals inherent in the environmental sound signal from the environmental sound signal and estimating the direction of arrival of each acoustic signal;
a second step of estimating an acoustic tag of the acoustic signal using the first acoustic database;
Reproducing the acoustic tag estimated in the second step, the direction of arrival estimated in the first step, and the acoustic tag and direction of arrival associated with the environmental sensor when the predetermined signal is received from the environmental sensor. The method is characterized in that the third step of transmitting the audio signal to the audio tag is executed, and the reproducing device reproduces the acoustic signal associated with the acoustic tag from the direction of arrival.

According to the present invention, in a reproduction method of a reproduction apparatus that receives an acoustic tag and an arrival direction from the above-mentioned sound collection device and is equipped with a plurality of speakers,
The playback device is
a second acoustic database for accumulating acoustic signals for each acoustic tag;
a first step of receiving an acoustic tag and a direction of arrival from a sound collection device;
and a second step of outputting from a plurality of speakers an environmental sound synthesized so that the acoustic signal associated with the acoustic tag can be heard from the direction of arrival of the acoustic tag using the second acoustic database. Features.

According to the sound collection device, system, program, and method of the present invention, environmental sound signals collected by a plurality of microphones can be transmitted so that they can be played back by a playback device so that they can be heard from a predetermined direction of arrival.
According to the present invention, specifically, the acoustic signal of each sound source on the sound collection side can be reproduced according to the position of each sound source on the reproduction side. Even if the sound collection device and the playback device are located at different bases, members staying remotely can share each other's environmental sounds.

It is an external view showing the environmental sound of base A where the sound collection device is arranged. It is a functional block diagram of the sound collection device in this invention. FIG. 2 is an explanatory diagram of a sound source separation unit and an acoustic tag estimation unit in the sound collection device. FIG. 2 is an explanatory diagram showing arrival direction detection using a blind sound source separation method. FIG. 3 is an explanatory diagram showing direction of arrival detection using a beamforming method. FIG. 2 is a functional configuration diagram of a playback device according to the present invention. FIG. 2 is an explanatory diagram showing data transmitted from a sound collection device to a playback device. FIG. 3 is an explanatory diagram of an environmental sound reproduction section of the reproduction device in pattern 1. 3 is an external view showing an acoustic signal reproduced by pattern 1. FIG. FIG. 6 is an explanatory diagram of an environmental sound reproduction section in pattern 2. 3 is an external view showing an acoustic signal reproduced by pattern 2. FIG. FIG. 2 is an explanatory diagram of a sound collection device that transmits an acoustic tag corresponding to an environmental sensor. FIG. 2 is an explanatory diagram of a sound collection device that transmits an acoustic tag estimated from a camera image.

Hereinafter, embodiments of the present invention will be described in detail using the drawings.

<Sound collection device 1>
FIG. 1 is an external view showing the environmental sounds of a base A where a sound collection device is placed.

According to the present invention, the sound collection device 1 includes at least the sound collection device 1 placed within the first base.
The sound collection device 1 detects the directions of arrival of a plurality of acoustic signals from the environmental sounds heard by the user within the first base, and transmits the acoustic tags and the directions of arrival to the playback device (pattern 1). Alternatively, the collected acoustic signal itself may be transmitted to the playback device (pattern 2).

According to FIG. 1, the appearance of the surrounding environment of users a1 and a2 is shown, and a water supply, windows, and a washing machine are arranged. At this time, users a1 and a2 hear environmental sounds that include a mixture of the following acoustic signals.
Water faucet sound "jar"
Window opening/closing sound “bap”
Washing machine driving sound "Grungurun"
The sound collection device 1 is equipped with a plurality of microphones for collecting environmental sounds, and also communicates with a remote playback device 2 via a network.

FIG. 2 is a functional configuration diagram of the sound collection device according to the present invention.

According to FIG. 2, the sound collection device 1 includes a plurality of microphones 101, a first acoustic database 11, a sound source separation section 12, an acoustic tag estimation section 13, an acoustic tag transmission section 14, and a video transmission section 15. and has. These functional components can be realized by executing a program that causes a computer installed in the device to function. Further, the processing flow of these functional components can also be understood as a sound collection and transmission method.

[Mike 101]
The microphone 101 includes a plurality of microphones that collect environmental sounds. For example, it may be something like a microphone array. A microphone array can acquire spatial information of sound by signal processing environmental sounds picked up by a plurality of microphones.

[First acoustic database 11]
The first acoustic database 11 stores acoustic signals for each acoustic tag.
Acoustic tag<->acoustic signal An “acoustic tag” is an identifier for specifying an acoustic signal.
The "acoustic signal" is not limited to the acoustic signal itself, but may be a standard pattern of acoustic features such as a time-series frequency spectrum.

[Sound source separation unit 12]
The sound source separation unit 12 detects one or more acoustic signals inherent in environmental sounds and estimates the direction of arrival of each acoustic signal.

FIG. 3 is an explanatory diagram of a sound source separation unit and an acoustic tag estimation unit in the sound collection device.
According to FIG. 3, the environmental sound picked up by the microphone 101 is input to the sound source separation unit 12. This environmental sound includes various acoustic signals such as the following.
"Bat"
"Grungrun"
"Jah"
The sound source separation unit 12 outputs acoustic signals separated and detected for each sound source and their direction of arrival.

The sound source separation unit 12 may employ a blind sound source separation method or a beamforming method in order to estimate the arrival direction of the acoustic signal. According to these methods, it is possible to detect acoustic signals mixed in environmental sounds and also detect the direction of arrival of each acoustic signal.

(Blind sound source separation method)
FIG. 4 is an explanatory diagram showing arrival direction detection using the blind sound source separation method.
A blind sound source separation method is, for example, when based on independent component analysis (see Non-Patent Document 5), separated signals are made independent of each other under the assumption that multiple sound sources are statistically independent from each other even if they are unknown. Configure a filter like this. Up to (number of microphones - 1) acoustic signals can be detected. Note that the blind sound source separation method does not in principle require information such as knowledge of the type and spatial position of the sound source, extraction of the target sound section, and synthesis conditions, and does not use assumptions about the harmonic structure of the sound source signal.

(Beamforming method)
FIG. 5 is an explanatory diagram showing detection of the direction of arrival using the beamforming method.
The beamforming method is a method in which each microphone detects an acoustic signal of a sound source in a target direction (see, for example, Non-Patent Document 5). Based on the fact that the sound waves propagate differently from the sound source to each microphone, the phase and amplitude are controlled by delays and filters. This reduces the sensitivity of acoustic signals in directions other than the target direction, and ensures the sensitivity (S/N ratio) of the acoustic signals in the target direction.
Specifically, the angle from the microphone array is divided into a plurality of parts (eight divisions according to FIG. 5), and an acoustic signal is collected in each angular range as a target direction.

[Acoustic tag estimation unit 13]
The acoustic tag estimation unit 13 uses the first acoustic database 11 to estimate the acoustic tag of the acoustic signal.

The acoustic tag estimating unit 13 extracts Mel frequency cepstral coefficients (MFCC) as features and identifies acoustic signals using a neural network based on deep learning (for example, see Non-Patent Documents 3 and 4). This is done by stacking hidden layers pre-trained by an autoencoder based on a constrained Boltzmann machine (RBM) to build a multilayer hierarchical network, and adding a discriminative network using the output of the final layer to complete the The acoustic tags are detected using supervised learning.
In the learning stage, the acoustic tag estimation unit 13 performs learning using teacher data in which acoustic tags and acoustic signals stored in the first acoustic database 11 are associated with each other. In the estimation stage, an acoustic signal from the sound source separation unit 12 is input, and an acoustic tag corresponding to the acoustic signal is output.

According to FIG. 3, the acoustic tag estimating unit 13 estimates an acoustic tag as follows, for example.
Acoustic tag 101 (water faucet sound)
Acoustic tag 167 (washing machine driving sound)
Acoustic tag 143 (window opening/closing sound)

[Acoustic tag transmitter 14]
The acoustic tag transmitter 14 transmits the acoustic tag and the direction of arrival to the playback device 2 (pattern 1).
In addition, as another embodiment, when the acoustic signal collected by the sound collection device 1 is reproduced as it is by the reproduction device 2, the acoustic tag transmitting unit 14 also transmits the “acoustic signal” itself to the reproduction device 2 (pattern 2).

[Video transmitter 15]
The video transmitter 15 transmits the video captured by the camera to the playback device 2. This is a telepresence system that transmits video from base A to base B.

<Playback device 2>
The playback device 2 is placed within the second base and receives the acoustic tag and direction of arrival from the sound pickup device. Then, the plurality of acoustic signals received from the sound collection device at the first base are synthesized and reproduced as environmental sounds within the second base. At this time, each acoustic signal can be reproduced so that it can be heard from a predetermined arrival direction. For this purpose, multi-channel audio technology can be used (see, for example, Non-Patent Document 6).

FIG. 6 is a functional configuration diagram of the playback device according to the present invention.
According to FIG. 6, the playback device 2 includes a speaker 201, a display 202, a second acoustic database 21, an acoustic tag receiving section 22, an environmental sound playback section 23, and a video playback section 24. These functional components can be realized by executing a program that causes a computer installed in the device to function. Further, the processing flow of these functional components can also be understood as an environmental sound reproduction method.

[Second acoustic database 21]
The second acoustic database 21 accumulates acoustic signals for each acoustic tag. Basically, the first acoustic database 11 of the sound collection device 1 has the same function as the second acoustic database of the reproduction device 2. The acoustic signal associated with the acoustic tag is preferably an acoustic signal as close to the original sound as possible.

Here, the acoustic tags and acoustic signals accumulated in the second acoustic database 21 of the playback device 2 are part or all of the acoustic tags and acoustic signals accumulated in the first acoustic database 11 of the sound collection device 1. There may be.
For example, if the acoustic tag and acoustic signal accumulated in the first acoustic database 11 of the sound collection device 1 are not accumulated in the second acoustic database 21 of the reproduction device 2, the acoustic signal is simply not reproduced. .
On the other hand, if the acoustic tags and acoustic signals accumulated in the first acoustic database 11 of the sound collection device 1 are also accumulated in the second acoustic database 21 of the reproduction device 2, Environmental sounds synthesized using acoustic signals from the acoustic database 21 are reproduced. That is, if the acoustic tag stored in the second acoustic database 21 of the playback device 2 and the acoustic tag accumulated in the first acoustic database 11 of the sound collection device 1 are different acoustic signals, the second The environmental sound converted by the acoustic signal of the acoustic database 21 will be reproduced.

[Acoustic tag receiving unit 22]
The acoustic tag receiving unit 22 receives the acoustic tag and direction of arrival (and acoustic signal) from the sound collection device 1 . The received acoustic tag and arrival direction (and acoustic signal) are output to the environmental sound reproduction section 23.

[Environmental sound reproduction section 23]
The environmental sound reproduction unit 23 synthesizes the acoustic signals associated with the received acoustic tags so that they can be heard from the received direction of arrival, and reproduces the environmental sounds. The environmental sound is output to multiple speakers 201.
The speaker 201 is composed of a plurality of speakers, and reproduces environmental sound so that the user can hear it from the direction of arrival of the sound source position at the base A where the sound collection device 1 is placed.

[Video playback section 24]
The video playback unit receives video from the sound collection device 1 and outputs the video to the display 202.
The display 202 plays back the video and allows the user to view it.

FIG. 7 is an explanatory diagram showing data transmitted from the sound collection device to the playback device.

According to FIG. 7, for example, there are two examples of the following patterns.
[Pattern 1]
The acoustic tag transmitter 14 of the sound collection device 1 transmits the “acoustic tag” and “direction of arrival” to the acoustic tag receiver 25 of the playback device 2.
(sound collection device)
Acoustic tag 101 (water faucet sound): Direction of arrival 1 ->
Acoustic tag 167 (washing machine driving sound): Direction of arrival 3 ->
Acoustic tag 143 (window opening/closing sound): Direction of arrival 8 ->
[Pattern 2]
The acoustic tag transmitting unit 14 of the sound collection device 1 transmits the “acoustic tag”, “direction of arrival”, and “acoustic signal” to the acoustic tag receiving unit 25 of the reproducing device 2.
(sound collection device)
Acoustic tag 101 (water faucet sound): Direction of arrival 1: Acoustic signal "Jar"->
Acoustic tag 167 (washing machine driving sound): Arrival direction 3: Acoustic signal "Grungurun"->
Acoustic tag 143 (window opening/closing sound): Arrival direction 8: Acoustic signal "bap"->

FIG. 8 is an explanatory diagram of the environmental sound reproduction section of the reproduction device in pattern 1.

According to FIG. 8, the environmental sound reproducing unit 23 inputs the acoustic tag and arrival direction from the acoustic tag receiving unit 22 regarding the acoustic signal inherent in the environmental sound.
Acoustic tag 101 (water faucet sound): Direction of arrival 1
Acoustic tag 167 (washing machine driving sound): Direction of arrival 3
Acoustic tag 143 (window opening/closing sound): Arrival direction 8
Further, according to the second acoustic database 21, the following acoustic signals are associated with each acoustic tag.
Acoustic tag 101 (water faucet sound): Acoustic signal “sha”
Acoustic tag 167 (washing machine driving sound): Acoustic signal "Krunkrun"
Acoustic tag 143 (window opening/closing sound): Acoustic signal “squeak”
The environmental sound reproduction unit 23 synthesizes the acoustic signals "Sha", "Crunchy", and "Squeak" registered in the second acoustic database 21 so that they can be heard from the sound sources in each direction of arrival at base A, and reproduces the environmental sound. .
Acoustic signal “Shar”: Arrival direction 1
Acoustic signal “Krunkrun”: Arrival direction 3
Acoustic signal “squeak”: Direction of arrival 8

FIG. 9 is an external view showing an acoustic signal reproduced by pattern 1.

According to FIG. 9, user b at base B, where the playback device 2 is located, can hear various acoustic signals from the locations of the water supply, windows, and washing machines that serve as sound sources at base A.
For example, as an environmental sound of base B, an acoustic signal "squeak" is played from the direction of the window of base A. In this case, when the sound of opening and closing a window "bap" is detected as an environmental sound of base A, the acoustic signal "squeak" to be reproduced is reproduced so that it comes from the same direction as the window of base A. It is preferable to reproduce sound as close to the original sound as possible.
In this way, the acoustic signals registered in the second acoustic database 21 can be reproduced as environmental sounds that can be heard from the sound source at the position according to the direction of arrival of the acoustic signals collected by the sound collecting device 1. can.

FIG. 10 is an explanatory diagram of the environmental sound reproduction section in pattern 2.

According to FIG. 10, the environmental sound reproducing unit 23 inputs an acoustic tag, an arrival direction, and an acoustic signal from the acoustic tag receiving unit 22 regarding the acoustic signal inherent in the environmental sound.
Acoustic tag 101 (water faucet sound): Direction of arrival 1: Acoustic signal “Jah”
Acoustic tag 167 (washing machine driving sound): Arrival direction 3: Acoustic signal "Grungurun"
Acoustic tag 143 (window opening/closing sound): Arrival direction 8: Acoustic signal “bap”
The environmental sound reproducing unit 23 synthesizes the received acoustic signals "jar", "gurungurung", and "bap" so that they can be heard from the sound sources in each direction of arrival at base A, and reproduces the environmental sound.
Acoustic signal “jar”: Arrival direction 1
Acoustic signal “Grungurun”: Arrival direction 3
Acoustic signal “bap”: Arrival direction 8

FIG. 11 is an external view showing the acoustic signal reproduced by pattern 2.

According to FIG. 11, the user b at the base B where the playback device 2 is located can hear each acoustic signal from the location of the water supply, window, and washing machine that are the sound sources of the base A.
Further, according to FIG. 11, a display 202 is arranged at the base B, and the video of the base A captured by the sound collection device 1 is reproduced. At this time, a "window" is reflected in the video. For example, as an environmental sound of site B, an acoustic signal "bap" is played from the direction of the window of site A. In this example, the sound signal "bap" to be played back is played so that it comes from the same direction as the window in the video of base A.
In this way, the sound signal picked up by the sound collection device 1 at base A can be reproduced at base B as an environmental sound that can be heard from a sound source located at a position corresponding to the direction of arrival.

FIG. 12 is an explanatory diagram of a sound collection device that transmits an acoustic tag corresponding to an environmental sensor.

According to FIG. 12, the sound collection device 1 is connected to an environmental sensor 17 and receives ON/OFF signals. The environmental sensor may be, for example, a window opening/closing sensor. The environmental sensor is tied to one of the acoustic tags. The ON/OFF signal of the environmental sensor is input to the acoustic tag transmitter 14. When the acoustic tag transmitter 14 receives a predetermined signal from the environmental sensor 17, the acoustic tag transmitter 14 transmits the acoustic tag and arrival direction corresponding to the environmental sound signal to the playback device 2. Thereby, for example, only the window opening/closing sound can be transmitted to the playback device 2.

FIG. 13 is an explanatory diagram of a sound collection device that transmits an acoustic tag estimated from a camera image.

According to FIG. 13, the sound collection device 1 receives an image captured by the camera 102.
Further, according to FIG. 13, the sound collection device 1 further includes an image database 180 and an image object detection engine 181.
The image database 180 stores image objects with associated acoustic tags.
The image object detection engine 181 uses the image database 180 to detect one or more image objects inherent in the video captured by the camera 102 and identifies the acoustic tag of the image object. The identified acoustic tag is output to the acoustic tag transmitter 14.

Specifically, the image object detection engine 181 surrounds an object (image object) with a frame (bounding box) from the input image or video, and identifies the type (category) of the object. This may be, for example, an SSD (Single Shot Multibox Detector). SSD divides an image into grids, and detects the bounding box at that position based on how well a plurality of bounding boxes fixed to each grid fit. One image object fits within that bounding box.
Further, the image object detection engine 181 may be a neural network such as a CNN (Convolutional Neural Network) based on RGB recognition, or a neural network such as YOLO (You Only Look Once) (registered trademark). .

As described above in detail, according to the sound collection device, system, program, and method of the present invention, environmental sound signals collected by a plurality of microphones can be reproduced by a reproduction device so that they can be heard from a predetermined direction of arrival. can be sent to.
According to the present invention, specifically, the acoustic signal of each sound source on the sound collection side can be reproduced on the reproduction side according to the position of the sound source on the sound collection side. Even if the sound collection device and the playback device are located at different bases, members staying remotely can share each other's environmental sounds.

Regarding the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical idea and viewpoint of the present invention can be easily made by those skilled in the art. The above description is merely an example and is not intended to be limiting in any way. The invention is limited only by the claims and their equivalents.

1 sound collection device 101 microphone 102 camera 11 first acoustic database 12 sound source separation section 13 acoustic tag estimation section 14 acoustic tag transmission section 15 video transmission section 17 environment sensor 180 image database 181 image object detection engine 2 playback device 201 speaker 202 display 21 Second acoustic database 22 Acoustic tag receiving section 23 Environmental sound reproduction section 24 Video reproduction section

Claims (10)

In a sound collection device that transmits environmental sound signals collected by multiple microphones to a playback device that reproduces them using multiple speakers,
It is connected to an acoustic tag and an environmental sensor linked to the direction of arrival.
a first acoustic database that accumulates acoustic signals for each acoustic tag;
Sound source separation means for detecting one or more acoustic signals inherent in the environmental sound signal from the environmental sound signal and estimating the direction of arrival of each acoustic signal;
acoustic tag estimating means for estimating an acoustic tag of an acoustic signal using a first acoustic database;
The reproduction device reproduces the acoustic tag estimated by the acoustic tag estimating means, the arrival direction estimated by the sound source separation means, and the acoustic tag and arrival direction associated with the environmental sensor when receiving a predetermined signal from the environmental sensor. What is claimed is: 1. A sound collection device comprising: an acoustic tag transmitting means for transmitting a signal to an acoustic tag, and causing a reproduction device to reproduce an acoustic signal associated with the acoustic tag from the direction of arrival. 2. The sound collection device according to claim 1, wherein the sound source separation means estimates the arrival direction of the acoustic signal by a blind sound source separation method using a plurality of microphones or by beamforming. connected to the camera,
an image database that stores image objects associated with acoustic tags;
further comprising an image object detection engine that uses the image database to detect one or more image objects within the video captured by the camera and identifies an acoustic tag of the image object;
3. The sound collection device according to claim 1 , wherein the acoustic tag transmitting means transmits the arrival direction of the acoustic signal associated with the acoustic tag specified by the image object detection engine. A system in which the sound collection device according to any one of claims 1 to 3 and a playback device equipped with a plurality of speakers are connected via a network,
The playback device is
a second acoustic database that accumulates acoustic signals for each acoustic tag;
acoustic tag receiving means for receiving the acoustic tag and the direction of arrival from the sound collection device;
It is characterized by having an environmental sound reproducing means for outputting, from a plurality of speakers, an environmental sound synthesized using a second acoustic database so that the acoustic signal linked to the acoustic tag can be heard from the direction of arrival of the acoustic tag. system. The acoustic tag transmitting means in the sound collection device further transmits the acoustic signal together with the acoustic tag and the direction of arrival,
5. The system according to claim 4 , wherein the environmental sound reproducing means in the reproducing device reproduces an acoustic signal received from a sound collection device instead of an acoustic signal tied to an acoustic tag. The acoustic tags and acoustic signals accumulated in the second acoustic database of the playback device are part or all of the acoustic tags and acoustic signals accumulated in the first acoustic database of the sound collection device,
Even if the acoustic tag stored in the second acoustic database of the playback device and the acoustic tag accumulated in the first acoustic database of the sound collection device are the same, it can be determined that the acoustic signals are based on different acoustic signals. 5. The system of claim 4 . A program that operates a computer installed in a sound collection device that transmits environmental sound signals collected by multiple microphones to a playback device that reproduces them through multiple speakers.
The sound collection device is connected to an environmental sensor that is associated with an acoustic tag and a direction of arrival,
a first acoustic database that accumulates acoustic signals for each acoustic tag;
a sound source separation means for detecting one or more acoustic signals inherent in the environmental sound signal and estimating the direction of arrival of each acoustic signal;
acoustic tag estimating means for estimating an acoustic tag of an acoustic signal using a first acoustic database;
The reproduction device reproduces the acoustic tag estimated by the acoustic tag estimating means, the arrival direction estimated by the sound source separation means, and the acoustic tag and arrival direction linked to the environmental sensor when receiving a predetermined signal from the environmental sensor. A program for a sound collection device, characterized in that the program causes a computer to function as an acoustic tag transmitting means to transmit an acoustic tag to a reproducing device, and causes a reproducing device to reproduce an acoustic signal associated with the acoustic tag from the direction of arrival. A program that receives an acoustic tag and a direction of arrival from the sound collection device according to any one of claims 1 to 3 and causes a computer installed in a playback device equipped with a plurality of speakers to function,
a second acoustic database that accumulates acoustic signals for each acoustic tag;
acoustic tag receiving means for receiving the acoustic tag and the direction of arrival from the sound collection device;
Using a second acoustic database, the computer functions as an environmental sound reproducing means for outputting from a plurality of speakers an environmental sound synthesized so that the acoustic signal linked to the acoustic tag can be heard from the direction of arrival of the acoustic tag. A playback device program featuring: In a sound collection method for a sound collection device that transmits environmental sound signals collected by multiple microphones to a playback device that reproduces them using multiple speakers,
The sound collection device is
It is connected to an acoustic tag and an environmental sensor linked to the direction of arrival.
a first acoustic database for accumulating acoustic signals for each acoustic tag;
a first step of detecting one or more acoustic signals inherent in the environmental sound signal from the environmental sound signal and estimating the direction of arrival of each acoustic signal;
a second step of estimating an acoustic tag of the acoustic signal using the first acoustic database;
Reproducing the acoustic tag estimated in the second step, the direction of arrival estimated in the first step, and the acoustic tag and direction of arrival associated with the environmental sensor when the predetermined signal is received from the environmental sensor. a third step of transmitting the sound to the device, and causing the playback device to play back the acoustic signal associated with the acoustic tag from the direction of arrival. A reproducing method for a reproducing device that receives an acoustic tag and a direction of arrival from a sound collection device according to any one of claims 1 to 3 and is equipped with a plurality of speakers,
The playback device is
a second acoustic database for accumulating acoustic signals for each acoustic tag;
a first step of receiving an acoustic tag and a direction of arrival from a sound collection device;
a second step of outputting from a plurality of speakers an environmental sound synthesized so that the acoustic signal associated with the acoustic tag can be heard from the direction of arrival of the acoustic tag using the second acoustic database; Featured playback method.

Images