JP2024007669A - Sound field reproduction program using sound source and position information of sound-receiving medium, device, and method - Google Patents

Abstract

To provide a sound field reproduction program with which it is possible to reproduce the sound field of a sound collection space for a sound-receiving medium even when a sound source and/or the sound-receiving medium is present at a discretionary position or movable.SOLUTION: The present program causes a computer to function as: position information acquisition means for acquiring measured position information at a sound source and a sound-receiving medium; input acoustic signal determination means for identifying, on the basis of the acquired position information, a microphone located in a direction leading to the corresponding position of the sound-receiving medium or a direction close by a prescribed distance or more to said direction in a sound-collection space, as seen from the sound source, and acquiring an acoustic signal that pertains to the sound collected by the identified microphone to determine an input acoustic signal; and output acoustic signal generation means for generating, on the basis of the acquired position information, an output acoustic signal, using the input acoustic signal, the output acoustic signal causing a sound equivalent to the collected sound to be outputted toward the sound-receiving medium from a direction leading toward the corresponding position of the sound source or a direction close by a prescribed distance or more to said direction in the reproduction space, as seen from the sound-receiving medium.SELECTED DRAWING: Figure 1

Description

The present invention relates to sound field reproduction technology including sound field reproduction.

In recent years, with the spread of web conferencing applications, remote session applications, and the like, services that provide a suitable or desired acoustic environment using microphones and speakers have become widely used. Additionally, the use of highly immersive remote conference systems and online concerts is being actively promoted. In providing such services, an important issue is to improve sound field reproduction (sound field reproduction) technology that reproduces (reproduces) the sound field related to the sound source in a reproduction field.

As the basis of this sound field reproduction (sound field reproduction) technology, for example, Non-Patent Document 1 introduces a sound field control method using a wave field synthesis method that spatially generates a wave field. Specifically, for example, the principle of boundary sound field control is explained by applying inverse system theory to the Kirchhoff-Helmholtz integral equation, which is a mathematical expression of Huygens' principle.

In addition, as a sound field reproduction method based on sound pressure gradient, impulse response, etc. as disclosed in Non-Patent Document 1, for example, Patent Document 1 discloses that the balance between recorded sound and reflected sound is A highly realistic sound field reproduction information transmitting device is disclosed that enables highly realistic sound field reproduction. Specifically, this device receives a dry source sound source signal, which is a signal acquired from a sound source in a no-acoustic environment, and sound field reproduction information, which is information for reproducing this dry source sound source signal in accordance with the receiving side environment. It is a device that transmits.

Here, the sound source has conventionally been assumed to be located outside the control area in which the sound should be reproduced, but on the other hand, for example, in Non-Patent Document 2, the sound field reproduction when the sound source is located within the control area is is being attempted. Specifically, we use an inverse filter to estimate a virtual sound source signal from an acoustic signal picked up by a microphone array consisting of multiple microphones, and create a virtual reality system with fewer restrictions on the sound source. A field reproduction method has been proposed.

Furthermore, for example, Non-Patent Document 3 describes a sound field that exhibits high robustness against disturbances in situations where disturbances inevitably occur, such as when a listener is present in the playback field where the sound is to be reproduced. A reproduction method is disclosed. In general, sound field reproduction is said to be completed by matching the manner of sound propagation in the sound collection field and the reproduction field. However, even if it is possible to reproduce the sound propagation in the sound collection field and the reproduction field in the absence of people, for example, the state of sound propagation changes when a person enters the reproduction field. In other words, it is inevitable that disturbances will occur due to the presence of people in the playback area.

Furthermore, this non-patent document 3 proposes a sound collection method using 24 acute directional microphones and a signal processing method for reproducing the signal generated by such sound collection. According to this signal processing method, even if the sound is only being collected and output, the impulse response can be measured and the sound component directly removed, for example, when the sound source is felt to be moving. It is said to be able to express spatial acoustic characteristics.

Further, for example, Patent Document 2 discloses a sound field reproduction device that employs a hard sphere model that includes a sound source and a hard sphere whose position is known, and reproduces a sound field at a listening position on the opposite side of the sound source with respect to the hard sphere. There is. Specifically, this device includes a transfer function storage section for supplying transfer functions obtained in advance at multiple locations on a predetermined line having a predetermined relationship with respect to a line connecting a rigid sphere in space and a sound source, and a sound field. a position estimating unit for determining information regarding a position to be reproduced; and a position estimating unit for determining information regarding a position to be reproduced, and receiving a corresponding transfer function from a transfer function storage unit based on the information regarding the determined position; and an output signal synthesis unit for converting according to a function, synthesizing and outputting an acoustic signal. With this functional configuration, it is possible to reproduce the sound field in real time in response to changes in the arrangement of objects in the space.

JP2005-086537A Japanese Patent Application Publication No. 2001-142471

Institute of Electronics, Information and Communication Engineers "Forest of Knowledge" Group 2 (Image, Sound, Language) - Volume 6 (Acoustic Signal Processing) - Chapter 7 Sound Field Reproduction, [online], [Retrieved June 27, 2020], Internet <URL: https://www.ieice-hbkb.org/files/ad_base/view_pdf.html?p=/files/02/02gun_06hen_07.pdf> Wataru Mizuno et al., “Theoretical study on free apex sound field reproduction system using microphone array,” IEICE Technical Report, Vol. 104, No. 614, pp. 7-12, 2005. Akira Omoto, “Recording and playback methods for creative reproduction of sound fields”, Proceedings of the 2022 Acoustical Society of Japan Spring Conference 1-12-10, March 2022

For example, the sound field reproduction method disclosed in Non-Patent Document 1 and Patent Document 1, which is performed based on pre-measured sound pressure gradients, impulse responses, etc., is a basic method aimed at improving reproduction accuracy. However, the sound pressure gradient and impulse response are easily influenced by the presence of a listener, and change significantly due to the movement of the listener. Therefore, with such a sound field reproduction method, it is difficult to increase robustness against disturbances caused by the presence of people in the reproduction field.

In addition, the sound field reproduction technology using acoustic signals collected by a microphone array disclosed in Non-Patent Document 2 is a method that can cope with the situation where the sound source is placed within the control area. It is not designed to generate a virtual sound source signal in response to the movement of the sound source. That is, as with the techniques disclosed in Non-Patent Document 1 and Patent Document 1, it is still difficult to reproduce a sound field that corresponds to the presence and movement of a listener.

On the other hand, the sound field reproduction technology disclosed in Non-Patent Document 3 collects sound using 24 acutely directional microphones, and is a highly robust method with respect to the presence of a listener. However, the listening position in the reproduction field is limited to the position corresponding to the position where the sharply directional microphone group is installed in the sound collection field. That is, since the auditory characteristics differ at other positions in the reproduction field, it must be said that it is difficult to reproduce the sound field in response to the movement of the listener.

On the other hand, the sound field reproduction device disclosed in Patent Document 2 does indeed reproduce changes in auditory characteristics due to changes in the listening position. However, in this sound field reproduction device, it is a major premise that the input sound source is a dry sound source recorded in advance, and that its position is known.

Therefore, the present invention provides a sound source located at any position within the sound collection space or movable within the sound collection space, and/or a sound receiver located at any position within the reproduction space. The purpose of the present invention is to provide a sound field reproduction program, device, and method capable of reproducing the sound field of a sound collection space for the sound receiver, even if the sound receiver exists or is movable within the reproduction space. do.

According to the present invention, there is provided a sound field reproduction program for reproducing a sound field related to a sound source in a sound collection space to a sound receiver in a reproduction space that has a correspondence relationship with the sound collection space in terms of position in the space. hand,
a position information acquisition means for acquiring information regarding the measured or set position of the sound source and the sound receiver;
At least one microphone located in a direction toward the corresponding position of the sound receiver as seen from the sound source in the sound collection space, or in a direction close to the sound direction until a predetermined condition is satisfied, based on the acquired information regarding the position. input acoustic signal determining means for determining an input acoustic signal by acquiring an acoustic signal related to the sound picked up by the identified microphone;
In the reproduction space, from the direction toward the corresponding position of the sound source as seen from the sound receiver, or from a direction close to the direction until a predetermined condition is satisfied, toward the sound receiver, the sound corresponding to or corresponding to the collected sound. A sound field reproduction program that causes a computer to function as an output acoustic signal generation means that generates an output acoustic signal in which a corresponding sound is output using the input acoustic signal based on the acquired information regarding the position. provided.

As a preferred embodiment of the sound field reproduction program according to the present invention, the position information acquisition means also acquires information regarding the output direction of the measured or set sound at the sound source,
The input acoustic signal determining means moves at least one other microphone located in the output direction or in a direction close to the output direction as seen from the sound source in the sound collection space to the acquired position. specifying based on the information and the information related to the output direction, obtaining another acoustic signal related to the sound picked up by the specified another microphone and determining another input acoustic signal,
It is also preferable that the output acoustic signal generating means generates the output acoustic signal by combining the other input acoustic signal and the input acoustic signal related to the sound picked up by the microphone.

Further, in the above embodiment, it is also preferable that the other microphone has higher directivity than the other microphone.

Furthermore, as another embodiment of the sound field reproduction program according to the present invention, the reproduction space is a real space,
The output acoustic signal generation means is configured to generate at least one speaker located in a direction toward a corresponding position of the sound source as viewed from the sound receiver in the reproduction space, or in a direction close to the sound source until a predetermined condition is satisfied. It is also preferable to specify the output sound signal based on information regarding the position and to generate the output acoustic signal to be supplied to the specified speaker.

In the other embodiment described above, the output acoustic signal generating means also outputs at least one other speaker located closer to the position of the sound receiver than the identified speaker or located further inward from the surrounding boundary. It is also preferable to identify and also generate the output audio signal to be supplied to the identified further speaker.

Furthermore, in the other embodiment described above, the sound source and the sound receiver are respectively movable within the sound collection space and the reproduction space, which are real spaces,
The input audio signal determination means identifies the microphone at the one time point based on the information regarding the position measured and acquired at the one time point by the position information acquisition means, and determines the input audio signal at the one time point. decide,
It is also preferable that the output acoustic signal generation means specifies the speaker at the one point in time based on information about the position acquired at the one point in time and generates the output acoustic signal at the one point in time. .

Furthermore, as yet another embodiment of the sound field reproduction program according to the present invention, the reproduction space is a virtual space related to sound image localization in a stereophone attached to the sound receiver,
The output acoustic signal generating means generates the sound signal toward the sound receiver from a direction toward the corresponding position of the sound source as seen from the sound receiver in the reproduction space, or from a direction close to the direction until a predetermined condition is satisfied. It is also preferable to generate an output acoustic signal to be supplied to a stereophone, in which a sound corresponding to or corresponding to the played sound is to be output.

In addition, in the embodiment using the above-mentioned "another microphone", the output acoustic signal generating means is configured to respond to the determined other input acoustic signal based on the distance between the position of the sound source and the position of the sound receiver. It is also preferable to perform amplitude adjustment processing to attenuate the amplitude accordingly, and to generate an output audio signal using the other input audio signal subjected to the processing.

Furthermore, in the embodiment using the above-mentioned "another microphone", the output acoustic signal generating means is configured to respond to the determined another input acoustic signal based on the proximity between the position of the sound source and the position of the sound receiver. Accordingly, a filtering process is performed to emphasize a predetermined high frequency band, or an impulse response between the position of the sound source and the position of the sound receiver is determined, and convolution processing is performed in the frequency domain related to the impulse response. It is also preferable to generate the output acoustic signal using the other input acoustic signal that has undergone the processing.

Furthermore, in the embodiment using the above-mentioned "another microphone", the output acoustic signal generating means determines the phase difference between the other input acoustic signal and the input acoustic signal based on the information regarding the position, It is also preferable to delay or advance one phase so as to eliminate the determined phase difference, and then combine the other input audio signal and the input audio signal to generate the output audio signal.

According to the present invention, there is also provided a sound field reproduction program for reproducing sound related to a sound source in a sound collection space to a sound receiver in a reproduction space that has a correspondence relationship with the sound collection space in terms of position in the space. There it is,
a position information acquisition means for acquiring information regarding the measured or set position of the sound source and the sound receiver, and information regarding the output direction of the measured or set sound at the sound source;
(a) At least one microphone located in the direction toward the corresponding position of the sound receiver as seen from the sound source in the sound collection space, or in a direction close to the sound direction until a predetermined condition is satisfied, is moved to the acquired position. (b) determining the input sound signal by determining the sound signal related to the sound picked up by the identified microphone based on the information; and (b) determining the output direction as seen from the sound source in the sound collection space. or at least one other microphone located in a direction close to the output direction that satisfies a predetermined condition, based on the obtained information regarding the position and the information regarding the output direction, and the identified another microphone input sound signal determining means for determining another input sound signal by acquiring another sound signal related to the sound collected by the input sound signal;
An output that combines the input acoustic signal and the other input acoustic signal to generate an output acoustic signal that results in outputting a sound equivalent to or corresponding to the collected sound toward the sound receiver. A sound field reproduction program is provided that causes a computer to function as an acoustic signal generating means.

According to the present invention, the present invention further provides a sound field reproduction program for reproducing a sound field related to a sound source in a sound collection space to a sound receiver in a reproduction space that has a correspondence relationship with the sound collection space in terms of position in the space. And,
a position information acquisition means for acquiring information regarding the measured or set position of the sound source and the sound receiver;
input acoustic signal determining means for determining an input acoustic signal by acquiring an acoustic signal related to a sound picked up by at least one microphone capable of picking up the sound from the sound source;
At least one speaker located in the direction toward the corresponding position of the sound source as viewed from the sound receiver in the playback space, or in a direction close to the sound source until a predetermined condition is satisfied, based on the acquired information regarding the position. A sound field reproduction program is provided that causes a computer to function as an output acoustic signal generation means for generating an output acoustic signal to be specified and supplied to the specified speaker.

According to the present invention, the present invention further provides sound field reproduction for reproducing a sound field related to a sound source in a sound collection space to a sound receiver in a reproduction space that has a correspondence relationship with the sound collection space in terms of position in the space. A device,
a position information acquisition means for acquiring information regarding the measured or set position of the sound source and the sound receiver;
At least one microphone located in a direction toward the corresponding position of the sound receiver as seen from the sound source in the sound collection space, or in a direction close to the sound direction until a predetermined condition is satisfied, based on the acquired information regarding the position. input acoustic signal determining means for determining an input acoustic signal by acquiring an acoustic signal related to the sound picked up by the identified microphone;
In the reproduction space, from the direction toward the corresponding position of the sound source as seen from the sound receiver, or from a direction close to the direction until a predetermined condition is satisfied, toward the sound receiver, the sound corresponding to or corresponding to the collected sound. There is provided a sound field reproducing device comprising an output acoustic signal generating means for generating an output acoustic signal in which a corresponding sound is to be output, using the input acoustic signal based on the acquired information regarding the position. Ru.

According to the present invention, there is also a sound field reproduction system for reproducing a sound field related to a sound source in a sound collection space to a sound receiver in a reproduction space that corresponds to the sound collection space in terms of position in the space. And,
a position information acquisition means for acquiring information regarding the measured or set position of the sound source and the sound receiver;
At least one microphone located in a direction toward the corresponding position of the sound receiver as seen from the sound source in the sound collection space, or in a direction close to the sound direction until a predetermined condition is satisfied, based on the acquired information regarding the position. input acoustic signal determining means for determining an input acoustic signal by acquiring an acoustic signal related to the sound picked up by the identified microphone;
In the reproduction space, from the direction toward the corresponding position of the sound source as seen from the sound receiver, or from a direction close to the direction until a predetermined condition is satisfied, toward the sound receiver, the sound corresponding to or corresponding to the collected sound. A sound field reproduction system is provided, comprising an output acoustic signal generating means for generating an output acoustic signal in which a corresponding sound is output, using the input acoustic signal based on the acquired information regarding the position. Ru.

According to the present invention, the computer further reproduces a sound field related to a sound source in a sound collection space to a sound receiver in a reproduction space that corresponds to the sound collection space in terms of position in the space. A sound field reproduction method comprising:
obtaining information regarding the measured or set position of the sound source and the sound receiver;
At least one microphone located in a direction toward the corresponding position of the sound receiver as seen from the sound source in the sound collection space, or in a direction close to the sound direction until a predetermined condition is satisfied, based on the acquired information regarding the position. determining an input acoustic signal by acquiring an acoustic signal related to the sound picked up by the identified microphone;
In the reproduction space, from the direction toward the corresponding position of the sound source as seen from the sound receiver, or from a direction close to the direction until a predetermined condition is satisfied, toward the sound receiver, the sound corresponding to or corresponding to the collected sound. A sound field reproduction method is provided, which includes the step of generating an output acoustic signal in which a corresponding sound is to be output, using the input acoustic signal based on the acquired information regarding the position.

According to the sound field reproduction program, device, and method of the present invention, even when the sound source is located at an arbitrary position within the sound collection space or is movable within the sound collection space, and/or the sound receiver Even if the sound receiver is located at an arbitrary position within the reproduction space or is movable within the reproduction space, it is possible to reproduce the sound field of the sound collection space for the sound receiver.

FIG. 1 is a functional block diagram showing the functional configuration of an embodiment of a sound field reproduction device according to the present invention. FIG. 2 is a schematic diagram of a sound collection space for explaining an embodiment of microphone identification processing according to the present invention. FIG. 2 is a schematic diagram of a sound collection space for explaining an embodiment of input acoustic signal determination processing (acoustic signal mixing processing) according to the present invention. FIG. 2 is a schematic diagram of a playback space for explaining an embodiment of speaker identification processing and speaker panning processing according to the present invention. FIG. 2 is a schematic diagram of a reproduction space for explaining an embodiment of ceiling speaker identification processing and ceiling speaker panning processing according to the present invention.

Hereinafter, embodiments of the present invention will be described in detail using the drawings.

[Sound field reproduction device]
FIG. 1 is a functional block diagram showing the functional configuration of an embodiment of a sound field reproduction device according to the present invention. Note that the same figure also shows an embodiment of a sound collection space and a reproduction space according to the present device.

A sound field reproducing device 1 as an embodiment of the present invention shown in FIG. In the embodiment, it is a device that can reproduce music to a listener.

Here, the sound collection space and the reproduction space are spaces that have a corresponding relationship with respect to their spatial positions (they have spatial positions that correspond one-to-one to each other). therefore,
(a) The position of the sound receiver (listener) in the sound collection space, and the relative position and direction (azimuth, direction) of the sound receiver (listener) as seen from the sound source (speaker);
(b) The position of the sound source (speaker) in the reproduction space, as well as the relative position and direction (azimuth, orientation) of the sound source (speaker) as seen from the sound receiver (listener) can be determined. It has become.

Here, in this embodiment, in order to further improve the reproduction accuracy of the sound field, the distance between the sound source (speaker) and the sound receiver (listener) and the sound source ( The directions (azimuth, direction) of the speaker) and the sound receiver (listener) are also set to match. However, the size and shape of the sound collection space (as an area partitioned by a group of microphones, which will be described later), and the size and shape of a reproduction space (as an area partitioned by a group of speakers, which will be described later) are as follows: They may be different from each other.

Furthermore, in this embodiment, the sound collection space and the playback space are a sound collection room and a playback room, respectively, and a plurality of microphones (hereinafter abbreviated as microphones) (2, 3) are located at the boundaries (in the horizontal plane) of the rooms. and a plurality of speakers (5) are installed. Furthermore, in this embodiment, both the sound collection space and the reproduction space are equipped with at least one depth sensor 4 (in this embodiment, two capable of depth measurement in opposite directions) at the boundary of the room, and The position of a sound source (speaker) in space and the position of a sound receiver (listener) in playback space can be measured.

Furthermore, in this embodiment, both the sound source (speaker) and the sound receiver (listener) are movable (for example, walking), and the depth sensor 4 is capable of measuring their positions in real time. . However, of course, it is also possible for one or both of the sound source (speaker) and the sound receiver (listener) to remain at a fixed position without moving (for example, sitting at a certain position). It is. In this case, the device 1 is capable of acquiring information related to this set fixed position (for example, through device input by the setter).

Under such circumstances, the sound field reproduction device 1 performs the following steps in order to perform sound field reproduction processing.
(A) a position information acquisition unit 111 that acquires “information related to the position” measured (or set) in the sound source (speaker) and the sound receiver (listener);
(B) Obtain at least one microphone located in the sound collection space in a direction toward the corresponding position of the sound receiver (listener) as seen from the sound source (speaker), or in a direction close to this direction until a predetermined condition is met. an input audio signal determining unit 112 that determines an "input audio signal" by determining an audio signal related to "sound collected" by the identified microphone based on the "information regarding the position"identified;
(C) In the reproduction space, from the direction toward the corresponding position of the sound source (speaker) as seen from the sound receiver (listener), or from a direction close to this direction until a predetermined condition is met, toward the sound receiver (listener) Then, an "output acoustic signal" that will output a sound equivalent to or corresponding to the "collected sound" is generated using the "input acoustic signal" based on the acquired "position information". It is characterized by having an output acoustic signal generation section 113 that performs the following steps.

Here, in the sound collection space of this embodiment shown in FIG. In FIG. 1, there are two microphones 3a and 3b located in the nearest direction and the second nearest direction. That is, in this embodiment, two microphones 3a and 3b are selected from a plurality of microphones installed surrounding the speaker based on "information related to the speaker's position" and "information related to the listener's position". (specified) and the "input acoustic signal" is determined from the acoustic signals related to the sounds collected by these microphones 3a and 3b.

Furthermore, as described later, in the reproduction space of this embodiment shown in FIG. The acoustic signals are to be supplied to the two speakers 5f and 5g selected (specified) based on "information related to the position of the listener" and "information related to the position of the speaker."

Here, these two speakers 5f and 5g, which will be explained in detail later, are speakers that are located close to the speaker's corresponding position as viewed from the listener until a predetermined condition is satisfied, and are so-called specific speakers. The two microphones 3a and 3b are positioned to face each other (with the speaker and listener in between). Therefore, the speakers 5f and 5g that have received the "output acoustic signal" can output sounds equivalent to or corresponding to the "sound collected" by the specified microphones 3a and 3b. It is.

Although this is also an embodiment that will be described in detail later, the output acoustic signal generation unit of (C) above assumes that the playback space is a virtual space related to sound image localization in a stereophone (e.g., headphones) worn by a listener. 113 corresponds to "collected sound" or is directed toward the listener from a direction toward the corresponding position of the speaker as seen from the listener in this reproduction space, or from a direction close to this direction until a predetermined condition is satisfied. An "output audio signal" may be generated to be supplied to the attached stereophone, from which the corresponding sound will be output.

In any case, the sound field reproduction device 1 specifies (selects) the microphone based on the acquired "information related to the speaker's position" and "information related to the listener's position" and "input acoustic signal ” and then generates the “output acoustic signal,” so even if the sound receiver (listener) is located at an arbitrary position within the playback space or is able to move within the playback space, this receiver It is possible to suitably reproduce the sound field of the sound collection space for the sound body (listener), for example (with a high degree of realism). Furthermore, even if the sound source (speaker) is located at an arbitrary position within the sound collection space or is movable within the sound collection space, the sound of the sound collection space may also be affected by the sound receiver (listener). It is also understood that the scene can be suitably reproduced, for example (with a high degree of realism).

By the way, as mentioned above, if the sound source (speaker) and the sound receiver (listener) are movable within the sound collection space and the reproduction space, which are real spaces, respectively,
(a) The input acoustic signal determination unit 112 in (B) above determines the position at this one time based on the “position information” measured and acquired at one time in the position information acquisition unit 111 in (A). identifying a microphone and determining an "input acoustic signal" for this one point in time;
(b) The output acoustic signal generation unit 113 in (C) above specifies the speaker at this one point in time based on the “position information” acquired at this one point in time, and specifies the “position information” at this one point in time. It is also preferred to generate an output acoustic signal.
In this case, for example, it is possible to reproduce the sound field in the sound collection space, which changes moment by moment as the sound source (speaker) moves, to the sound receiver (listener) moving within the playback space, almost in real time. It is.

Further, in this embodiment, the microphones (2, 3) and the speaker (5) are installed at fixed positions at the boundaries of the sound collection space and the reproduction space, respectively, but they are not fixed and their positions change. It may be something. That is, at least one of the plurality of microphones (2, 3) may be movable on the boundary of the sound collection space, or may be a microphone attached to a robot that can move within the sound collection space. There may be. Furthermore, at least one of the plurality of speakers (5) may also be movable on the boundary of the playback space, or may be a speaker attached to a robot that is movable within the playback space. .

In this case, the input acoustic signal determination unit 112 of (B) above selects a sound receiver (listener) from a group of microphones including moving microphones at one point in time, for example, as seen from the sound source (speaker). A microphone located in a direction toward the corresponding position or in a direction close to this direction until a predetermined condition is satisfied is specified. Here, for example, a microphone located near a position that meets the above conditions may be moved to the relevant position and then identified. Furthermore, the output acoustic signal generation unit 113 of (C) above, for example, at this one time point, moves in a direction toward the corresponding position of the sound source (speaker) as seen from the sound receiver (listener), or until a predetermined condition is satisfied in this direction. This means identifying a speaker located in a direction close to . Here, for example, a speaker located near a position that satisfies the above conditions may be moved to the corresponding position and then identified.

Furthermore, in the present invention, the input acoustic signal determining section 112 of the above (B), which is a component of the sound field reproduction device 1, and the output acoustic signal generating section 113 of the above (C) are provided in separate devices. It is also possible to take for example,
(a) A device equipped with a position information acquisition unit 111 and an input acoustic signal determination unit 112 installed in or near a sound collection space;
(b) A device installed in or near the playback space and equipped with a position information acquisition unit 111 and an output acoustic signal generation unit 113, which exchanges information with the device in (a) above by communication. A sound field reproduction system according to the present invention may be configured, including a device capable of.

[Device configuration, sound field reproduction program/method]
Hereinafter, the functional configuration of the sound field reproduction device 1 as an embodiment of the present invention will be explained in more detail. Similarly, in the functional block diagram of FIG. 1, the sound field reproduction device 1 includes a communication interface 101, a keyboard (KB)/display (DP) 102, and a processor/memory (arithmetic processing system with memory function). Here, the processor memory stores the sound field reproduction program according to the present invention, has a computer function, and executes the sound field reproduction process by executing the sound field reproduction program.

Further, from this, the sound field reproduction device 1 may be a device dedicated to sound field reproduction processing, but may also be a general-purpose cloud server or a non-cloud server equipped with the sound field reproduction program according to the present invention. Furthermore, it is also possible to use a personal computer (PC), a notebook or tablet computer, a smartphone, or even a wearable device such as a head mounted display (HMD).

In addition, the processor memory includes a position information acquisition section 111, an omnidirectional microphone identification section 112a, an input mixing section 112b, a sharply directional microphone identification section 112c, and an input audio mixing section 112d as functional components. The determination unit 112 communicates with an output acoustic signal generation unit 113 including an amplitude adjustment unit 113a, a tone adjustment unit 113b, an output mixing unit 113c, a speaker (SP) specific panning unit 113d, and a near speaker (SP) specific panning unit 113e. It has a control section 121 and an input/output control section 122. Note that these functional components can be regarded as functions realized by executing the sound field reproduction program according to the present invention stored in the processor memory. Furthermore, the process flow shown by connecting the functional components of the sound field reproducing device 1 with arrows in the functional block diagram of FIG. 1 can be understood as an embodiment of the sound field reproducing method according to the present invention.

<Location information acquisition means>
Similarly, in the functional block diagram of FIG. 1, the position information acquisition unit 111 in this embodiment:
(a) Information related to the measured or set position of the speaker who is the sound source and the listener who is the sound receiver, in this embodiment, "position coordinate information"; and (b) Information about the measured or set position of the speaker Alternatively, information related to the set output direction of the sound, in this embodiment, "face azimuth information" which is information related to the direction (direction) of the speaker's face.
get.

Specifically, in this embodiment, both the sound collection room (sound collection space) and the playback room (playback space) have the information (a) and (b) above in the room at the boundary in their own horizontal plane. Two depth sensors 4 are provided that are capable of measuring depth in opposite directions (to ensure reliable acquisition). Here, by performing image recognition processing that can recognize people and faces on the depth images obtained by these depth sensors 4, position coordinate information and face azimuth angle information of the speaker in the sound collection room are obtained. Position coordinate information of the listener in the playback room is generated. The position information acquisition unit 111 may acquire this information from an image recognition processing device connected to the depth sensor 4, or may perform such image recognition processing on the depth image obtained from the depth sensor 4 by itself. It is also possible to generate and obtain this information.

Furthermore, as a modification, a visible light (and infrared) camera is used instead of the depth sensor 4 or in addition to the depth sensor 4, and image recognition processing that can recognize people and faces is performed on images obtained by these cameras. By doing so, position coordinate information and face azimuth angle information of the speaker in the sound collection room and position coordinate information of the listener in the reproduction room may be generated. For example, it is also possible to employ Microsoft's Azure Kinect DK as the depth sensor 4. The Azure Kinect DK is equipped with a ToF depth sensor and an RGB camera, and can generate position coordinate information and face azimuth information by linking with an image recognition cloud service. A further modification is that the speaker or listener is carrying a terminal equipped with a magnetic sensor, acceleration sensor, or electrostatic sensor that can determine the position in the room, and information regarding the position can be obtained from this terminal. is also possible.

In this embodiment, a plurality of (eight in FIG. 1) omnidirectional microphones 2a to 2h are installed in the sound collection room (sound collection space) so as to surround the inside of the space at the boundary within its own horizontal plane. , and a plurality of (eight in FIG. 1) sharply directional microphones 3a to 3h are installed. Here, the sharply directional microphones 3a to 3h are "different microphones" from the omnidirectional microphones 2a to 2h, and naturally have higher directivity than the omnidirectional microphones 2a to 2h. . For example, the omnidirectional microphones 2a to 2h may be omnidirectional ambient microphones, and the sharply directional microphones 3a to 3h may be gun microphones.

Further, in this embodiment, one omnidirectional microphone 2 and one sharply directional microphone 3 are installed as a pair at one position, and the installation positions of these pairs are rectangular as a whole. forming the boundary of However, of course, the arrangement of the microphones is not limited to this. For example, the omnidirectional microphone 2 and the sharply directional microphone 3 may be installed at different positions, for example, at alternate positions. The installation position may form, for example, a circular, oval, or polygonal (including triangular) boundary as a whole.

Furthermore, in this embodiment, acoustic signals related to sounds picked up by these microphones, position coordinate information and face azimuth angle information (or depth image information) generated using the depth sensor 4 described above are, for example, A sound signal is received by the communication interface 101 of the sound field reproduction device 1 via the communication network from an acoustic signal/location information management device (not shown) installed in or near the sound room (sound collection space), and the same The communication control section 121 of the device 1 is to provide the input acoustic signal determination section 112 and the position information acquisition section 111. Furthermore, the position information acquisition unit 111 acquires information regarding the set (or measured) positions of the omnidirectional microphone 2 and the acutely directional microphone 3, and the set (or measured) position of the microphone 5. Information related to this is also received from the sound collection room side and the playback room side (or by direct input to the device 1), and provides this information to the input audio signal determining section 112 and the output audio signal generating section 113.

<Input acoustic signal determining means>
Similarly, in the functional block diagram of FIG. 1, the omnidirectional microphone identification section 112a of the input acoustic signal determination section 112 is
(a) In the sound collection room (sound collection space), located in the direction toward the corresponding position of the listener (sound receiver) as viewed from the speaker (sound source), or in a direction close to this direction until a predetermined condition is satisfied. At least one (in this embodiment, two) omnidirectional microphones 2 is identified (selected) based on the acquired positional coordinate information, and the identified two omnidirectional microphones 2 (2a and 2b in FIG. 1) are used to collect information. Acquire an acoustic signal related to the emitted sound.

Here, the omnidirectional microphone 2 is a microphone that captures the acoustics (sound field) of the sound collection space itself. Therefore, the omnidirectional microphone 2 (2a and 2b) specified above is a microphone that captures the acoustics (sound field) of the sound collection space itself. It is possible to pick up sound that corresponds to the sound (sound field) that the sound receiver (sound receptor) perceives with the sense of hearing.

Further, the acute directional microphone identification unit 112c of the input acoustic signal determination unit 112,
(b) At least one book located in the sound collection room (sound collection space) in the direction in which the speaker's face is facing, or in a direction close enough to this direction to satisfy a predetermined condition, when viewed from the speaker (sound source). In the embodiment, two sharply directional microphones 3 are identified (selected) based on the acquired position coordinate information and face azimuth information, and the sound is collected by the identified sharply directional microphones 3 (3c and 3d in FIG. 1). Acquire an acoustic signal related to the sound.

Here, the sharply directional microphone 3 is a microphone that captures the sound itself from the sound source, so the sharply directional microphone 3 (3c and 3d) specified as above is used to direct the listener (sound receiver) to the listener (sound receiver). This makes it possible to reliably capture the sound (voice) from the speaker (sound source) that should be delivered.

By the way, the information specifying the identified microphone is notified to the sound collection room side, and only the acoustic signal related to the sound collected by the identified omnidirectional microphone 2 or acute directional microphone 3 is received from the sound collection room side. Alternatively, it is also possible to receive all the acoustic signals related to the sounds picked up by these microphones from the sound collection room side, and then select the acoustic signals related to the identified microphone from there.

FIG. 2 is a schematic diagram of a sound collection space for explaining one embodiment of the microphone identification process according to the present invention. Here, an xy position coordinate system that defines a position in a horizontal plane is set in the sound collection room, which is a sound collection space.

As shown in FIG. 2, in this embodiment, the omnidirectional microphone identification unit 112a (FIG. 1) detects (a) from the speaker's position S(x_s, y_s) to the listener's position R(x_r, y_r). Calculate the intersection point AM(x_amb, y_amb) between the line segment and the boundary after further extending the line segment connecting
(b) The direction from position S(x_s, y_s) to its own installation position is closest to the direction from position S(x_s, y_s) to position AM(x_amb, y_amb) (the angle between them is the smallest) ) Specify (select) the omnidirectional microphone 2b and the omnidirectional microphone 2a that is the second closest (the angle between them is the second smallest).

Further, in this embodiment, the sharp directional microphone identification unit 112c (FIG. 1)
(c) Calculate the intersection point GM(x_gun, y_gun) between the boundary and the line segment extending from the speaker's position S(x_s, y_s) in the direction of the face azimuth angle θ_s,
(d) The direction from position S(x_s, y_s) to its own installation position is closest to the direction from S(x_s, y_s) to GM(x_gun, y_gun) (the angle between them is the smallest) The directional microphone 3d and the sharply directional microphone 3c that is the second closest (the angle between them is the second smallest) are specified (selected).

Here, for both the omnidirectional microphone 2 and the sharply directional microphone 3, for example, only the closest one can be specified (selected). However, in this embodiment, the above-mentioned steps are taken so that a natural sound field is reproduced even when the speaker's position changes from moment to moment, that is, so that changes in the sound field due to the speaker's movement are reproduced naturally. We have identified (selected) two such things. Of course, it is also possible to specify (select) three or more for the same reason.

Returning to the functional block diagram of FIG. 1, the input mixing section 112b of the input acoustic signal determining section 112 uses the omnidirectional microphones (2a and 2b in FIG. 1) specified (selected) by the omnidirectional microphone specifying section 112a. Such acoustic signals are combined (eg, mixed) to generate a first input acoustic signal. In addition, the input mixing section 112d of the input acoustic signal determining section 112 combines (for example, mixes) the acoustic signals related to the sharply directional microphones (3c and 3d in FIG. 1) specified (selected) by the sharply directional microphone specifying section 112c. ) generating a second input acoustic signal.

FIG. 3 is a schematic diagram of a sound collection space for explaining an embodiment of input acoustic signal determination processing (acoustic signal mixing processing) according to the present invention. Incidentally, the microphone settings and the states of the speakers and listeners in the sound collection room (sound collection space) shown in the figure are the same as those in the sound collection room shown in FIG. 2.

First, using FIG. 3(A), the acoustic signal (amplitude intensity) t_2a(p) related to the specified (selected) omnidirectional microphone 2a performed by the input mixing unit 112b (FIG. 1), The process of generating the first input acoustic signal (amplitude intensity) t_amb(p) by mixing the acoustic signal (amplitude intensity) t_2b(p) related to the specified (selected) omnidirectional microphone 2b will be explained. do. Here, p is a parameter representing a time point or a sampling index corresponding to the time point.

As shown in Figure 3(A),
(a) Let α be the angle formed by the line segment connecting the speaker's position S and the position P2a of the omnidirectional microphone 2a and the line segment connecting the speaker's position S and the position P2b of the omnidirectional microphone 2b,
(b) Let β be the angle formed by the line segment connecting the speaker's position S and the position P2a of the omnidirectional microphone 2a and the line segment connecting the speaker's position S and position AM (explained in Figure 2). In case,
After calculating these α and β, the first input acoustic signal t_amb(p) is calculated using the following formula (1) t_amb(p)=t_2a(p)×(sin(α/2)+sin(α/2− β))
+t_2b(p)×(sin(α/2)−sin(α/2−β))
It is derived by

In the above equation (1), the coefficients of t_2a(p) and t_2b(p) in the first term on the right-hand side are the proportion to be mixed of the acoustic signals related to the omnidirectional microphone 2a and the acoustic signal related to the omnidirectional microphone 2b, respectively. This is the ratio at which the signals should be mixed. Incidentally, when β = 0 and β = α, the first input acoustic signal t_amb(p) is a constant multiple of t_2a(p) and a constant multiple of t_2b(p), respectively, and two microphones are specified (selected) in advance. ), but essentially the acoustic signal related to one (one) microphone is used as the input acoustic signal. In other words, in this embodiment, when the speaker's position S, the listener's position R, and the microphone position are on the same straight line in this order, the setting is such that exactly that (one) microphone is specified (selected). It's okay to stay.

Next, similarly using FIG. 3(A), the acoustic signal t_3c(p) related to the specified (selected) sharp directional microphone 3c and the specified (selected) ) and the acoustic signal t_3d(p) from the sharply directional microphone 3d to generate the second input acoustic signal t_gun(p) will be described.

As shown in Figure 3(A),
(a) Let δ be the angle formed by the line segment connecting the speaker's position S and the position P3c of the sharply directional microphone 3c and the line segment connecting the speaker's position S and the position P3d of the sharply directional microphone 3d,
(b) Let γ be the angle formed by the line segment connecting the speaker's position S and the position P3c of the sharply directional microphone 3c, and the line segment connecting the speaker's position S and position GM (explained in FIG. 2). ,
After calculating these δ and γ, the second input acoustic signal t_gun(p) is calculated using the following formula (2) t_gun(p)=t_3c(p)×(sin(δ/2)+sin(δ/2−) γ))
+t_3d(p)×(sin(δ/2)−sin(δ/2−γ))
It is derived by Here, regarding the number of microphones to be specified (selected) in generating the second input acoustic signal t_gun(p), the situation is similar to that described in relation to equation (1) above.

<Output acoustic signal generation means>
Returning to the functional block diagram of FIG. 1, in this embodiment, the amplitude adjustment unit 113a of the output acoustic signal generation unit 113 determines the position of the speaker (sound source) and the listening position for the second input acoustic signal t_gun(p). An amplitude adjustment process is performed to attenuate the amplitude according to the distance from the position of the person (sound receiver). Note that this processing target is the second input acoustic signal t_gun(p) (related to the sharply directional microphone 3), which is the sound picked up by the sharply directional microphone 3 (3c and 3d) and delivered to the listener. In fact, the closer the speaker and listener are to each other, the louder the sound (voice) from the speaker is, and conversely, the farther away the listener is, the lower the volume is. This is because adjusting the amplitude improves the reproduction accuracy of the sound field and enhances the sense of presence. Of course, it is also possible to omit such amplitude adjustment processing in order to shorten the calculation time.

Generally, distance attenuation y (dB) at a position of distance d (from the sound source) in an acoustic signal is determined by y=20log ₁₀ (d_0/d), where d_0 is the reference distance. Generally, the amplitude of an acoustic signal and decibel dB have a relationship of (amplitude)=10 ^dB/20 . Therefore, specifically, the amplitude adjustment unit 113a converts the amplitude adjustment-processed second input acoustic signal tj_gun(p) into the following equation (3) tj_gun(p)=t_gun(p)×amp
amp=10y ^/20
y＝20log ₁₀ (d_SR／d_SG)
can be generated by In the above equation (3), the (reference) distance d_SR and distance d_SG are the distance between the speaker's position S and the listener's position R, and the speaker's position S, respectively, as shown in FIG. 3(B). and the distance from the position GM.

Similarly, in the functional block diagram of FIG. 1, the timbre adjustment section 113b of the output acoustic signal generation section 113 is a tone adjustment section 113b that collects the sound from the speaker (sound source) to be delivered to the listener (sound receiver). A timbre adjustment process is performed on the second input acoustic signal related to the directional microphone 3 so that it becomes the timbre that a listener (sound receiver) would actually receive. Here, it is generally known that high frequency components of sound attenuate as it propagates (as it moves away from the sound source). Therefore, for example, the closer the listener is to the speaker, the closer the listener is to the speaker by emphasizing a predetermined high frequency band in the input acoustic signal. This makes it possible to improve the reproduction accuracy of the sound field and enhance the sense of presence.

As such timbre adjustment processing, the timbre adjustment section 113b adjusts the amplitude adjustment-processed second input acoustic signal tj_gun(p) to
(A: Biquadratic filtering processing for time information) Biquadratic filtering processing is performed to emphasize a predetermined high frequency band depending on the proximity between the position of the speaker (sound source) and the position of the listener (sound receiver). to generate a filtered second input acoustic signal tf_gun(p), or
(B: Convolution processing in the frequency domain) Determine the impulse response between the position of the speaker (sound source) and the position of the listener (sound receiver), and perform convolution processing on this impulse response in the frequency domain. , this convolution process is performed to generate the convolution-processed second input acoustic signal tc_gun(p).
Incidentally, it is of course possible to omit such timbre adjustment processing in order to shorten the calculation time.

(Biquadratic filtering processing on time information)
First, the above (a) will be explained in detail. For example, a high shelf filter is adopted as a biquad filter, and the settings of this filter are as follows, assuming that the sound output from the speaker is a speech sound that includes consonant components in the 4kHz band.
・Freq (threshold for boosting frequency bands higher than this value) = 4000 (Hz)
・Q (width of frequency band peak) = 1.0
・Gain (filter gain) = amp where amp = 10 ^y/20 , y = 20log ₁₀ (d_SR/d_SG)
etc. Then from here, the biquad transfer function:
(4) H(z)=(b0+b1×z ^-1 +b2×z ^-2 )/(a0+a1×z ^-1 +a2×z ^-2 )
The denominator and numerator coefficients a0, a1, a2, b0, b1, and b2 are calculated, and the biquadratic filtered second input acoustic signal tf_gun(p) is calculated using the following formula (5) tf_gun(p) )=(1/a0)×(b0×tj_gun(p)+b1×tj_gun(p-1)+
b2×tj_gun(p-2)－a1×tf_gun(p-1)－a2×tj_gun(p-2))
It is generated using .

Here, the calculation method of coefficients a0, a1, a2, b0, b1, and b2 in various filters is described, for example, in a non-patent document: “Cookbook formula for audio equalizer biquad filter coefficients”, [online], [June 2020] 27th search], Internet <URL: https://webaudio.github.io/Audio-EQ-Cookbook/audio-eq-cookbook.html>, non-patent literature: "++C++; // Unidentified Flight C" , [online], [Retrieved June 27, 2020], Disclosed on the Internet <URL: https://ufcpp.net/study/sp/digital_filter/biquad/>. Incidentally, it is preferable that the selection of the filter as described above and the filter settings such as Freq, Q, and Gain are performed as appropriate depending on what kind of sound field is desired to be reproduced and the situation in the sound collection space and reproduction space. .

(Convolution processing in frequency domain)
Next, the above (a) will be explained in detail. First, prepare an impulse response that reflects the absorption characteristics of air, or an impulse response of a filter with multiple peaks that simulates the absorption characteristics of air, between the speaker's position S and the listener's position R. , this is defined as i(p). Such an impulse response i(p) is a quantity that depends on the distance d_SR (FIG. 3(B)). Here, it is also preferable to appropriately set the time length N of the impulse response i(p) depending on the capability of the computer. For example, typically N = 1024, but if a PC with low processing speed is used, N = 2048 or N = 4096 may be used.

Next, the impulse response i(p) is Fourier transformed to derive the transfer function I(ω) (ω is the frequency), and for the second input acoustic signal tj_gun(p) cut out at the set time length N. A Fourier transform is performed to calculate the second input acoustic signal tj_gun(ω) after the Fourier transform. As a result, the second input acoustic signal tc_gun(p) that has been subjected to convolution processing in the frequency domain can be calculated using the following equation (6) tc_gun(p)=IF[tc_gun(ω)]
tc_gun(ω)＝I(ω)＊tj_gun(ω)
It can be generated by Here, IF[] is an inverse Fourier transform operator, and * is a convolution integral operator.

Above, we have explained the two processes (a) and (b) as timbre adjustment processes, but in situations where real-time sound field reproduction is important, the calculation time is relatively small (A: duality in time information). On the other hand, in situations where you want to perform complex filtering processing (for example, processing that emphasizes multiple identified different high frequency bands) to improve sound field reproduction and enhance the sense of presence, (I: Convolution processing in the frequency domain) may also be employed. That is, it is also preferable that the settings be such that either one of the timbre adjustment processes can be selected and executed depending on the situation.

Similarly, in the functional block diagram of FIG. 1, the output mixing section 113c of the output acoustic signal generation section 113 mixes the generated first input acoustic signal (t_amb(p)) and the generated second input acoustic signal (tf_gun). (p), tc_gun(p)) are combined (mixed) to generate an output acoustic signal t_out(p). In other words, the following equation (7) t_out(p)=t_amb(p)+tf_gun(p) or
t_out(p)＝t_amb(p)＋tc_gun(p)
The output acoustic signal t_out(p) is derived by .

In one aspect, the output mixing unit 113c outputs a first input acoustic signal (t_amb(p)) and a second input acoustic signal (tf_gun(p) based on the speaker's position S and the listener's position R. ), tc_gun(p)), and after delaying or advancing one phase so as to eliminate the determined phase difference, the first input acoustic signal (t_amb(p)) and It is also preferable to generate the output acoustic signal t_out(p) by combining (mixing) the two input acoustic signals (tf_gun(p), tc_gun(p)).

Incidentally, the above phase difference is determined by (a) the distance d_SA between the speaker's position S and the position AM related to the first input acoustic signal (Fig. 3(B)), and (b) related to the second input acoustic signal. This occurs when the distance d_SG (FIG. 3(B)) between the speaker's position S and the position GM is different, and causes interference when mixing both signals. However, of course, it is also possible to not perform such a phase difference adjustment process, which requires time and effort for appropriate adjustment, and to reduce the calculation time accordingly.

Specifically, the output mixing unit 113c first calculates the phase difference p_th using the following formula (8) p_th=fs×(d_SA−d_SG)/c
Calculated by Here, c (m/sec) is the speed of sound, and fs (Hz) is the sampling frequency.

Next, in the case of d_SA≧d_SG, the output mixing unit 113c adds a delay to the second input acoustic signal (tf_gun(p), tc_gun(p)) related to the distance d_SG (position GM) to calculate the phase difference. After eliminating the problem, the output acoustic signal t_out(p) is generated. In other words, the following formula (if d_SA≧d_SG)
(9) t_out(p)=t_amb(p)+tf_gun(p+p_th) or
t_out(p)＝t_amb(p)＋tc_gun(p＋p_th)
The output acoustic signal t_out(p) is derived by . On the other hand, in the case of d_SA<d_SG, a delay is added to the first input acoustic signal (t_amb(p)) related to the distance d_SA (position AM) to eliminate the phase difference, and then the output acoustic signal is Generate t_out(p). In other words, the following formula (if d_SA<d_SG)
(9') t_out(p)=t_amb(p+p_th)+tf_gun(p) or
t_out(p)＝t_amb(p＋p_th)＋tc_gun(p)
Thus, the output acoustic signal t_out(p) is derived.

Similarly, in the functional block diagram of FIG. 1, the speaker specific panning section 113d of the output acoustic signal generation section 113 is
(a) At least one speaker located in a direction toward the corresponding position of the speaker (sound source) as viewed from the listener (sound receiver) in the reproduction room (reproduction space), or in a direction close enough to satisfy a predetermined condition in this direction. 5 (in FIG. 1, the two speakers 5f and 5g) are identified based on the acquired positional coordinate information (positional information) of the listener (sound receiver) and the speaker (sound source),
(a) Generate an output acoustic signal to be supplied to the specified speakers 5 (5f and 5g).
The speaker identification process (a) and the speaker panning process (b) will be specifically explained below using FIG. 4.

FIG. 4 is a schematic diagram of a playback space for explaining an embodiment of speaker identification processing and speaker panning processing according to the present invention. Here, an xy position coordinate system that defines a position in a horizontal plane is set in the playback room, which is a playback space.

As shown in FIG. 4, in this embodiment, the speaker specific panning section 113d (FIG. 1) first
(A1) Calculate the intersection point SP (x_sp, y_sp) between the line segment and the boundary after further extending the line segment connecting the listener's position R to the speaker's position S,
(A2) A speaker 5g whose direction from its installation position to the listener's position R is closest to the direction from the position SP (x_sp, y_sp) to the listener's position R (the angle between them is the smallest) and the speaker 5f that is the second closest (the angle between them is the second smallest) is specified (selected).

Next, in this embodiment, the speaker specific panning unit 113d (FIG. 1)
(b) For speakers 5f and 5g, where the specified (selected) speakers 5f and 5g will output a sound equivalent to or corresponding to "the sound picked up (by the microphone)" toward the listener. generate an output acoustic signal using an output acoustic signal t_out(p) (generated using the first and second input acoustic signals) based on the obtained positional coordinate information of the listener and the speaker. .

Specifically, as shown in Figure 4,
(a) Let ρ be the angle formed by the line segment connecting the position P5f of the speaker 5f and the listener's position R, and the line segment connecting the position P5g of the speaker 5g and the listener's position R,
(b) When σ is the angle formed by the line segment connecting the position P5f of the speaker 5f and the listener's position R, and the line segment connecting the position SP and the listener's position R,
The output acoustic signal t5f_out(p) for the speaker 5f (to be supplied to the speaker 5f) and the output acoustic signal t5g_out(p) for the speaker 5g (to be supplied to the speaker 5g) are calculated by calculating these ρ and σ. Then, the following formula (10) t5f_out(p)=t_out(p)×(sin(ρ/2)+sin(ρ/2−σ))
t5g_out(p)=t_out(p)×(sin(ρ/2)−sin(ρ/2−σ))
It is derived by

Here, the above equation (10) uses the speakers 5f and 5g to implement so-called (amplitude) panning, which realizes sound image localization by the difference in volume (difference in amplitude intensity) between both speakers (channels). It becomes.

As described above, using Figures 1 to 4, the sound field in the sound collection space (sound collection room) including the sound source (speaker) is reproduced to the sound receiver (listener) existing in the playback space (playback room). Although the processing has been explained, the processing in this embodiment will be briefly summarized here. In this embodiment,
(a) From the multi-channel omnidirectional microphone 2, select the 2-channel omnidirectional microphone 2 based on the position coordinate information of the sound source (speaker) and the sound receiver (listener), and select the 1-channel (monaural) omnidirectional microphone 2. ) determining a first input acoustic signal;
(b) From the multi-channel sharp directional microphones 3, the 2-channel sharp directional microphone 3 is selected based on the position coordinate information and face azimuth information of the sound source (speaker), and the 1-channel (monaural) 2, determine the input acoustic signal of
(c) After mixing these monaural (two channels in total) input audio signals to generate a one-channel (monaural) output audio signal, the multi-channel speaker 5 outputs a sound source (speaker) and Based on the positional coordinate information of the sound receiver (listener), the two-channel speaker 5 is selected, and an output acoustic signal is generated and provided that enables the selected speaker 5 to localize the sound image.

Returning to the functional block diagram of FIG. 1, the near speaker identification panning section 113e of the output acoustic signal generation section 113 is configured to At least one "another speaker" located nearby or further inside (as viewed from the room wall) is also identified, and an output acoustic signal to be supplied to the identified "another speaker" (speaker 5_cell in FIG. 1) is also generated. . This reproduces a sound field (acoustic) that cannot be achieved by the surround speakers 5a to 5h alone, for example, a sound field (acoustic) that makes you feel that the sound source is nearby, that is, a sound field (acoustic) that localizes the sound image closer. Or it can be reproduced.

In the embodiment shown in FIG. 1, the speaker 5_cell installed at the center of the ceiling of the playback room is a "different speaker" from the surround speakers 5a to 5h arranged in the playback room, and This is a speaker that can present sounds from inside the room. In this case, the near speaker specific panning unit 113e calculates the output acoustic signal tcell_out(p) for the speaker 5_cell using the following formula (11) tcell_out(p)=tf_gun(p)
(12) tcell_out(p)=tc_gun(p)
(13) tcell_out(p)=t_out(p)
The decision can be made based on either of the following. An embodiment in which a plurality of such "other speakers" exist will be described below using FIG. 5.

FIG. 5 is a schematic diagram of a reproduction space for explaining an embodiment of ceiling speaker identification processing and ceiling speaker panning processing according to the present invention.

According to FIG. 5, in the playback space, the surround speakers 5a to 5h (omitted in FIG. 5) are "separate speakers" at multiple positions on the ceiling boundary (surrounding the inside of the space) and at the center of the ceiling. Ceiling speakers (9 in total in Figure 5) are installed. Here, the near speaker identification panning unit 113e (FIG. 1) identifies (selects) a predetermined number (three in FIG. 5) of ceiling speakers based on the listener's position coordinate information (position information), and An output acoustic signal is generated to be provided to each ceiling speaker so that panning can be performed by the ceiling speakers.

Specifically, in FIG. 5, three ceiling speakers are located at the vertices of the smallest "triangle" with the ceiling speaker as the apex, and which includes the ceiling position above the listener's head (marked with a cross in FIG. 5). 5_cell1, 5_cell2, and 5_cell3 are specified (selected). The near speaker specific panning unit 113e (FIG. 1) converts the output acoustic signals for the specified ceiling speakers 5_cell1, 5_cell2, and 5_cell3 into, for example, VBAP so that the sound can be heard from the ceiling position (cross mark) above the listener's head. (Vector Based Amplitude Panning) method.

Here, the VBAP method is a typical control method for sound image localization in three-dimensional multi-channel sound (sound field), for example, non-patent literature: Akio Ando, "High Presence Technology for Acoustics", Journal of the Institute of Image Information and Media Engineers, It is explained in detail in Vol.66, No.8, pp.671-677, 2012.

In this embodiment, the ceiling speaker is fixedly installed on the ceiling of the playback room (playback space), but the speaker may not be fixed and its position may change. That is, at least one of the plurality of ceiling speakers may be movable on the ceiling of the playback room, for example. Further, as the "another speaker", a speaker attached to a robot that can move within the playback room can also be employed. In this case, for example, a robot equipped with a speaker may move as appropriate to the vicinity of the listener in order to realize sound image localization in the vicinity of the listener. Furthermore, it is also possible to use a dodecahedral speaker 5p installed in the center of the floor of the reproduction room or movable on the floor as "another speaker".

The sound field reproduction processing performed by the output acoustic signal generation unit 113 (FIG. 1) using the speakers installed in the reproduction room (reproduction space) has been described above. Hereinafter, sound field reproduction processing in a stereo on, which is headphones in this embodiment, worn by a listener (sound receiver) will be described. In this case, the reproduction space is not a physical space but a virtual space (sound image space) related to sound image localization in headphones worn by the listener.

Returning to the functional block diagram of FIG. 1, the output acoustic signal generation unit 113 uses the output acoustic signal t_out(p) generated by the output mixing unit 113c to ) A sound equivalent to or corresponding to the "collected sound" is output toward the listener from a direction toward the corresponding position of the speaker (sound source) or from a direction close to this direction until a predetermined condition is met. generate an output audio signal th_out(p) to be supplied to this headphone, which will be

Specifically, the Head-Related Transfer Function (HRTF) of the headphones, which reflects the speaker's position as seen from the listener and the speaker's orientation for the listener's both ears, is used as the output acoustic signal. The output acoustic signal th_out(p) can be generated by convolving t_out(p) in the frequency domain (performing convolution integral processing). Regarding HRTF, for example, non-patent literature: Takanori Nishino, "Head-related transfer function database", [online], [searched on June 27, 2020], Internet <URL: https://sites.google. com/site/takanorinishinomu/research/hrtf/database-j> is explained in detail.

Also, as mentioned above, when generating the output audio signal th_out(p) for headphones, instead of the output audio signal t_out(p),
・This output acoustic signal t_out(p) is generated by convolving (performing convolution integral processing) an inverse filter generated from the room transfer function (RTF, Room Transfer Function) of the sound collection room (sound collection space). Output acoustic signal tan_out(p)
may also be used. As a result, the listener can also enjoy clear sound with the reverberations of the sound collection room removed, for example, using the headphones. Of course, this output acoustic signal tan_out(p) may be used in other acoustic output systems such as speakers in a sound collection room, in addition to headphones.

In yet another embodiment, the output acoustic signal generation unit 113 provides the output acoustic signal t_out(p) output from the output mixing unit 113d to the outside of the device 1 as it is (for example, without performing panning processing). is also possible. Even in this case, this output acoustic signal t_out(p) is used as an acoustic signal that causes a sound equivalent to or corresponding to the "collected sound" to be output toward the sound receiver outside the device 1. It is now possible to do so.

Furthermore, as a further embodiment, the input acoustic signal determination unit 112 may select at least one pre-specified The input acoustic signal may be determined by acquiring acoustic signals related to sounds picked up by two microphones. In this case as well, the output acoustic signal generation unit 113 continues in the direction toward the corresponding position of the sound source (speaker) as seen from the sound receiver (listener) in the reproduction space (reproduction room), or until the predetermined condition is met in this direction. At least one speaker 5 located in a direction close to is specified based on the acquired information on the positions of the sound receiver (listener) and the sound source (speaker), and an output acoustic signal to be supplied to the specified speaker 5. It generates.

Similarly, in the functional block diagram of FIG. 1, as explained above, the output acoustic signal generated by the output acoustic signal generation section 113 is transmitted from the communication interface 101 via the communication control section 121 to the reproduction space via the communication network. The signal is transmitted to each installed (identified) speaker or to an audio signal distribution control device (not shown) installed in or near the playback room (playback space). Here, it is also preferable that the transmitted output acoustic signal is given identification information of the (specified) speaker to which the signal is to be supplied.

Further, the sound field reproduction processing as described above may be performed by the input/output control unit 122 outputting a processing execution instruction inputted from the keyboard (KB) 102 to the corresponding functional component, for example. . Further, for example, the input/output control unit 122 may appropriately acquire information on the identified microphone and speaker and information on the positions of the speaker and the listener from the corresponding functional components, for example, as shown in the upper part of FIG. The display (DP) 102 may display images showing the current situation in the sound collection room and the playback room.

As described in detail above, according to the present invention, the microphone is identified and the input acoustic signal is determined based on the acquired information regarding the positions of the sound source and the sound receiver, and the output acoustic signal is further generated. Even when the sound receiver is located at an arbitrary position within the reproduction space or is movable within the reproduction space, the sound field of the sound collection space can be reproduced with respect to the sound receiver. Furthermore, even if the sound source is located at any position within the sound collection space or is movable within the sound collection space, it is possible to reproduce the sound field of the sound collection space to the sound receiver. Become.

Furthermore, the present invention provides a sound field that corresponds to or corresponds to the sound field in the sound collection space, for example, in web conferences, remote sessions, remote conferences, and even online concerts, which have been widely used in recent years. This can greatly contribute to faithfully reproducing the space for the recipient.

Further, according to the present invention, for example, (a) music teaching materials relating to singers/players who sing and perform while moving on a stage; (b) physical education teaching materials relating to voices and movement sounds relating to moving athletes; (c) contains language teaching materials related to conversations while walking side by side with native speakers or while performing various collaborative tasks, and (d) records of various sounds occurring at various locations in facilities such as factories and exhibition halls. We create social field study materials with high sound field reproduction, and use these high-quality materials to provide excellent music education, physical education, language education, social education, etc. to children not only in urban areas but also in rural areas. For example, it is possible to provide students with the opportunity to receive training online, or in some cases in real time. In other words, the present invention contributes to Goal 4 of the Sustainable Development Goals (SDGs) led by the United Nations: “Provide inclusive, equitable and quality education for all and promote lifelong learning opportunities.” It is also possible to do so.

Regarding the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical idea and viewpoint of the present invention can be easily made by those skilled in the art. The above description is merely an example and is not intended to be limiting in any way. The invention is limited only by the claims and their equivalents.

1 Sound field reproduction device 101 Communication interface 102 Keyboard (KB)/Display (DP)
111 Position information acquisition section 112 Input acoustic signal determination section 112a Omnidirectional microphone identification section 112b, 112d Input mixing section 112c Sharp directional microphone identification section 113 Output acoustic signal generation section 113
113a Amplitude adjustment section 113b Tone adjustment section 113c Output mixing section 113d Speaker (SP) specific panning section 113e Near speaker (SP) specific panning section 121 Communication control section 122 Input/output control section 2, 2a, 2b, 2c, 2d, 2e, 2f, 2g, 2h Omnidirectional microphone 3, 3a, 3b, 3c, 3d, 3e, 3f, 3g, 3h Sharp directional microphone 4 Depth sensor 5, 5a, 5b, 5c, 5d, 5e, 5f, 5g, 5h Speaker 5_cell1, 5_cell2, 5_cell3 Ceiling speaker 5p Dodecahedral speaker

Claims (15)

A sound field reproduction program for reproducing a sound field related to a sound source in a sound collection space to a sound receiver in a reproduction space that has a correspondence relationship with the sound collection space in terms of position in the space, the program comprising:
a position information acquisition means for acquiring information regarding the measured or set position of the sound source and the sound receiver;
At least one microphone located in a direction toward the corresponding position of the sound receiver as seen from the sound source in the sound collection space, or in a direction close to the sound direction until a predetermined condition is satisfied, based on the acquired information regarding the position. input acoustic signal determining means for determining an input acoustic signal by acquiring an acoustic signal related to the sound picked up by the identified microphone;
In the reproduction space, from the direction toward the corresponding position of the sound source as seen from the sound receiver, or from a direction close to the direction until a predetermined condition is satisfied, toward the sound receiver, the sound corresponding to or corresponding to the collected sound. The computer is characterized by causing the computer to function as an output acoustic signal generation means for generating an output acoustic signal in which a corresponding sound is output, using the input acoustic signal based on the acquired information regarding the position. Sound field reproduction program. The position information acquisition means also acquires information regarding the output direction of the measured or set sound at the sound source,
The input acoustic signal determining means selects at least one other microphone located in the output direction or in a direction close to the output direction as seen from the sound source in the sound collection space, based on the acquired position. and determining a different input acoustic signal by obtaining another acoustic signal related to the sound picked up by the identified other microphone,
2. The output acoustic signal generating means generates the output acoustic signal by combining the other input acoustic signal and the input acoustic signal related to the sound picked up by the microphone. The sound field reproduction program described. 3. The sound field reproduction program according to claim 2, wherein the other microphone has higher directivity than the other microphone. The playback space is a real space,
The output acoustic signal generating means acquires at least one speaker located in a direction toward a corresponding position of the sound source as seen from the sound receiver in the reproduction space or in a direction close to the direction until a predetermined condition is satisfied. 4. The sound field reproduction program according to claim 1, wherein the sound field reproduction program generates the output acoustic signal to be supplied to the specified speaker based on information regarding the specified position. The output acoustic signal generating means also identifies at least one other speaker that is closer to the position of the sound receiver than the identified speaker or is located more inward from the surrounding boundary, and 5. The sound field reproduction program according to claim 4, further comprising generating the output acoustic signal to be supplied to a speaker. The sound source and the sound receiver are movable within the sound collection space and the reproduction space, which are real spaces, respectively;
The input acoustic signal determining means specifies the microphone at the one time point based on the information regarding the position measured and acquired at the one time point by the position information acquisition means, and determines the input sound signal at the one time point. determine the acoustic signal;
The output acoustic signal generation means specifies the speaker at the one point in time based on information about the position acquired at the one point in time and generates the output acoustic signal at the one point in time. The sound field reproduction program according to claim 4. The playback space is a virtual space related to sound image localization in the stereophone attached to the sound receiver,
The output acoustic signal generating means is configured to generate a signal toward the sound receiver from a direction toward a corresponding position of the sound source as seen from the sound receiver in the reproduction space, or from a direction close to the direction until a predetermined condition is satisfied. According to any one of claims 1 to 3, the output audio signal is generated to be supplied to the stereophone, from which a sound corresponding to or corresponding to the collected sound is output. Sound field reproduction program. The output acoustic signal generating means performs an amplitude adjustment process on the determined another input acoustic signal to attenuate the amplitude according to the distance between the position of the sound source and the position of the sound receiver, and performs the process. 4. The sound field reproduction program according to claim 2, wherein the output acoustic signal is generated using the other input acoustic signal subjected to the above processing. The output acoustic signal generation means performs a filtering process on the determined another input acoustic signal to emphasize a predetermined high frequency band depending on the proximity between the position of the sound source and the position of the sound receiver, Alternatively, determine the impulse response between the position of the sound source and the position of the sound receiver, perform convolution processing in the frequency domain related to the impulse response, and use the other input acoustic signal that has undergone the processing. 4. The sound field reproduction program according to claim 2, wherein the sound field reproduction program generates an output acoustic signal. The output acoustic signal generating means determines a phase difference between the other input acoustic signal and the input acoustic signal based on the information regarding the position, and delays the phase of one of the input acoustic signals so as to eliminate the determined phase difference. 4. The sound field reproducing program according to claim 2, further comprising the step of combining said other input sound signal and said input sound signal to generate said output sound signal. A sound field reproduction program for reproducing a sound field related to a sound source in a sound collection space to a sound receiver in a reproduction space that has a correspondence relationship with the sound collection space in terms of position in the space, the program comprising:
a position information acquisition means for acquiring information regarding the measured or set position of the sound source and the sound receiver, and information regarding the output direction of the measured or set sound at the sound source;
(a) At least one microphone located in the direction toward the corresponding position of the sound receiver as seen from the sound source in the sound collection space, or in a direction close to the sound direction until a predetermined condition is satisfied, is moved to the acquired position. (b) determining the input sound signal by determining the sound signal related to the sound picked up by the identified microphone based on the information; and (b) determining the output direction as seen from the sound source in the sound collection space. or at least one other microphone located in a direction close to the output direction that satisfies a predetermined condition, based on the obtained information regarding the position and the information regarding the output direction, and the identified another microphone input sound signal determining means for determining another input sound signal by acquiring another sound signal related to the sound collected by the input sound signal;
An output that combines the input acoustic signal and the other input acoustic signal to generate an output acoustic signal that results in outputting a sound equivalent to or corresponding to the collected sound toward the sound receiver. A sound field reproduction program characterized by causing a computer to function as an acoustic signal generating means. A sound field reproduction program for reproducing a sound field related to a sound source in a sound collection space to a sound receiver in a reproduction space that has a correspondence relationship with the sound collection space in terms of position in the space, the program comprising:
a position information acquisition means for acquiring information regarding the measured or set position of the sound source and the sound receiver;
input acoustic signal determining means for determining an input acoustic signal by acquiring an acoustic signal related to a sound picked up by at least one microphone capable of picking up the sound from the sound source;
At least one speaker located in the direction toward the corresponding position of the sound source as viewed from the sound receiver in the playback space, or in a direction close to the sound source until a predetermined condition is satisfied, based on the acquired information regarding the position. A sound field reproduction program characterized by causing a computer to function as an output acoustic signal generation means for specifying and generating an output acoustic signal to be supplied to the specified speaker. A sound field reproduction device for reproducing a sound field related to a sound source in a sound collection space to a sound receiver in a reproduction space that has a correspondence relationship with the sound collection space in terms of position in the space,
a position information acquisition means for acquiring information regarding the measured or set position of the sound source and the sound receiver;
At least one microphone located in a direction toward the corresponding position of the sound receiver as seen from the sound source in the sound collection space, or in a direction close to the sound direction until a predetermined condition is satisfied, based on the acquired information regarding the position. input acoustic signal determining means for determining an input acoustic signal by acquiring an acoustic signal related to the sound picked up by the identified microphone;
In the reproduction space, from the direction toward the corresponding position of the sound source as seen from the sound receiver, or from a direction close to the direction until a predetermined condition is satisfied, toward the sound receiver, the sound corresponding to or corresponding to the collected sound. A sound field characterized by comprising an output acoustic signal generating means for generating an output acoustic signal in which a corresponding sound is to be output, using the input acoustic signal based on the acquired information regarding the position. playback device. A sound field reproduction system for reproducing a sound field related to a sound source in a sound collection space to a sound receiver in a reproduction space that has a corresponding relationship with the sound collection space in terms of position in the space, the system comprising:
a position information acquisition means for acquiring information regarding the measured or set position of the sound source and the sound receiver;
At least one microphone located in a direction toward the corresponding position of the sound receiver as seen from the sound source in the sound collection space, or in a direction close to the sound direction until a predetermined condition is satisfied, based on the acquired information regarding the position. input acoustic signal determining means for determining an input acoustic signal by acquiring an acoustic signal related to the sound picked up by the identified microphone;
In the reproduction space, from the direction toward the corresponding position of the sound source as seen from the sound receiver, or from a direction close to the direction until a predetermined condition is satisfied, toward the sound receiver, the sound corresponding to or corresponding to the collected sound. A sound field characterized by comprising an output acoustic signal generating means for generating an output acoustic signal in which a corresponding sound is to be output, using the input acoustic signal based on the acquired information regarding the position. playback system. A sound field reproduction method implemented by a computer for reproducing a sound field related to a sound source in a sound collection space to a sound receiver in a reproduction space that corresponds to the sound collection space in terms of position in the space, ,
obtaining information regarding the measured or set position of the sound source and the sound receiver;
At least one microphone located in a direction toward the corresponding position of the sound receiver as seen from the sound source in the sound collection space, or in a direction close to the sound direction until a predetermined condition is satisfied, based on the acquired information regarding the position. determining an input acoustic signal by acquiring an acoustic signal related to the sound picked up by the identified microphone;
In the reproduction space, from the direction toward the corresponding position of the sound source as seen from the sound receiver, or from a direction close to the direction until a predetermined condition is satisfied, toward the sound receiver, the sound corresponding to or corresponding to the collected sound. A sound field reproduction method comprising the step of generating an output acoustic signal in which a corresponding sound is to be output, using the input acoustic signal based on the acquired information regarding the position.

Images