JP7493411B2 - Binaural playback device and program - Google Patents

Description

The present invention relates to a binaural playback device and program.

In recent years, progress has been made in putting object-based audio systems and AR/VR audio, which combine audio signals and audio metadata, to practical use. Object-based audio and AR/VR audio are characterized by providing a large number of audio signals and associated audio metadata, and rendering and playing audio signals according to the position and posture of the listener in both real and virtual playback spaces. When earphones or headphones are used as playback devices, binauralization using head-related transfer functions is often included in the rendering process described above. Binauralization is a technique that simulates the propagation of sound waves in space at the entrance to the ear canal, and is said to enable three-dimensional directional perception of sound.

Non-patent documents 1 and 2 describe audio signals and audio metadata. Non-patent documents 3, 4, and 5 describe object-based audio systems. Non-patent document 6 describes AR/VR audio. Non-patent document 7 describes binauralization using head-related transfer functions.

Recommendation: ITU-R BS.2076-1, "Audio Definition Model", June 2017, International Telecommunication Union. Recommendation: ITU-R BS.2125-0, "A serial representation of the Audio Definition Model", January 2019, International Telecommunication Union. ISO/IEC 23008-3:2019, "Information technology - High efficiency coding and media delivery in heterogeneous environments - Part 3: 3D audio, Second edition", February 2019, International Organization for Standardization. ETSI TS 103 190-2 V1.2.1, Technical Specification, "Digital Audio Compression (AC-4) Standard; Part 2: Immersive and personalized audio", February 2018, European Telecommunications Standards Institute. ATSC Standard: A/342 Part 3, "MPEG-H SYSTEM", March 2017, Advanced Television Systems Committee ISO/IEC 23090-4, "MPEG-I Immersive Audio Coding", The Moving Picture Experts Group, [Retrieved June 6, 2020], Internet <URL: https://mpeg.chiariglione.org/standards/mpeg-i> Kazuhiro Iida, Masayuki Morimoto, "Spatial Acoustics", edited by the Acoustical Society of Japan, Corona Publishing, 2010

For binauralization using head-related transfer functions, the transfer functions to both ears are used when the sound source is considered to be a point source. Therefore, when measuring head-related transfer functions, it is common to use full-range speakers that can be assumed to be a point source, and it is not possible to reflect the radiation characteristics that depend on the radiation angle that exists in an actual sound source.

On the other hand, in AR/VR content, in order to further improve the sense of realism, it is necessary to describe phenomena with greater precision than conventional three-dimensional audio, and it is reasonable to reflect the angle-dependent radiation characteristics of the sound source. However, as mentioned above, conventional head-related transfer functions cannot reflect the angle dependency of the radiation characteristics. However, developing and implementing a new head-related transfer function that takes into account the angle dependency of the radiation characteristics would require high development costs and computational load, making it difficult to say it is practical. Therefore, in object-based audio, AR/VR audio, etc., a binaural playback device that can reproduce the angle dependency of the radiation characteristics of the sound source using conventional head-related transfer functions is required.

The present invention was made in consideration of the above circumstances, and aims to provide a binaural playback device and program that can reflect the angular dependency of the radiation characteristics of a sound source using a conventional head-related transfer function.

[1] In order to solve the above problem, a binaural playback device according to one aspect of the present invention includes a head-related transfer function database that holds transfer functions according to the direction from the center of the listener's head, a propagation path derivation unit that derives a sound propagation path from the sound source to the ear based on the position of the sound source, the shape of the listener's head, and the position of the listener's ear, a sound source radiation direction determination unit that determines the radiation direction from the sound source to the ear based on the derived propagation path, a sound source database that outputs an audio signal having acoustic characteristics corresponding to the determined radiation direction, a head-related transfer function selection unit that selects the transfer function from the head-related transfer function database based on the direction from the listener's head center to the sound source specified by the position of the listener's head center and the position of the sound source, and a playback signal generation unit that generates a playback signal for the ear based on the output audio signal, the radiation characteristics selected by the sound source radiation characteristic selection unit, and the transfer function selected by the head-related transfer function selection unit.

[2] In one aspect of the present invention, in the binaural playback device described above, the ears are the left ear and the right ear, the propagation path derivation unit derives the propagation path of the sound from the sound source to the left ear and the right ear, the head-related transfer function database holds the transfer functions of the left ear and the right ear as a left-ear head-related transfer function and a right-ear head-related transfer function, the sound source radiation direction determination unit determines the radiation direction from the sound source to the left ear and the right ear, and the sound source database holds the transfer functions of the left ear and the right ear as a left-ear head-related transfer function and a right-ear head-related transfer function. An audio signal having acoustic characteristics corresponding to the radiation direction toward the ear is selected as an audio signal for the left ear, and an audio signal having acoustic characteristics corresponding to the radiation direction toward the right ear is selected as an audio signal for the right ear, the head-related transfer function selection unit selects a left-ear head-related transfer function and a right-ear head-related transfer function corresponding to the direction toward the sound source, and the playback signal generation unit generates the playback signal for the left ear based on the left-ear audio signal and the left-ear head-related transfer function, and generates the playback signal for the right ear based on the right-ear audio signal and the right-ear head-related transfer function.

[3] In one aspect of the present invention, in the binaural playback device described above, the sound sources are multiple, the sound source database outputs the audio signal for each of the sound sources, the propagation path derivation unit derives the propagation path for each of the sound sources, the sound source radiation direction determination unit determines the radiation direction for each of the sound sources, the head-related transfer function selection unit selects the transfer function from the head-related transfer function database for each of the sound sources, and the playback signal generation unit generates the playback signal for each of the sound sources.

[4] In one aspect of the present invention, in the binaural playback device described above, the playback signal generation unit generates a superimposed playback signal by superimposing the playback signals generated for each of the sound sources.

[5] In one aspect of the present invention, in the binaural playback device described above, the listeners are multiple, the propagation path derivation unit derives the propagation path for each of the listeners, the sound source radiation direction determination unit determines the radiation direction for each of the listeners, the head-related transfer function selection unit selects the transfer function for each of the listeners, and the playback signal generation unit generates the playback signal for each of the listeners.

[6] In one aspect of the present invention, in the binaural playback device described above, the sound source database holds audio signals having the acoustic characteristics corresponding to at least one of the following: the distance from the sound source to the listener, the type of mora contained in the human voice (voice) emitted from the sound source, the type of phoneme contained in the voice emitted from the sound source, the gender of the voice emitted from the sound source, the age of the voice emitted from the sound source, and the type of musical instrument of the sound source; and selects an audio signal having the acoustic characteristics corresponding to at least one of the following: the distance from the sound source to the listener, the type of mora contained in the voice emitted from the sound source, the type of phoneme contained in the voice emitted from the sound source, the gender of the voice emitted from the sound source, the age of the voice emitted from the sound source, and the type of musical instrument of the sound source.

[7] In one aspect of the present invention, in the binaural playback device described above, the propagation path derivation unit derives the propagation path in which direct propagation from the sound source to the ear is the shortest path when the ear is visible from the sound source, and in which diffracted propagation from the sound source to the ear is the shortest path when the ear is not visible from the sound source, and the sound source radiation direction determination unit determines the radiation direction depending on whether the propagation path is direct propagation or diffracted propagation.

[8] In one aspect of the present invention, in the binaural playback device described above, the playback signal generation unit includes a binaural signal generation unit that generates a playback signal for the ear by synthesizing a shortest path component related to the shortest path direction and a non-shortest path component related to a radiation direction other than the shortest path direction among the components of the sound arriving at the ear from the sound source, and the binaural signal generation unit generates the shortest path component by applying a transfer function according to the direction from the head center of the listener to an audio signal having acoustic characteristics in the shortest path direction, and generates the non-shortest path component by applying a transfer function according to the direction from the head center of the listener to a weighted sum based on a weighting coefficient related to the propagation path of a single or multiple audio signals having acoustic characteristics corresponding to each direction of the propagation path other than the shortest path, and the weighting coefficient is determined according to the radiation direction of the sound wave from the sound source, the direction of the shortest path, and the direction from the sound source to the head center.

[9] A binaural playback device according to one aspect of the present invention includes a head impulse response database that holds head impulse responses according to a direction from the center of the listener's head; a propagation path derivation unit that derives a propagation path of sound from the sound source to the ear based on the position of the sound source, the shape of the listener's head, and the position of the listener's ear; a sound source radiation direction determination unit that determines a radiation direction from the sound source to the ear based on the derived propagation path; a sound source database that holds an audio signal having acoustic characteristics corresponding to the radiation direction from the sound source and outputs an audio signal having acoustic characteristics corresponding to the determined radiation direction; a head impulse response selection unit that selects the head impulse response from the head impulse response database based on the direction from the listener's head center to the sound source specified by the position of the listener's head center and the position of the sound source; and a playback signal generation unit that generates a playback signal for the ear based on the output audio signal and the head impulse response selected by the head impulse response selection unit.

[10] Another aspect of the present invention is a program for causing a computer to function as the binaural playback device described in any one of [1] to [9] above.

According to the present invention, it is possible to reflect the angular dependency of the radiation characteristics of a sound source using an existing head-related transfer function. This makes it possible to increase the realism and accuracy of the audio signal with a light computational load.

1 is a block diagram showing a schematic functional configuration of a binaural reproduction device according to an embodiment of the present invention. 2 is a schematic diagram showing an oblique view of a virtual acoustic space on which the embodiment is based. FIG. 4 is a block diagram showing an internal functional configuration of a left ear propagation path determination unit according to the embodiment. FIG. 4 is a block diagram showing a detailed internal functional configuration of a left-ear reproduction signal generating unit according to the embodiment. FIG. 6 is a flowchart showing a processing procedure for generating a playback signal by the binaural playback device according to the embodiment. FIG. 2 is a schematic diagram illustrating a virtual acoustic space envisioned by the embodiment. 1 is a schematic diagram illustrating a plane including the coordinate positions of both ears of a listener and the coordinate position of a sound source in a virtual acoustic space assumed in the embodiment. FIG.

Next, one embodiment of the present invention will be described with reference to the drawings. This embodiment realizes binaural playback that reflects the radiation characteristics of the sound source in object-based audio content and AR/VR content. Note that AR is an abbreviation for "Augmented Reality." Also, VR is an abbreviation for "Virtual Reality." AR/VR technology itself is an existing technology.

In the following, the configuration of the binaural playback device will be described assuming that there is mainly one listener and one sound source (a point sound source with radiation direction (angle) dependency). In the case of two or more listeners, the binaural playback device can perform the same processing for each listener as in the case of one listener. In the case of two or more sound sources, the binaural playback device can also perform processing for each sound source as in the case of one sound source, and superimpose multiple output signals corresponding to those sound sources.

The binaural playback device determines output signals for the right and left ears of the listener. As will be described in more detail below, the binaural playback device models the position of the sound source, the shape and size of the listener's head, and the positions of the two ears on the listener's head, and determines output signals for the right and left ears according to the model.

1 is a block diagram showing a schematic functional configuration of a binaural playback device according to this embodiment. As shown in the figure, the binaural playback device 1 includes a listener information acquisition unit 11, a sound source information acquisition unit 12, a listener head shape acquisition unit 15, a left ear coordinate acquisition unit 17, a right ear coordinate acquisition unit 18, a left ear propagation path determination unit 19, a right ear propagation path determination unit 20, a left ear sound source radiation direction determination unit 21, a right ear sound source radiation direction determination unit 22, a sound source database 24, a head related transfer function database 31, a head related transfer function selection unit 32, a left ear playback signal generation unit 35, and a right ear playback signal generation unit 36. Each of these functional units can be realized, for example, by a computer and a program. Each functional unit also has a storage unit as needed. The storage unit is, for example, a variable in the program or a memory allocated by the execution of the program. Also, non-volatile storage units such as a magnetic hard disk drive or a solid state drive (SSD) may be used as needed. In addition, at least some of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program. The functions of each unit are as follows.

The listener information acquisition unit 11 acquires information on the position and orientation (posture) of the listener within the virtual acoustic space. The listener can be at any position and in any orientation (posture) within the virtual acoustic space. In other words, the listener's information has six degrees of freedom (6DoF). The position can be expressed by three coordinate values of x, y, and z in a Cartesian coordinate system. Alternatively, the position can be expressed by the coordinate values of r (distance), φ (azimuth angle), and θ (elevation angle) in polar coordinates. The orientation can be expressed by rotation in three directions: yaw, roll, and pitch.

The interval (timing) at which the listener information acquisition unit 11 acquires information may be appropriately determined according to the time resolution of the device. However, it is preferable that the period at which the listener information acquisition unit 11 acquires the listener information is generally equal to or shorter than the frame processing time of the audio signal. For example, when the sampling frequency of the audio signal is 48 kHz (kilohertz), it is preferable to acquire the listener information at a period equivalent to 2048 samples or less. Note that the listener information acquisition unit 11 can acquire the listener information using existing technology. As an example, the listener information acquisition unit 11 captures images of the listener in real space from multiple directions with a camera, and measures the position of a specific part of the listener using a triangulation method. The listener may be marked so that it can be recognized as an image in real space. Note that the listener information acquisition unit 11 may acquire the listener information using other methods.

The sound source information acquisition unit 12 acquires information (rotation information) on the position and orientation of the sound source in the virtual acoustic space. The information on the position and orientation of the sound source may be information with six degrees of freedom (6DoF) like the information on the position and orientation of the listener described above. The information on the position and orientation of the sound source may be provided as metadata of the audio signal, for example. The interval (timing) at which the sound source information acquisition unit 12 acquires the information on the position and orientation of the sound source may be appropriately determined according to the time resolution of the device. However, it is preferable that the sound source information acquisition unit 12 acquires the information at an interval equal to or shorter than the frame processing time of the audio signal. For example, when the sampling frequency of the audio signal is 48 kHz (kilohertz), it is preferable that the sound source information acquisition unit 12 acquires the sound source information at a period equivalent to within 2048 samples.

The listener's head shape acquisition unit 15 acquires information on the shape of the listener's head. The head shape information includes information on the positions of the left and right ears. The listener's head shape acquisition unit 15 may acquire information on the specifically measured head shape and ear positions of a representative listener or an actual individual listener. Alternatively, the listener's head shape acquisition unit 15 may acquire information on the head shape and ear positions based on a predetermined head model. The head model may be, for example, one in which the head shape is a sphere with a radius of a, and the left and right ears are positioned at an azimuth angle of ±90 degrees and an elevation angle of 0 degrees when the azimuth angle of the front direction of the head (the direction the face is facing) is 0 degrees.

The left ear coordinate acquisition unit 17 acquires the left ear coordinates of the listener in the virtual acoustic space based on the information on the position and orientation of the listener's head acquired by the listener information acquisition unit 11 and the listener's head shape (including information on the position of the left ear on the head) acquired by the listener head shape acquisition unit 15. Note that the left ear may be considered to be a point in the virtual acoustic space.

The right ear coordinate acquisition unit 18 acquires the right ear coordinates of the listener in the virtual acoustic space based on the information on the position and orientation of the listener's head acquired by the listener information acquisition unit 11 and the listener's head shape (including information on the position of the right ear on the head) acquired by the listener head shape acquisition unit 15. Note that the right ear may be considered as a point in the virtual acoustic space.

The left ear propagation path determination unit 19 is also called a "propagation path derivation unit." The left ear propagation path determination unit 19 derives the propagation path of the sound from the sound source to the left ear based on the position of the sound source, the shape of the listener's head, and the position of the listener's ear. The left ear propagation path determination unit 19 also determines whether the propagation path from the sound source to the left ear in the virtual acoustic space is direct propagation or diffraction propagation, and outputs the determination result. If the left ear can be seen directly from the sound source, it is direct propagation. If the left ear cannot be seen directly from the sound source because it is in the shadow of the head or the like, it is diffraction propagation. Specifically, the left ear propagation path determination unit 19 determines the above-mentioned propagation path based on the position and orientation of the listener acquired by the listener information acquisition unit 11, the head shape acquired by the listener head shape acquisition unit 15, the coordinates of the left ear acquired by the left ear coordinate acquisition unit 17, and the sound source information acquisition unit 12. A more detailed method of determining the propagation path will be described later.

The right ear propagation path determination unit 20 is also called the "propagation path derivation unit." The right ear propagation path determination unit 20 derives the sound propagation path from the sound source to the right ear based on the position of the sound source, the shape of the listener's head, and the position of the listener's ears. The right ear propagation path determination unit 20 also determines whether the propagation path from the sound source to the right ear in the virtual acoustic space is direct propagation or diffraction propagation, and outputs the determination result. The method of determination is the same as that by the left ear propagation path determination unit 19 described above.

The propagation path to the left ear and the propagation path to the right ear may be different. For example, if the shape of the head is spherical and a straight line connecting the left and right ears on the spherical surface passes through the center point of the sphere, strictly speaking, from a sound source at a finite distance, at least one of the propagation paths to the left ear or the right ear is diffracted.

The left ear sound source radiation direction determination unit 21 is also called the "sound source radiation direction determination unit." The left ear sound source radiation direction determination unit 21 determines the radiation direction of sound waves to the left ear as viewed from the reference direction of the sound source in the virtual acoustic space. Specifically, the left ear sound source radiation direction determination unit 21 determines the radiation direction from the sound source to the ear based on the derived propagation path. The reference direction of the sound source is, for example, an azimuth angle (φ) of 0 degrees and an elevation angle (θ) of 90 degrees in the coordinate system of the direction of the sound source. Specifically, the left ear sound source radiation direction determination unit 21 determines the radiation direction to the left ear as viewed from the reference direction of the sound source based on the propagation path from the sound source to the left ear determined by the left ear propagation path determination unit 19 and the direction of the sound source (rotation information) acquired by the sound source information acquisition unit 12. When the propagation path is direct propagation, the left ear sound source radiation direction determination unit 21 determines the direction of a straight line connecting the sound source and the left ear (the direction from the sound source to the left ear) as the radiation direction. When the propagation path is diffractive propagation, the left ear sound source radiation direction determination unit 21 determines the direction of the shortest path among the paths along which sound waves propagate to the left ear after reaching the head and diffracting on the surface of the head. A more detailed method for determining the radiation direction will be described later.

The right ear sound source radiation direction determining unit 22 is also called the "sound source radiation direction determining unit." The right ear sound source radiation direction determining unit 22 determines the radiation direction of sound waves to the right ear as viewed from the reference direction of the sound source in the virtual acoustic space. The processing method for determining the judgment is the same as that by the left ear sound source radiation direction determining unit 21 described above. The left ear sound source radiation direction determining unit 21 and the right ear sound source radiation direction determining unit 22 each output the radiation direction that they have determined to the sound source database 24.

As explained above, the propagation path derivation unit (left ear propagation path determination unit 19 and right ear propagation path determination unit 20) derives a propagation path by direct propagation from the sound source to the ear when the ear is visible from the sound source. When the ear is not visible from the sound source, the propagation path derivation unit derives a propagation path by diffraction propagation that diffracts the head from the sound source to the ear. The sound source radiation direction determination unit (left ear sound source radiation direction determination unit 21 and right ear sound source radiation direction determination unit 22) determines the radiation direction depending on whether the propagation path determined by the above propagation path derivation unit is a direct path or a diffraction path.

The sound source database 24 holds audio signals emitted from a sound source in a virtual acoustic space. Specifically, the sound source database 24 holds audio signals having acoustic characteristics corresponding to each radiation direction from the sound source. For example, the sound source database 24 holds audio signals corresponding to combinations of arguments r (distance), φ (azimuth angle), θ (elevation angle), and ω (frequency (or angular frequency)) using polar coordinate display. The intervals of each argument may be set to an appropriate size. The radiation characteristics according to the radiation direction are expressed by acoustic characteristic values. The acoustic characteristic value is, for example, a value of the ratio to the strength of the sound wave component at the reference position. Here, the reference position may be, for example, a position in the reference direction of the sound source (azimuth angle (φ) is 0 degrees, elevation angle (θ) is 90 degrees) with a distance of unit length (r = 1). The acoustic characteristic value may be a value obtained by multiplying the strength ratio by a complex number with an absolute value of 1 corresponding to the phase difference. Here, the phase difference corresponds to, for example, the difference between the phase of the sound wave component at the reference position and the phase of the sound wave component propagated in the radial direction. For example, a transfer function defined for each frequency as the sound wave component can be used as the acoustic characteristic value.

The sound signal having radiation characteristics according to the radiation direction is obtained, for example, by synthesizing the multiplied values obtained by multiplying each frequency component of the sound signal at a predetermined reference position by the acoustic characteristic value for that frequency in frequency space. The sound source database 24 is not necessarily limited to this example, as long as it can provide a sound signal having acoustic characteristics corresponding to the radiation direction as an argument. The sound source database 24 may be configured to radiate sound from each sound source and store the sound signals of the sounds collected in advance for each radiation direction in association with the radiation direction. The sound source database 24 may be configured to set the sound signals of the sounds radiated from the sound source, and may define the set sound signals as sound signals having radiation characteristics for each radiation direction using a predetermined calculation formula or model. As the predetermined calculation formula or model, for example, multiplication or convolution of a transfer function set for each radiation direction, a geometric acoustic model that gives radiation characteristics for each radiation direction, etc. may be used. The sound source database 24 may assume radiation of sound based on a predetermined sound signal from the sound source, perform a simulation using a predetermined model, and generate sound signals of sounds arriving for each radiation direction. In the simulation, techniques such as the ray acoustic method and wave field synthesis can be used.

When the radiation direction is input from the left-ear sound source radiation direction determination unit 21, the sound source database 24 outputs an audio signal corresponding to the input radiation direction as an audio signal according to the radiation direction of the sound source for the left ear, i.e., an audio signal for the left ear, to the left-ear reproduction signal generation unit 35. Similarly, when the radiation direction is input from the right-ear sound source radiation direction determination unit 22, the sound source database 24 outputs an audio signal corresponding to the input radiation direction as an audio signal according to the radiation direction of the sound source for the right ear, i.e., an audio signal for the right ear, to the right-ear reproduction signal generation unit 36.
The sound source database 24 may take arguments other than those listed above. The sound source database 24 may omit the distance argument. When the distance argument is included, the radiation direction substantially indicates the radiation position from which the sound is radiated. The sound source database 24 may be realized, for example, as a multidimensional information table expanded on a memory. Alternatively, the sound source database 24 may be realized using a database management system (DBMS). The sound source database 24 may use the above method to generate a sound signal to which acoustic characteristics corresponding to the radiation direction are applied each time the radiation direction is input, and output the generated sound signal. In this case, parameters used for calculation using a calculation formula or model or for generating a sound signal are set in advance in the sound source database 24.

The head-related transfer function database 31 holds information on the head-related transfer function of the listener. The head-related transfer function has information on the transfer function according to the direction from the center of the listener's head. As the transfer function for the left ear and the transfer function for the right ear for each direction from the center of the head, a head-related transfer function for the left ear and a head-related transfer function for the right ear are held, respectively. The step size (decomposition step) of the direction is appropriately determined in advance. The head-related transfer function may be obtained by measuring for a specific listener, or may be obtained by measuring using a dummy head or the like. The head-related transfer function itself does not need to be specially acquired for this embodiment. If there is a head-related transfer function used in the conventional technology, the information on the head-related transfer function may be held as it is in the head-related transfer function database 31. When the head-related transfer function database 31 is queried by specifying a direction, it responds with the head-related transfer function corresponding to that direction. The head-related transfer function database 31 may be realized, for example, as a multidimensional information table expanded on a memory. Alternatively, the head-related transfer function database 31 may be realized using a database management system (DBMS).

The head-related transfer function selection unit 32 selects a head-related transfer function based on information on the position and orientation of the listener (six-axis degrees of freedom) and information on the position and orientation of the sound source (six-axis degrees of freedom). Specifically, the head-related transfer function selection unit 32 selectively acquires a head-related transfer function to be used for generating a binaural signal from the head-related transfer functions held in the head-related transfer function database 31 based on the position and orientation of the listener and the position and orientation of the sound source. In this case, the head-related transfer function selection unit 32 selects a head-related transfer function from the center of the listener's head to the direction of the sound source. Specifically, in this embodiment, the head-related transfer function selection unit 32 selects a transfer function for that direction from the head-related transfer function database 31 based on the direction from the center of the listener's head to the sound source, which is specified by the position of the center of the listener's head and the position of the sound source.

The left ear playback signal generating unit 35 is also called the "playback signal generating unit." The left ear playback signal generating unit 35 generates a playback signal for the left ear. Specifically, the left ear playback signal generating unit 35 generates a playback signal for the left ear based on the audio signal output from the sound source database 24 and the transfer function selected by the head-related transfer function selecting unit 32. Further details of the processing by the left ear playback signal generating unit 35 will be described later.

The right ear playback signal generating unit 36 is also called the "playback signal generating unit." The right ear playback signal generating unit 36 generates a playback signal for the right ear. In terms of specific processing, the right ear playback signal generating unit 36 performs the same processing for the right ear as the processing performed by the left ear playback signal generating unit 35 described above.

Next, we will explain the virtual acoustic space assumed in this embodiment.

Figure 2 is a schematic diagram showing an example of the configuration of a virtual acoustic space. The virtual acoustic space is a three-dimensional space that can be expressed in a Cartesian coordinate system of x, y, and z axes. Naturally, the virtual acoustic space can also be expressed in other than Cartesian coordinates, such as polar coordinates. As shown in the figure, a listener and a sound source (a trumpet, a wind instrument, in the example shown) exist in the virtual acoustic space. In the virtual acoustic space, sound waves emitted from the sound source in accordance with the radiation characteristics of the sound source propagate through the space and reach the listener. The listener captures the arriving sound waves with their ears and hears the sound. The signal that the listener hears is generated using a head-related transfer function that matches the direction in which the sound waves are incident on the listener in the virtual acoustic space. The binaural playback device 1 generates a signal of this playback sound. The process of generating this playback signal is called binauralization.

Figure 3 is a block diagram showing a more detailed functional configuration inside the left ear propagation path determination unit 19. As shown in the figure, the left ear propagation path determination unit 19 includes a sound source-to-head center distance calculation unit 191, a left ear-to-head center distance calculation unit 192, a left ear-to-sound source distance calculation unit 193, and a comparison determination unit 194.

To determine the propagation path to the left ear, the left ear propagation path determination unit 19 calculates the distance between the sound source and the head center, the distance between the left ear and the head center, and the distance between the left ear and the sound source in the virtual acoustic space. The left ear propagation path determination unit 19 also uses the shape of the listener's head as a determination material. Based on these, the left ear propagation path determination unit 19 determines whether the left ear can be directly seen from the sound source. That is, if the left ear can be directly seen from the sound source, the left ear propagation path determination unit 19 determines that the propagation is direct. If the left ear cannot be directly seen from the sound source, the left ear propagation path determination unit 19 determines that the propagation is diffracted. As an example, the left ear propagation path determination unit 19 may perform the above-mentioned determination of whether the ear can be directly seen on the assumption that the head is a sphere.

The sound source-to-head center distance calculation unit 191 calculates the distance between the sound source (point) and the head center in the virtual acoustic space. The sound source-to-head center distance calculation unit 191 calculates the above distance based on the coordinates (position vector) of the sound source and the coordinates (position vector) of the head center.

The left ear-to-head center distance calculation unit 192 calculates the distance between the left ear and the head center based on the coordinates of the left ear position (position vector) and the coordinates of the head center (position vector).

The left ear-to-sound source distance calculation unit 193 calculates the distance between the left ear and the sound source based on the coordinates of the left ear position (position vector) and the coordinates of the sound source (position vector).

The comparison judgment unit 194 makes a judgment based on the sound source-to-head center distance calculated by the sound source-to-head center distance calculation unit 191, the left ear-to-head center distance calculated by the left ear-to-head center distance calculation unit 192, and the left ear-to-sound source distance calculated by the left ear-to-sound source distance calculation unit 193.
When the left ear is directly visible from the sound source, the comparison and determination unit 194 determines that the propagation is direct. When the left ear is not directly visible from the sound source, the comparison and determination unit 194 determines that the propagation is diffracted.

As an example, the comparison and determination unit 194 may make the above determination based on a model in which the head is a sphere and the left ear is a point on the surface of the sphere. In this case, the comparison and determination unit 194 compares the square of the distance between the sound source and the center of the head with the sum of the squares of the distance between the left ear and the center of the head and the distance between the left ear and the sound source. The following determination method can be applied by using a geometric arrangement using a circle on a plane as the shape of the head and straight lines between each part. If the square of the distance between the sound source and the center of the head is greater than the square of the distance between the left ear and the center of the head and the sum of the squares of the distance between the left ear and the sound source, or if both are equal, the comparison and determination unit 194 may determine that the left ear can be seen directly from the sound source. If the square of the distance between the sound source and the center of the head is smaller than the square of the distance between the left ear and the center of the head and the sum of the squares of the distance between the left ear and the sound source, the comparison and determination unit 194 may determine that the left ear cannot be seen directly from the sound source.

The comparison and determination unit 194 may also make a determination based on a model other than the above-mentioned simple geometric model.

The right ear propagation path determination unit 20 has a configuration for the right ear similar to the above-mentioned left ear propagation path determination unit 19. With such a configuration, the right ear propagation path determination unit 20 determines whether the propagation path to the right ear is direct propagation or diffracted propagation.

Further details of the processing by the left ear propagation path determination unit 19 and the right ear propagation path determination unit 20 will be described later.

Figure 4 is a block diagram showing a detailed internal functional configuration of the left ear playback signal generation unit 35. As shown in the figure, the left ear playback signal generation unit 35 includes an audio signal acquisition unit 351 and a binaural signal generation unit 353. The functions of each of these units are as follows:

The audio signal acquisition unit 351 acquires an audio signal corresponding to the radiation direction of the sound source with respect to the left ear, which is output from the sound source database 24. The audio signal acquisition unit 351 outputs the acquired audio signal to the binaural signal generation unit 353.

The binaural signal generation unit 353 uses the left ear component of the head-related transfer function to generate a playback signal for the left ear from the audio signal output from the audio signal acquisition unit 351. Specifically, the binaural signal generation unit 353 generates a playback signal from the audio signal acquired by the audio signal acquisition unit 351 based on the transfer function selected by the head-related transfer function selection unit 32 from the head-related transfer function database 31.

As a modified example, the binaural signal generator 353 may generate a left-ear playback signal using the left-ear component of the head impulse response. The head impulse response can be regarded as a parameter set that represents the head-related transfer function in the time domain.

The right-ear playback signal generating unit 36 has a configuration for the right ear similar to that of the left-ear playback signal generating unit 35 described above. With such a configuration, the right-ear playback signal generating unit 36 generates a playback signal for the right ear.

Further details of the processing by the left ear playback signal generation unit 35 and the right ear playback signal generation unit 36 will be described later.

Figure 5 is a flowchart showing the procedure of the process for generating a playback signal by the binaural playback device 1. Note that this flowchart shows the process of generating a playback signal for either the left ear or the right ear, whichever ear is being focused on. To generate playback signals for both the left and right ears, the process of this flowchart can be executed for each ear. The following description will be given in accordance with this flowchart.

In step S11, the listener information acquisition unit 11 acquires information (six-axis degrees of freedom) about the listener's position and orientation (posture) in the virtual acoustic space.

In step S12, the sound source information acquisition unit 12 acquires information (six-axis degrees of freedom) about the position and orientation (posture) of the sound source in the virtual acoustic space.

In step S13, the listener's head shape acquisition unit 15 acquires the shape of the listener's head. The head shape information acquired by the listener's head shape acquisition unit 15 includes information on the position (coordinates) of each of the left and right ears when the center point of the head is used as a reference point. Then, either the left ear coordinate acquisition unit 17 or the right ear coordinate acquisition unit 18, which corresponds to the ear of interest, determines the coordinates of the ear of interest (left ear or right ear) in the virtual acoustic space based on the information on the position and orientation of the listener acquired by the listener information acquisition unit 11 and the information on the position of the ear of interest acquired by the listener's head shape acquisition unit 15. In other words, either the left ear coordinate acquisition unit 17 or the right ear coordinate acquisition unit 18 acquires and outputs the coordinate values of the left ear or right ear in the coordinate system of the virtual acoustic space.

In step S14, either the left ear propagation path determination unit 19 or the right ear propagation path determination unit 20, whichever corresponds to the ear of interest, determines whether or not the ear of interest is directly visible from the position of the sound source. If it is directly visible, the propagation to that ear is direct propagation. If it is not directly visible (such as when the sound source is located in the shadow of the head), the propagation to that ear is diffracted propagation. The left ear propagation path determination unit 19 or the right ear propagation path determination unit 20 outputs information on whether the propagation to that ear is direct propagation or diffracted propagation.

In step S15, the left ear sound source radiation direction determining unit 21 or the right ear sound source radiation direction determining unit 22, whichever corresponds to the currently focused ear, branches the process based on the determination result received from the left ear propagation path determining unit 19 or the right ear propagation path determining unit 20, whichever corresponds to the ear. Specifically, if the propagation is direct (step S16: YES), proceed to step S18. If the propagation is not direct, i.e., if the propagation is diffracted (step S16: NO), proceed to step S17.

When the process proceeds to step S17 (i.e., in the case of diffraction propagation), in this step, either the left-ear sound source radiation direction determination unit 21 or the right-ear sound source radiation direction determination unit 22, whichever corresponds to the ear currently being considered, derives a path of diffraction and propagation. The diffraction and propagation path is the path that has the shortest total length among the paths from the sound source that reaches a point on the surface of the head to the ear along the head surface.

For example, if we assume that the head is spherical, the shortest diffraction path to reach the ear can be uniquely determined, unless the ear is located directly opposite the sound source across the head (in other words, unless there is a straight line connecting the ear, the center of the head (sphere), and the sound source).

After processing in step S17, proceed to step S18.

In step S18, either the left ear sound source radiation direction determining unit 21 or the right ear sound source radiation direction determining unit 22, which corresponds to the ear currently being considered, determines the radiation direction from the sound source by a method appropriate for either direct propagation or diffraction propagation. In the case of direct propagation, the direction of a straight line connecting the sound source to the ear being considered is the radiation direction from the sound source. In the case of diffraction propagation, the direction of radiation from the sound source is the direction of a straight line connecting the sound source to the "point on the head surface" on the "path with the shortest total length among the paths from the sound source to a point on the head surface and along the head surface to the ear" described in step S17.

An example of how to find the radial direction when assuming the head is a sphere will be explained in more detail later.

In step S19, the sound source database 24 determines the ear of interest as the left ear and outputs the audio signal corresponding to the radiation direction input from the left ear sound source radiation direction determination unit 21 to the left ear playback signal generation unit 35 as an audio signal according to the radiation direction of the sound source for the left ear. Also, the sound source database 24 determines the ear of interest as the right ear and outputs the audio signal corresponding to the radiation direction input from the right ear sound source radiation direction determination unit 22 to the right ear playback signal generation unit 36 as an audio signal according to the radiation direction of the sound source for the right ear.

In step S20, the head-related transfer function selection unit 32 selects a head-related transfer function to be used to generate a playback signal from among the head-related transfer functions stored in the head-related transfer function database 31. Specifically, the head-related transfer function selection unit 32 selects a left-ear head-related transfer function and a right-ear head-related transfer function as head-related transfer functions from the center of the listener's head in the direction of the sound source, based on information on the position and orientation of the listener acquired from the listener information acquisition unit 11 and information on the position and orientation of the sound source acquired from the sound source information acquisition unit 12.

In step S21, either the binaural signal generator 353 of the left ear playback signal generator 35 or the binaural signal generator 363 (not shown) of the right ear playback signal generator 36, whichever corresponds to the ear currently being focused on, generates a playback signal for that ear. Specifically, either the binaural signal generator 353 (for the left ear) or the binaural signal generator 363 (for the right ear) generates a playback signal for that ear using an audio signal corresponding to the radiation direction of the sound source for that ear and a head related transfer function for the ear on the side of focus.

In step S22, either the binaural signal generating unit 353 (for the left ear) or the binaural signal generating unit 363 (for the right ear) outputs the playback signal (signal for binaural playback) generated in step S21 for the ear of interest.

Next, we will formulate and explain an algorithm for implementing the binaural reproduction device 1. Note that this algorithm will be explained in the following five sections.
1) Setting conditions 2) Selection of sound source radiation characteristics: Condition for sound to propagate directly from the sound source to the ear 3) Selection of sound source radiation characteristics: Condition for sound to diffract through the head and propagate to the ear 4) Selection of head-related transfer functions 5) Generation of binaural playback signals

[1. Condition setting]
FIG. 6 is a schematic diagram showing a virtual acoustic space. The diagram shows a perspective view of a three-dimensional space that is the virtual acoustic space. In this virtual acoustic space, a model of the virtual listener's head exists. The hemisphere shown by the dashed line in the diagram corresponds to the upper hemisphere of the listener's head (which may be assumed to be a sphere). The radius of the sphere corresponding to the head is a. This head model has a right ear position and a left ear position. In addition, the sound source in this virtual acoustic space is a point sound source that does not have a volume. In addition, the sound source has directionality with respect to the sound it emits.

As shown in equation (1) below, e is an orthonormal basis along the x-, y-, and z-axes of a three-dimensional orthogonal coordinate system that serves as the reference in the virtual acoustic space.

As shown in the following formula (2), e _L is a three-dimensional orthonormal basis based on the head of the listener in the virtual acoustic space. e _L is obtained by rotating (yaw, roll, pitch) the above e.

For convenience, the directions are as follows:

を正面方向（リスナーの顔が向く方向）とする。

is the front direction (the direction the listener faces).

を左耳方向（リスナーの頭部中心から左耳への方向）とする。

is the direction to the left ear (the direction from the center of the listener's head to the left ear).

を右耳方向（リスナーの頭部中心から右耳への方向）とする。

is the direction to the right ear (the direction from the center of the listener's head to the right ear).

The listener information acquisition unit 11 acquires information on the position and orientation (rotation) of the listener.

As shown in the following formula (3), e _s is a three-dimensional orthonormal basis based on a sound source in a virtual acoustic space. e _s can be obtained by rotating (yaw, roll, pitch) the above e.

を、便宜上、音源の正面方向とする。

For convenience, the direction is defined as the front direction of the sound source.

は、バーチャル音響空間における音源の位置ベクトルである。

is the position vector of the sound source in the virtual acoustic space.

The sound source information acquisition unit 12 acquires information on the position and orientation (rotation) of the sound source (point sound source).

は、バーチャル音響空間における、リスナーの頭部中心の位置ベクトルである。

is the position vector of the center of the listener's head in the virtual acoustic space.

は、バーチャル音響空間における、リスナーの左耳の位置ベクトルである。

is the position vector of the listener's left ear in the virtual acoustic space.

は、バーチャル音響空間における、リスナーの右耳の位置ベクトルである。

is the position vector of the listener's right ear in the virtual acoustic space.

は、左右の耳の位置ベクトルのうち、現在着目している側の耳の位置ベクトルである。

is the position vector of the ear currently being considered, out of the position vectors of the left and right ears.

The above e _L and e _s are expressed as the following equations (4) and (5) using transformation matrices W _L and W _s , respectively.

The transformation matrix W _L is acquired by the listener information acquisition unit 11. The transformation matrix W _s is acquired by the sound source information acquisition unit 12. Note that the transformation matrices W _L and W _s are not affine transformations that take into account the movement of the listener or sound source, but are rotation-only operators. Therefore, the position vectors of both ears are expressed as in the following formulas (6) and (7).

[2. Selection of sound source radiation characteristics: conditions for direct propagation from the sound source to the ear]
The propagation path derivation unit determines whether the propagation path is direct propagation. The sound source radiation direction determination unit determines the radiation direction from the sound source based on the path derived by the propagation path derivation unit. The sound source radiation characteristic selection unit determines (selects) the radiation characteristic from the sound source based on the radiation direction.

In the virtual acoustic space described above, the condition for sound waves to travel directly from the sound source to the listener's ear (left or right ear) is that the ear is visible from the sound source. This can be expressed as equation (8) below.

The direction from the sound source to the currently focused ear is expressed by equation (9) below.

The acoustic characteristics of a sound wave radiated along the direction expressed by the above formula (9) are the specific directional components in the radiation characteristics of the sound source. The specific direction is expressed by the following formula (10). Note that W _s ^-1 is the inverse matrix of W _s .

As described above, the sound source database 24 is constructed in advance. The sound source database 24 holds audio signals having acoustic characteristics for each direction. In other words, when a direction is specified and the sound source database 24 is queried, the sound source database 24 returns an audio signal having the acoustic characteristics for that direction.

In the three-dimensional space spanned by the orthonormal basis e, the vector indicating the radiation direction of the sound source with respect to the currently focused ear is expressed by the above equation (10).

への正射影オペレーターを、下の式（１１）とする。

The orthogonal projection operator onto is expressed as the following equation (11).

への、式（１０）のベクトルの、正射影は、上の式（１１）のオペレーターを用いて、下の式（１２）のように表される。

The orthogonal projection of the vector of equation (10) onto is expressed as equation (12) below using the operator of equation (11) above.

を基準とした方位角

は、下の式（１３）で表わされる。

Azimuth angle based on

is expressed by the following equation (13).

In the three-dimensional space spanned by the orthonormal basis e, the vector indicating the radiation direction of the sound source with respect to the currently focused ear is expressed by the above equation (10).

への正射影オペレーターを、下の式（１４）とする。

The orthogonal projection operator onto is expressed as equation (14) below.

への、式（１０）のベクトルの、正射影は、上の式（１４）のオペレーターを用いて、下の式（１５）のように表される。

The orthogonal projection of the vector of equation (10) onto is expressed as equation (15) below using the operator of equation (14) above.

を基準とした仰角

は、下の式（１６）で表わされる。

Elevation angle based on

is expressed by the following equation (16).

[3. Selection of radiation characteristics of sound source: Conditions for sound to diffract through the head and propagate to the ears]
The propagation path derivation unit determines whether the propagation path is a diffraction propagation. The sound source radiation direction determination unit determines the radiation direction from the sound source based on the path derived by the propagation path derivation unit. The sound source radiation characteristic selection unit determines (selects) the radiation characteristic from the sound source based on the radiation direction.

In a virtual acoustic space, the condition for sound waves to diffract through the listener's head and propagate from a sound source to the listener's ear (left or right ear) is that the ear is not visible from the sound source. This can be expressed as equation (17) below.

Figure 7 is a schematic diagram showing a plane Q including the coordinate positions of the listener's ears and the coordinate position of the sound source in the virtual acoustic space. The coordinates of the ears and the sound source are as follows:

の終点が両耳（それぞれ、左耳および右耳）の座標である。

The end points of are the coordinates of both ears (left ear and right ear, respectively).

の終点が音源の座標である。

The end point of is the coordinate of the sound source.

Consider the condition m, n>0 in plane Q.

はベクトルであり、下の式（１８）、式（１９）、式（２０）を満たす。なお、式（１９）および式（２０）の左辺の演算子「＜｜＞」は、ベクトルの内積をとる演算を表す。

is a vector and satisfies the following formulas (18), (19), and (20). Note that the operator "<|>" on the left side of formulas (19) and (20) represents an operation for taking an inner product of vectors.

は、リスナーの頭部に対応する球面上の一点の位置ベクトルであり、この点は平面Ｑに属する。音源から、直線による伝播で当該球面上の一点に到達し、その点から球面にそって回折して右耳（この例では右耳に着目）に至る経路は多数存在するが、それらの経路のうちの長さが最短となる経路において上記のように「直線による伝播で当該球面上の一点に到達」するときのその点の位置ベクトルが

である。

is the position vector of a point on the sphere corresponding to the listener's head, and this point belongs to plane Q. There are many paths from the sound source that reach a point on the sphere by propagation in a straight line, and then diffract along the sphere to the right ear (in this example, the right ear is the focus). Among these paths, the position vector of the point when "reaching a point on the sphere by propagation in a straight line" as described above is the shortest path.

It is.

の終点は、平面Ｑにおける、音源から放射された音波の伝播方向を表す直線（頭部表面の円と接する直線）と頭部表面との接点である。このとき、音源から着目している側の耳（ここでは右耳）に最初に到達する音波は、音源から、下の式（２１）で表わす方向に放射される音波である。

The end point of is the tangent point of the head surface with a straight line (a straight line tangent to a circle on the head surface) representing the propagation direction of the sound wave radiated from the sound source on plane Q. In this case, the sound wave that first reaches the ear on the side of interest (the right ear in this case) from the sound source is the sound wave radiated from the sound source in the direction expressed by the following equation (21).

After reaching the contact point on the head surface, the sound wave is diffracted on the head surface and reaches the ear of interest (the right ear in this case). This sound wave is the component in the radiation characteristics of the sound source in the direction expressed by the following equation (22).

への正射影オペレーターを、下の式（２３）とする。

The orthogonal projection operator onto is expressed as equation (23) below.

への、式（２２）のベクトル（音源の放射方向を示すベクトル）の、正射影は、上の式（２３）のオペレーターを用いて、下の式（２４）のように表される。

The orthogonal projection of the vector of equation (22) (vector indicating the radiation direction of the sound source) onto is expressed as equation (24) below using the operator of equation (23) above.

を基準とした方位角

は、下の式（２５）で表わされる。

Azimuth angle based on

is expressed by the following equation (25).

Consider the orthogonal projection of the vector expressed by equation (22) (the vector indicating the radiation direction of the sound source) in the three-dimensional space spanned by the orthonormal basis e.

への正射影オペレーターを、下の式（２６）とする。

The orthogonal projection operator onto is expressed as equation (26) below.

への、式（２２）のベクトルの、正射影は、上の式（２６）のオペレーターを用いて、下の式（２７）のように表される。

The orthogonal projection of the vector of equation (22) onto is expressed as equation (27) below using the operator of equation (26) above.

を基準とした仰角

は、下の式（２８）で表わされる。

Elevation angle based on

is expressed by the following equation (28).

4. Selection of Head-Related Transfer Function
The head-related transfer function employed is a transfer function that corresponds to the direction indicated by the vector of Equation (29) in the three-dimensional space spanned by the orthonormal base e based on the positional relationship between the listener and the sound source. Note that the direction indicated by the vector is determined in accordance with the head-related transfer function database 31.

In the following, the azimuth angle φ _inc and the elevation angle θ _inc in polar coordinates of the vector expressed by the above equation (29) will be derived.

への正射影オペレーターを、下の式（３０）とする。

The orthogonal projection operator onto is expressed as the following equation (30).

への、式（２９）のベクトルの正射影は、上の式（３０）のオペレーターを用いて、下の式（３１）のように表される。

The orthogonal projection of the vector of equation (29) onto is expressed as equation (31) below using the operator of equation (30) above.

を基準とした方位角φ_ｉｎｃは、下の式（３２）で表わされる。

The azimuth angle φ _inc based on is expressed by the following equation (32).

への正射影オペレーターを、下の式（３３）とする。

The orthogonal projection operator onto is expressed as the following equation (33).

への、式（２９）のベクトルの正射影は、上の式（３３）のオペレーターを用いて、下の式（３４）のように表される。

The orthogonal projection of the vector of equation (29) onto is expressed as equation (34) below using the operator of equation (33) above.

を基準とした仰角θ_ｉｎｃは、下の式（３５）で表わされる。

The elevation angle θ _inc based on is expressed by the following equation (35).

5. Generation of binaural playback signals
In the following, a method of generating a binaural reproduction signal will be described for both a case in which only the shortest path from the sound source to the ears is taken into consideration, and a case in which paths other than the shortest path from the sound source to the ears are taken into consideration.

First, we will explain the case where only the shortest route is considered.

Under conditions where sound propagates directly from the sound source to the ear, the angular frequency ω component of the playback signal is expressed by the following equation (36). Note that the head-related transfer function corresponding to the azimuth angle φ, elevation angle θ, and angular frequency ω is defined as H(φ, θ, ω). Also, the audio signal radiated in the direction of the azimuth angle φ and elevation angle θ at a position at a distance d from the sound source is defined as S(d, φ, θ, ω).

は、直接伝播する条件における再生信号の角周波数ωの成分である。

is the angular frequency ω component of the read signal under direct propagation conditions.

The signal represented by the above equation (36) focuses on one ear and is one channel (the signal corresponding to that ear).

Under conditions where sound is diffracted and propagated from the sound source to the ear, the angular frequency ω component of the playback signal is expressed by equation (37) below.

は、回折伝播する条件における再生信号の角周波数ωの成分である。

is the angular frequency ω component of the reproduced signal under the condition of diffraction and propagation.

Here, we calculate the binaural playback signal for two channels (i.e., both ears) when one ear is in direct propagation and the other ear is in diffracted propagation. For example, when the left ear is in direct propagation condition and the right ear is in diffracted propagation condition (essentially the same even when the left and right are reversed), the binaural playback signal B(ω) for two channels is expressed by the following equation (38).

When the diffraction propagation conditions are met for both ears, the binaural playback signal B(ω) for two channels is expressed by the following equation (39).

Note that, although the head-related transfer function is H(φ, θ, ω) above, the distance d from the sound source to the listener can also be used as an additional argument, and the head-related transfer function can be H(d, φ, θ, ω), etc.

Next, we will explain the case where paths other than the shortest path from the sound source to the ear are also taken into consideration. In other words, here we consider the contribution of sound waves emitted from the sound source in directions other than the shortest path.

In other words, when a sound wave is emitted from a sound source in an arbitrary direction and propagates directly from the sound source to the ear of interest, and when it propagates by diffraction, the angular frequency ω component of the playback signal can be expressed by the following equations (40) and (41), respectively.

With respect to equations (40) and (41), it is as follows: φ _rad and θ _rad are the radiation directions (azimuth and elevation angles, respectively) of the sound wave from the sound source.

は、着目している耳への最短経路の方向（それぞれ、直接伝播の条件の場合と回折伝播の条件の場合の、方位角および仰角）である。

are the directions of the shortest paths to the ear under consideration (azimuth and elevation angles for direct and diffractive propagation conditions, respectively).

φ _inc and θ _inc are the directions (azimuth and elevation) from the sound source to the center of the head.

は、直接伝播の条件にあたる場合の重み付け係数である。即ち、直接伝播に係る重み付け係数は、音源からの音波の放射方向、音源から着目している耳への最短経路の方向、および音源から頭部中心への方向に対応して定まる。従って、式（４０）は、各耳に伝播する再生信号の各周波数ωの成分を、直接伝播に係る最短経路の成分（以下、直接最短経路成分）と、最短経路以外の放射方向の成分（以下、非直接最短経路成分）を合成して得られることを示す。直接最短経路成分は、音源から頭部中心への方向の伝達関数を、音源から着目している耳への最短経路の方向への音響特性を有する音声信号の周波数成分に作用、つまり乗算して得られる。非直接最短経路成分は、音源から頭部中心への方向の伝達関数に、音源から着目している耳への音源の放射方向ごとの音響特性を有する音声信号の周波数成分と直接伝播における重み係数との乗算値の放射方向間の総和、つまり加重和となる。

is a weighting coefficient when the condition of direct propagation is met. That is, the weighting coefficient related to direct propagation is determined in accordance with the radiation direction of the sound wave from the sound source, the direction of the shortest path from the sound source to the ear of interest, and the direction from the sound source to the head center. Therefore, formula (40) indicates that the component of each frequency ω of the reproduction signal propagating to each ear is obtained by combining the component of the shortest path related to direct propagation (hereinafter, direct shortest path component) and the component of the radiation direction other than the shortest path (hereinafter, non-direct shortest path component). The direct shortest path component is obtained by acting on, that is, multiplying, the frequency component of the audio signal having the acoustic characteristics in the direction of the shortest path from the sound source to the ear of interest by the transfer function in the direction from the sound source to the head center. The non-direct shortest path component is the sum between the radiation directions of the multiplication values of the frequency component of the audio signal having the acoustic characteristics for each radiation direction of the sound source from the sound source to the ear of interest and the weighting coefficient in direct propagation, that is, the weighted sum, of the transfer function in the direction from the sound source to the head center.

は、回折伝播の条件にあたる場合の重み付け係数である。即ち、回折伝播に係る重み付け係数は、音源からの音波の放射方向、音源から着目している耳への最短経路の方向および音源から頭部中心への方向に対応して定まる。従って、式（４１）は、各耳に伝播する再生信号の各周波数ωの成分を、回折伝播に係る最短経路の成分（以下、回折最短経路成分）と、最短経路以外の放射方向の成分（以下、非回折最短経路成分）を合成して得られることを示す。回折最短経路成分は、音源から頭部中心への方向の伝達関数を、音源から着目している耳への回折伝播に係る最短経路の方向への音響特性を有する音声信号周波数成分に作用、つまり、乗算して得られる。非回折最短経路成分は、音源から頭部中心への方向の伝達関数に、音源から着目している耳への音源の放射方向ごとの音響特性を有する音声信号の周波数成分と回折伝播における重み係数との加重和となる。

is a weighting coefficient when the condition of diffraction propagation is met. That is, the weighting coefficient related to diffraction propagation is determined in accordance with the radiation direction of the sound wave from the sound source, the direction of the shortest path from the sound source to the ear of interest, and the direction from the sound source to the head center. Therefore, equation (41) indicates that the component of each frequency ω of the reproduction signal propagating to each ear is obtained by combining the component of the shortest path related to diffraction propagation (hereinafter, the diffraction shortest path component) and the component in the radiation direction other than the shortest path (hereinafter, the non-diffraction shortest path component). The diffraction shortest path component is obtained by acting on, that is, multiplying, the transfer function in the direction from the sound source to the head center on the sound signal frequency component having the acoustic characteristics in the direction of the shortest path related to diffraction propagation from the sound source to the ear of interest. The non-diffraction shortest path component is the weighted sum of the transfer function in the direction from the sound source to the head center, the frequency component of the sound signal having the acoustic characteristics for each radiation direction of the sound source from the sound source to the ear of interest, and the weighting coefficient in diffraction propagation.

However, it is difficult to determine these weighting coefficients analytically. Any number (for directions) of sound source signals weighted by radiation direction may be added with any weighting. The binaural signal generation unit may preset weighting coefficients for each set of the radiation direction of sound waves from the sound source, the direction of the shortest path from the sound source to the ear of interest, and the direction from the sound source to the center of the head for each of direct propagation and diffractive propagation. From the weighting coefficients that have been set, the binaural signal generation unit can select weighting coefficients that correspond to the radiation direction of sound waves from the sound source, the direction of the shortest path from the sound source to the ear of interest, and the direction from the sound source to the center of the head.

The range and resolution (sampling fineness) of φ _rad and θ _rad , which indicate the direction from the sound source, can be freely determined as appropriate.

The binaural signal generating unit can derive the binaural signal by the above formulas (40) and (41). For the derivation of the binaural signal, see formulas (38) and (39).

At least some of the functions of the binaural playback device in the above-mentioned embodiment can be realized by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the function. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memories, and storage devices such as hard disks built into computer systems. Furthermore, the term "computer-readable recording medium" may also include devices that temporarily and dynamically hold a program, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and devices that hold a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in such a case. Furthermore, the above-mentioned program may be for realizing some of the above-mentioned functions, and may further be capable of realizing the above-mentioned functions in combination with a program already recorded in the computer system.

The options and variations of the embodiment described above will be explained below. Multiple options or variations may be combined to the extent that they can be combined.

(1) As described in the embodiment, the sound source database 24 responds with data of a sound signal having the acoustic characteristics of a sound source according to the queried argument. In this case, φ (azimuth angle), θ (elevation angle), and ω (frequency (or angular frequency)) are required as arguments, but r (distance) may be optional. In other words, the sound source database 24 may hold or generate data of a sound signal according to the distance r, or may hold or generate data of a sound signal that is independent of the distance r. The left ear sound source radiation direction determination unit 21 and the right ear sound source radiation direction determination unit 22 select a sound signal stored in the sound source database 24 using the radiation direction and frequency from the sound source as required arguments. In this case, the left ear sound source radiation direction determination unit 21 and the right ear sound source radiation direction determination unit 22 may or may not pass the distance (r) as an argument to the sound source database 24. In other words, the left-ear sound source radiation direction determining unit 21 and the right-ear sound source radiation direction determining unit 22 may select an audio signal having acoustic characteristics that depend on distance, or may select an audio signal having acoustic characteristics that do not depend on distance. The distance can be calculated from the listener's coordinates and the sound source's coordinates. The listener's coordinates may be the coordinates of the head center, the coordinates of the left ear, the coordinates of the right ear, or the coordinates of the arrival point of the sound wave on the head surface in the case of diffraction propagation (the tangent point of the circle and the straight line shown in FIG. 7), etc.

The distance (r) above is the distance from the sound source to the listener. This distance may be the straight-line distance from the sound source to the ear of interest. In the case of diffraction propagation, this distance may be the distance from the sound source to the point on the surface of the listener's head that is first reached. In the case of diffraction propagation, this distance may be the length of the entire path, including the path taken when the sound is diffracted. This distance may be approximated by the distance from the sound source to the center of the listener's head, etc.

(2) Also, when the head-related transfer function selection unit 32 selects a head-related transfer function from the head-related transfer function database 31, the head-related transfer function may be selected depending on the distance, or may be selected to be independent of the distance.

(3) The acoustic characteristics may be further classified based on various case classifications, such as by mora, by phoneme, by single note, by gender, by age, by instrument, etc. In this case, the sound source database 24 holds audio signals having the acoustic characteristics for each of the cases listed here. The left ear sound source radiation direction determination unit 21 and the right ear sound source radiation direction determination unit 22 selectively acquire from the sound source database 24 audio signals having acoustic characteristics according to each condition for the case classification.

The above modified examples (2) and (3) are implemented as follows. The sound source database 24 holds information on radiation characteristics according to at least one of the following: not only the direction, but also the distance from the sound source to the listener, the type of mora contained in the human voice emitted from the sound source, the type of phoneme contained in the human voice emitted from the sound source, the gender of the person who emitted the human voice emitted from the sound source, the age of the person who emitted the human voice emitted from the sound source, and the type of musical instrument that serves as the sound source. Human voice means a voice produced by a person speaking. In that case, the sound source can be a person. The sound source radiation direction determination unit (the sound source radiation direction determination unit 21 for the left ear and the sound source radiation direction determination unit 22 for the right ear) queries the audio signal using an argument corresponding to the configuration of the sound source database 24. That is, the sound source radiation direction determination unit selects and acquires from the sound source database 24 an audio signal having the sound source radiation characteristics according to at least one of the following: not only the direction, but also the distance from the sound source to the listener, the type of mora contained in the voice emitted from the sound source, the type of phoneme contained in the voice emitted from the sound source, the gender of the person who emitted the voice from the sound source, the age category of the person who emitted the voice from the sound source, and the type of musical instrument that serves as the sound source. The sound source radiation direction determination unit may appropriately acquire information required for the above-mentioned inquiry (information such as the type of mora, the type of phoneme, the gender of the person who emitted the voice, the age or age group of the person who emitted the voice, the type of musical instrument, etc.), or may have it preset. The sound source radiation direction determination unit may acquire information required for the above-mentioned arguments, etc., from metadata of the content, for example, or may have it preset.

(4) When generating a playback signal, the head-related transfer function is generally a response in an anechoic chamber, but a binaural room impulse response (BRIR), which is a head impulse response in an anechoic chamber, may be used instead. In this case, the head impulse response is measured in advance and stored in a head impulse response database (not shown). The head impulse response may be measured in an anechoic chamber. The head impulse response database holds head impulse responses for each ear for each direction from the head. Then, a head impulse response selection unit (not shown) selects from the head impulse response database a head impulse response for the left ear and a head impulse response for the right ear in the direction from which sound waves arrive at the head or ear. The binaural signal generating unit 353 (for the left ear) or the binaural signal generating unit 363 (for the right ear) uses the head impulse response for the left ear and the head impulse response for the right ear selected above instead of the left ear head related transfer function and the right ear head related transfer function described in the embodiment, to generate a playback signal for the left ear and a playback signal for the right ear, respectively, from the audio signal output from the sound source database 24.

That is, in this modification, instead of the head transfer function database, there is a head impulse response database. The head impulse response database holds head impulse responses according to the direction from the center of the listener's head. The head impulse response selection unit selects a specific head impulse response from the head impulse response database. More specifically, the head impulse response selection unit selects a head impulse response from the head impulse response database based on the direction from the center of the listener's head to the sound source, which is specified by the position of the center of the listener's head and the position of the sound source. Then, the playback signal generation unit generates a playback signal for the ear of interest based on the audio signal of the sound source, information on the position and direction of the sound source at the time of acquiring the audio signal, the sound source database, the radiation characteristics selected by the sound source radiation characteristic selection unit, and the head impulse response selected by the head impulse response selection unit. That is, in this modification, the playback signal generation unit generates a playback signal that is the result of the acoustic action in and near the head by using the head impulse response instead of the head transfer function.

(5) Both the head-related transfer function and the head-impulse response are used to obtain the results of acoustic actions on the listener's head and in its vicinity. In other words, anything that performs an acoustic action on the head and its vicinity, including the head-related transfer function and the head-impulse response, can be more generally called a head-related acoustic operator. The head-related acoustic operator database (including the head-related transfer function database and the head-impulse response database) holds head-related acoustic operators corresponding to at least the direction from the head. The head-related acoustic operator selection unit (including the head-related transfer function selection unit and the head-impulse response selection unit) selects a head-related acoustic operator corresponding to a specific direction from the head-related acoustic operator database based on the direction of the sound source from the head. In general, the binaural signal generation unit 353 (for the left ear) or the binaural signal generation unit 363 (for the right ear) generates a playback signal from the sound signal output from the sound source database 24 using the head-related acoustic operator selected above.

(6) As described above, the binaural playback device 1 can be configured to generate playback signals even when there are multiple sound sources. In this case, the binaural playback device 1 generates playback signals for each of the multiple sound sources by the method described above. That is, the propagation path derivation unit (the left ear propagation path determination unit 19 and the right ear propagation path determination unit 20) derives a propagation path for each sound source. The sound source radiation direction determination unit (the left ear sound source radiation direction determination unit 21 and the right ear sound source radiation direction determination unit 22) determines the radiation direction for each sound source. The head-related transfer function selection unit selects a transfer function from the head-related transfer function database 31 for each sound source. The playback signal generation unit (the left ear playback signal generation unit 35 and the right ear playback signal generation unit 36) generates a playback signal for each sound source.

(7) In the case where there are multiple sound sources, the playback signal generating unit (left ear playback signal generating unit 35 and right ear playback signal generating unit 36) may generate and output a superimposed playback signal by superimposing the playback signals generated for each sound source. The playback signal generating unit may generate and output a superimposed playback signal by superimposing the playback signals generated for all sound sources, for example. The sound source information acquiring unit 12 may also acquire information on the type, position, and direction of the sound source for each of one or more sound sources. The sound source database 24 is capable of holding or generating audio signals emitted from each type of sound source for each of the multiple types of sound sources. The sound source database outputs to the playback signal generating unit, among the audio signals corresponding to the types of individual sound sources acquired by the sound source information acquiring unit 12, audio signals having acoustic characteristics corresponding to the radiation direction determined for each individual sound source by the radiation direction determining unit. The type of each sound source may be specified in the same manner as in the "case" described in (3) above, such as the type of mora, the type of phoneme, the gender of the voice, the age or age group of the voice, the type of instrument, or a combination of these.

(8) As described above, even when there are multiple listeners, the binaural reproduction device 1 can be configured to generate a reproduction signal for each listener. In this case, the binaural reproduction device 1 generates a reproduction signal for each of the multiple listeners by the method described above. The binaural reproduction device 1 outputs the generated reproduction signal for each listener to, for example, headphones (audio output means) for each listener. That is, the propagation path derivation unit derives a propagation path for each listener. The sound source radiation direction determination unit determines the radiation direction for each listener. The head-related transfer function selection unit selects a transfer function for each listener. The reproduction signal generation unit generates a reproduction signal for each listener. With this configuration, the binaural reproduction device 1 can generate binaural reproduction signals for multiple listeners with different positions, postures, etc. for the same sound source (or a set of sound sources).

The above describes the embodiments and options or variations of this invention in detail with reference to the drawings, but the specific configuration is not limited to this embodiment and also includes designs that do not deviate from the gist of this invention.

The present invention can be used, for example, in devices and programs for binaural playback. However, the scope of use of the present invention is not limited to the examples given here.

1 Binaural reproduction device 11 Listener information acquisition unit 12 Sound source information acquisition unit 15 Listener head shape acquisition unit 17 Left ear coordinate acquisition unit 18 Right ear coordinate acquisition unit 19 Left ear propagation path determination unit (propagation path derivation unit)
20 Right ear propagation path determination unit (propagation path derivation unit)
21 Left ear sound source radiation direction determination unit (sound source radiation direction determination unit)
22 Right ear sound source radiation direction determination unit (sound source radiation direction determination unit)
24 Sound source database 31 Head-related transfer function database 32 Head-related transfer function selection unit 35 Left-ear playback signal generation unit (playback signal generation unit)
36 Right ear reproduction signal generation unit (reproduction signal generation unit)
191 Sound source-head center distance calculation unit 192 Left ear-head center distance calculation unit 193 Left ear-sound source distance calculation unit 194 Comparison determination unit 351 Audio signal acquisition unit 353 Binaural signal generation unit

Claims (10)

A head-related transfer function database that stores transfer functions according to directions from the center of the listener's head;
a propagation path derivation unit that derives a propagation path of sound from the sound source to the ears based on a position of a sound source, a head shape of the listener, and a position of the listener's ears;
a sound source radiation direction determination unit that determines a radiation direction from the sound source to the ear based on the derived propagation path;
a sound source database for outputting a sound signal having acoustic characteristics corresponding to the determined radiation direction;
a head-related transfer function selection unit that selects the transfer function from the head-related transfer function database based on a direction from a center of the head of the listener to a sound source, the direction being specified by a position of the center of the head of the listener and a position of the sound source;
a reproduction signal generating unit that generates a reproduction signal for the ear based on the output audio signal and the transfer function selected by the head-related transfer function selecting unit;
A binaural playback device comprising: The ears are a left ear and a right ear,
the head-related transfer function database holds transfer functions of the left ear and the right ear as a left-ear head-related transfer function and a right-ear head-related transfer function,
the propagation path derivation unit derives the propagation paths of the sound from the sound source to the left ear and the right ear,
the sound source radiation direction determination unit determines the radiation direction from the sound source to each of the left ear and the right ear,
the sound source database selects an audio signal having acoustic characteristics corresponding to a radiation direction to the left ear as an audio signal for the left ear, and selects an audio signal having acoustic characteristics corresponding to a radiation direction to the right ear as an audio signal for the right ear,
the head-related transfer function selection unit selects a left-ear head-related transfer function and a right-ear head-related transfer function corresponding to a direction to the sound source,
the reproduction signal generation unit generates the reproduction signal for the left ear based on the left ear audio signal and the left ear head-related transfer function, and generates the reproduction signal for the right ear based on the right ear audio signal and the right ear head-related transfer function.
2. A binaural reproduction device according to claim 1. The sound sources are multiple,
the sound source database outputs the audio signal for each of the sound sources;
the propagation path derivation unit derives the propagation path for each of the sound sources,
the sound source radiation direction determination unit determines the radiation direction for each of the sound sources,
The head-related transfer function selection unit selects the transfer function from the head-related transfer function database for each of the sound sources,
The reproduction signal generating unit generates the reproduction signal for each of the sound sources.
3. A binaural reproduction device according to claim 1 or 2. The reproduction signal generating unit generates a superimposed reproduction signal by superimposing the reproduction signals generated for the respective sound sources.
4. A binaural reproduction device according to claim 3. the listener is a plurality of listeners,
the propagation path derivation unit derives the propagation path for each of the listeners,
the sound source radiation direction determination unit determines the radiation direction for each of the listeners,
The head-related transfer function selection unit selects the transfer function for each of the listeners,
the reproduction signal generation unit generates the reproduction signal for each of the listeners;
A binaural reproduction device according to any one of claims 1 to 4. The sound source database includes:
the distance from the sound source to the listener; and
A type of mora contained in the human voice emitted from the sound source;
types of phonemes contained in the human voice emitted from the sound source;
The gender of the voice generated by the sound source; and
Age classification of the voice generated from the sound source;
The type of musical instrument of the sound source;
and storing an audio signal having the acoustic characteristics corresponding to at least one of the above.
the distance from the sound source to the listener; and
A type of mora contained in the human voice emitted from the sound source;
types of phonemes contained in the human voice emitted from the sound source;
The gender of the voice generated by the sound source; and
Age classification of the voice generated from the sound source;
The type of musical instrument of the sound source;
Selecting an audio signal having the acoustic characteristics corresponding to at least one of the above.
A binaural reproduction device according to any one of claims 1 to 5. the propagation path derivation unit derives the propagation path in which direct propagation from the sound source to the ear is the shortest path when the ear is visible from the sound source, and in which diffracted propagation from the sound source to the ear through the head is the shortest path when the ear is not visible from the sound source,
The sound source radiation direction determination unit determines the radiation direction depending on whether the propagation path is due to direct propagation or due to diffractive propagation.
A binaural reproduction device according to any one of claims 1 to 6. The reproduction signal generating unit
a binaural signal generator that generates a playback signal for the ear by synthesizing a shortest path component related to the shortest path direction and a non-shortest path component related to a radiation direction other than the shortest path direction, among components of the sound arriving at the ear from the sound source;
The binaural signal generating unit includes:
applying a transfer function according to a direction from the center of the head of the listener to an audio signal having acoustic characteristics in the direction of the shortest path to generate the shortest path component;
generating the non-shortest path components by applying a transfer function according to a direction from the center of the head of the listener to a weighted sum of a single or a plurality of sound signals having acoustic characteristics corresponding to each direction of a propagation path other than the shortest path, the weighted sum being based on a weighting coefficient related to the propagation path;
The binaural reproduction device according to claim 7 , wherein the weighting coefficients are determined in accordance with a radiation direction of a sound wave from the sound source, a direction of the shortest path, and a direction from the sound source to the center of the head. a head impulse response database that stores head impulse responses according to directions from the center of the listener's head;
a propagation path derivation unit that derives a propagation path of sound from the sound source to the ears based on a position of a sound source, a head shape of the listener, and a position of the listener's ears;
a sound source radiation direction determination unit that determines a radiation direction from the sound source to the ear based on the derived propagation path;
a sound source database for outputting a sound signal having acoustic characteristics corresponding to the determined radiation direction;
a head impulse response selection unit that selects the head impulse response from the head impulse response database based on a direction from a center of the listener's head to a sound source, the direction being specified by a position of the center of the listener's head and a position of the sound source;
a reproduction signal generating unit that generates a reproduction signal for the ear based on the output audio signal and the head impulse response selected by the head impulse response selecting unit;
A binaural playback device comprising: Computer,
A binaural reproduction device according to any one of claims 1 to 9,
A program to function as a

Images