JP6592838B2 - Binaural signal generation apparatus, method, and program - Google Patents

Description

  The present invention relates to a technique for collecting a sound signal with a microphone installed in a certain sound field and reproducing the sound field using the sound signal.

  As a technique for generating a binaural signal for virtually reproducing a remote sound field using a plurality of microphones, for example, a technique described in Non-Patent Document 1 is known. In the technique described in Non-Patent Document 1, a binaural signal is generated by convolving one head related transfer function (HRTF) corresponding to one direction.

Tatsuya Hirahara, et al., "Problems related to measurement of head-related transfer functions and binaural reproduction," Fundamentals Review, IEICE Fundamentals / Boundary Society Fundamentals Review, vol. 2, no. 4, pp. 68-85, 2009.

  In the technique described in Non-Patent Document 1, only one head-related transfer function is used. For this reason, the accuracy of the generated binaural signal is not necessarily high.

  An object of the present invention is to provide a binaural signal generation apparatus, method, and program for generating a binaural signal with higher accuracy.

A binaural signal generation device according to an aspect of the present invention matches a desired sound field with a synthesized sound field in consideration of a plurality of head-related transfer functions based on reproduced sounds from circular speaker arrays arranged in a circle. It includes a filtering unit that generates a binaural signal by convolving the obtained filter and a circular microphone array observation signal in a spatio-temporal direction, j is an imaginary number, k is a wave number, and J _m Be an m-order Bessel function and H _m ⁽¹⁾ 'M-th order Hankel function H _m ⁽¹⁾ The position of one of the listener's ears in the cylindrical coordinate system is expressed as (R _L , φ _L , 0) and ~ G ^L Is the head-related transfer function expressed in the cylindrical harmonic function region, ~ G is the transfer function expressed in the cylindrical harmonic function region of each speaker, and ~ G is the radius R _ref , R _m Is the radius of the circular microphone array and R _s Where F is defined by the following equation (A) or (B) ^L Is a filter obtained by inverse Fourier transform.

The binaural signal generation device according to one aspect of the present invention is an observation using a spherical microphone array arranged in a spherical shape with a filter obtained by matching a desired sound field and a synthesized sound field in consideration of a spherical speaker array arranged in a spherical shape. A filtering unit that obtains a signal by space-time convolution of the signal, a time signal corresponding to each speaker position of the spherical speaker array of the obtained signal, and a plurality of head-related transfer functions based on reproduced sound from the spherical speaker array A signal generator for generating a binaural signal in the time domain by adding together the time signals corresponding to the respective speaker positions of the spherical speaker array in the time direction by the number of speakers constituting the spherical speaker array. , J is an imaginary number, k is a wave number, and n and m are spherical harmonic functions Y _n ^m And the position r of the spherical speaker array in the spherical coordinate system. _s R _s = (R _s , θ _s , φ _s ) And h _n ⁽¹⁾ 'N-th order first-class sphere Hankel function h _n ⁽¹⁾ ~ G is the transfer function expressed in the cylindrical harmonic function region of the speaker constituting the spherical speaker array, and ~ G is the radius R _ref , R _m Is the radius of the spherical microphone array and R _s Is the radius of the spherical speaker array, and the filter is defined by the following formula (E) or formula (F). ^L It is.

  By using a plurality of head-related transfer functions, a highly accurate binaural signal can be generated.

The functional block diagram which shows the example of the binaural signal generator of 1st embodiment. The flowchart which shows the example of the binaural signal generation method of 1st embodiment. The figure for demonstrating a coordinate system. The functional block diagram which shows the example of the binaural signal generator of 1st embodiment. The flowchart which shows the example of the binaural signal generation method of 1st embodiment.

Embodiments of the present invention will be described below with reference to the drawings. In the following explanation, the symbols “~”, “ ^- ”, “^”, etc. used in the text should be written directly above the previous character, but immediately after the character due to restrictions on the text notation. It describes. In the formula, these symbols are written in their original positions. Further, the processing performed for each element of a vector or matrix is applied to all elements of the vector or matrix unless otherwise specified.

[First embodiment]
The first embodiment relates to a binaural signal generation apparatus and method for generating a binaural signal based on signals collected by a two-dimensionally arranged circular microphone array. First, the technical background of the first embodiment will be described.

<Technical background>
It is assumed that a circular speaker array composed of three or more speakers is arranged on the circumference of the radius R _s and the head exists at the center of the circle. The speaker position is written as r _s = (R _s , φ _s , 0) in a cylindrical coordinate system (see FIG. 3). The positions of the left and right ears are r _L = (R _L , φ _L , 0) and r _R = (R _R , φ _R , 0), respectively, and the HRTF in the time-frequency domain is set to G ^L (r _L −r _s ), G ^R (r _R -r _s ). G ^L (r _L -r _s ) and G ^R (r _R -r _s ) are measured in advance and are assumed to be known. It is not always necessary to measure the HRTF using a speaker array, and the HRTF may be measured separately from each direction. The head shape may be modeled and the HRTF may be obtained by simulation. Hereinafter, only the time frequency domain signal P ^L (r _L ) of the left ear will be discussed, but the same argument holds for the right ear.

Considering the synthesis of the left ear signal P ^L (r _L ) by convolving the signal D (r _s ) with the transfer function from each speaker position, P ^L (r _L ) is I can write.

  Here, a variable with “˜” represents that the cylindrical harmonic function region is expressed, and m is an order in the cylindrical harmonic function region. Further, j is an imaginary number, e is a Napier number, and π is a circular ratio.

Now, D (•) may be a drive signal for synthesizing a desired sound field P ^des (r).

Let r = (r, φ, 0) be an arbitrary position in the inner region of the circle with radius R _s , and let P ^syn (r) be the sound field synthesized using a speaker on the circumference without a head. To do. At this time,

It becomes. Here, G (rr _s ) represents the transfer characteristic of each speaker, and is obtained by modeling a measurement or physical phenomenon (an example of modeling as a line sound source will be described later). In the past, signal conversion was formulated assuming plane wave propagation from the speaker direction at the time of obtaining HRTF, but here the following formulation is performed assuming spherical wave propagation from a point source. .

It is assumed that a cylindrical harmonic spectrum ~ P ^des (r, m) = ⁻ P ^des (m) J _m (kr) of the desired sound field is estimated on the sound collection side. This may be synthesized by simulation. Here, k is the wave number, and J _m is the mth order Bessel function. Also, ^- P ^des (m) are the coefficients of the cylindrical harmonic expansion of the desired sound field. Since P ^syn (r) and P ^des (r) need to match,

It becomes. When acquiring ~ G (rR _s , m) on the circumference of radius R _ref

It becomes.
Therefore, a binaural signal can be obtained by substituting Equation (2) into Equation (1).

Cylindrical harmonic spectrum of the desired sound field ~ P ^des (r, m) = P ^des (m) J _m (kr) has several estimation methods, but here on a cylindrical rigid body baffle with radius R _m Consider using an array of circular microphones. The circular microphone array is composed of three or more microphones. Here, a baffle means what attaches a microphone and a speaker. The presence of the baffle itself affects the sound transfer function during sound collection or playback. Further, depending on whether the baffle is a rigid body or a sound absorber, the relational expression between the cylindrical harmonic spectrum of the desired sound field and the cylindrical harmonic spectrum of the observation signal is also affected.

Assuming that the cylindrical harmonic spectrum of the observation signal from the microphone array is ~ P ^rcv (R _m , m), ~ P ^des (r, m) is as follows.

Here, H _m ⁽¹⁾ ′ is a derivative of the m-th order first-class Hankel function H _m ⁽¹⁾ . As another example, consider the use of a circular microphone array of radius R _m on the spherical rigid radius R _m baffle. Assuming that the cylindrical harmonic spectrum of the observation signal from the microphone array is ~ P ^rcv (R _m , m), ~ P ^des (r, m) is as follows.

Note that ~ P ^des (r, m) may be defined by an expression other than the above, depending on the arrangement of the microphone array and the shape and properties of the baffle (properties such as rigid bodies, sound absorbers, and arrangement in the air). .

Here, P _n ^m is the associated Legendre functions. Substituting equation (4) into equation (3),

It becomes. From this equation, a binaural signal can be synthesized from a circular microphone array signal by spatiotemporal convolution of a filter in the time-frequency domain defined below.

Rather, a filter in the spatial domain when the determined by inverse Fourier transform of F ^L is the convolution. That is, using the filter when space region obtained by inverse Fourier transform of F ^L, by performing a convolution space when spatiotemporal contrast observation signals of the circular microphone array, it is possible to generate a binaural signal.

The filter coefficient ~ F ^L (m) in the spatio-temporal frequency domain of order m

In other words, the binaural signal can be expressed by the filter coefficient ~ F ^L (m) in the spatio-temporal frequency domain of order m and the circular microphone array signal as follows.

  Equation (5) was assumed to use ~ G obtained by measurement, but here we consider the case of modeling as a line source.

Than

It becomes. However, since an actual speaker has characteristics closer to a point sound source than a line sound source, it is necessary to multiply (2πj / k) ^1/2 as a correction term.

The follow-up to the movement of the listener's head can be realized by processing the cylindrical harmonic spectrum acquired by the circular microphone array on the recording side. For simple rotation, a simple phase shift may be applied. Cylindrical harmonic spectrum ~ P ^rot (R _m , m) using a phase shift with the rotation angle as φ _rot is

It becomes.
It is also possible to estimate a cylindrical harmonic spectrum with the origin at a position different from the center of the circular microphone array. For example, a cylindrical conditioning spectrum at position ^{_{(R t, φ t) -}} P trans is

As obtained. When cylindrical harmonic spectra at a plurality of positions can be acquired, estimation by the least square method is possible.

A binaural signal may be generated by performing convolution in the spatial direction using the filter of Expression (A) or Expression (B) as shown in the following expression. The convolution in the spatial direction when generating a binaural signal, where there is a need to be the same as the number of microphones constituting the number = circular microphone array speaker that constitutes a circular speaker array, N _L is the speaker Number = number of microphones. φ _i is a direction corresponding to φ _L.

<Binaural signal generation apparatus and method>
The binaural signal generation device according to the first embodiment includes a filtering unit 1 and a filter generation unit 2 as shown in FIG. The binaural signal generation apparatus implements the binaural signal generation method by executing the process of step S1 shown in FIG.

  The filtering unit 1 receives a filter obtained by matching a desired sound field and a synthesized sound field in consideration of a plurality of head-related transfer functions based on reproduced sounds from circular speaker arrays arranged in a circle. The

  This filter is a filter obtained by performing an inverse Fourier transform on a filter defined by Expression (A) or Expression (B). This filter is generated in advance by the filter generation unit 2 prior to the processing of the filtering unit 1.

  In addition, an observation signal from a circular microphone array arranged in a circle is input to the filtering unit 1.

  The filtering unit 1 generates a binaural signal by convolving the input filter and the input observation signal in the space-time direction (step S1).

  The processing of the filtering unit 1 is only an example. The filtering unit 1 may generate a binaural signal by the other filter processing or convolution processing described in the section <Technical background>.

That is, for example, the filtering unit 1, instead of the filter F ^L, may be performed filtering processing using the filter which is obtained by inverse Fourier transform of F ^L.

Further, for example, when the listener changes the direction by φ _rot using ~ P ^rcv (R _m , m) as the cylindrical harmonic spectrum of the observed signal, the filtering unit 1 replaces the observed signal with the formula (C ) ~ P ^rcv (R _m , m) defined by

If the listener moves to the position (R _t, φ _t), instead of the the observed signal is defined by the formula ^{(D) - P trans (R} m, m) and may convolving the above filter.

  Thus, by considering a plurality of head related transfer functions, a highly accurate binaural signal can be generated.

[Second Embodiment]
The second embodiment is a binaural signal generation apparatus and method for generating a binaural signal based on signals collected by a spherical microphone array arranged three-dimensionally. First, the technical background of the second embodiment will be described.

<Technical background>
It is assumed that a spherical speaker array composed of three or more speakers is arranged on a spherical surface with a radius R _s and that a head exists at the center thereof. The speaker position is written as r _s = (R _s , θ _s , φ _s ) in a spherical coordinate system. The left and right ear positions are r _L = (R _L , θ _L , φ _L ) and r _R = (R _R , θ _R , φ _R ), respectively, and the HRTF in the time-frequency domain is set to G ^L (r _L − r _s ), G ^R (r _R -r _s ). G ^L (r _L -r _s ) and G ^R (r _R -r _s ) are measured in advance and are assumed to be known. It is not always necessary to measure the HRTF using a speaker array, and the HRTF may be measured separately from each direction. The head shape may be modeled and the HRTF may be obtained by simulation. Hereinafter, only the time frequency domain signal P ^L (r _L ) of the left ear will be discussed, but the same argument holds for the right ear.

Considering the synthesis of the left ear signal P ^L (r _L ) by convolving the signal D (r _s ) with the transfer function from each speaker position, P ^L (r _L ) is I can write.

  Note that, unlike the first embodiment using a circular array, the second embodiment using a spherical array cannot be described as a convolution of space.

Now, D (•) may be a drive signal for synthesizing a desired sound field. Let r = (r, θ, φ) be an arbitrary position in the inner region, and let P ^syn (r) be a sound field synthesized using a spherical speaker without a head. At this time, assuming the axial symmetry of each speaker, the north pole position of η = (0,0, R _s )

It becomes. SO (3) is a third-order rotation group (special orthogonal group). Here, G (rr _s ) represents the transfer characteristic of each speaker, and is obtained by measurement or modeling. In the past, signal conversion was formulated assuming a plane wave from a sound source at the time of obtaining HRTF, but here the following formulation is performed assuming a spherical wave from a point sound source. Y _n ^m is a spherical harmonic function, and n and m are their orders. Here, the variable with “˜” represents the spherical harmonic spectrum region.

It is assumed that the spherical harmonic spectrum ~ P ^des (r, n, m) = ⁻ P ^des (n, m) j _n (kr) of the desired sound field is estimated on the sound collection side. This may be synthesized by simulation. Since P ^syn (r) and P ^des (r) should match,

It becomes. j _n is an nth-order first-order spherical Bessel function. When obtaining ~ G (rR _s , n, 0) on a sphere with radius R _ref

It becomes.
Therefore, a binaural signal can be obtained by substituting Equation (7) into Equation (6).

Desired sound field of spherical harmonic spectrum ^{~ P des (r, n,} m) = - P des (n, m) j n (kr) is a number of estimation methods, here, the radius R _m sphere baffle Consider using a spherical microphone array placed above. The spherical microphone array is composed of three or more microphones.

  Here, a baffle means what attaches a microphone and a speaker. The presence of the baffle itself affects the sound transfer function during sound collection or playback. Further, depending on whether the baffle is a rigid body or a sound absorber, the relational expression between the spherical harmonic spectrum and the spherical harmonic spectrum of the observation signal is affected.

Assuming that the spherical harmonic spectrum of the observation signal from the microphone array is ~ P ^rcv (R _m , n, m), ~ P ^des (r, n, m) is as follows. h _n ⁽¹⁾ ′ means a derivative of the _n- th order first-class Hankel function h _n ⁽¹⁾ .

  Substituting into this equation (8),

It becomes. S is a set of speakers constituting the spherical speaker array used for HRTF measurement. Unlike a circular array, signal conversion with a single filter convolution is not possible, but after applying the filter of the following equation (E) to the microphone array signal, convolution with each HRTF is measured for HRTF. The binaural signal may be a signal obtained by adding the number of speakers used in the above.

  That is, a time signal corresponding to each speaker position of the spherical speaker array of the obtained signal is obtained by space-time convolution of the filter of equation (E) and the observation signal from the spherical microphone array arranged in a spherical shape. And the time signal corresponding to each speaker position of the spherical speaker array of a plurality of head-related transfer functions based on the reproduced sound from the spherical speaker array is the number of speakers constituting the spherical speaker array The binaural signal in the time domain is generated by adding together.

The filter coefficient ~ F ^L (r _s , n, m) of order n, m

, The binaural signal is expressed as follows using a filter coefficient of order n and m and a circular microphone array signal.

  Equation (F) was assumed to use ~ G obtained by measurement, but here, consider the case of modeling as a point sound source.

Than

It becomes. The filter of the formula (F) may be used instead of the filter of the formula (E).

<Binaural signal generation apparatus and method>
The binaural signal generation device of the second embodiment includes a filtering unit 1 and a filter generation unit 2 as shown in FIG. The binaural signal generation apparatus implements the binaural signal generation method by executing the process of each step shown in FIG.

  The filtering unit 1 receives a filter obtained by matching a desired sound field and a synthesized sound field in consideration of a spherical speaker array arranged in a spherical shape.

  This filter is a filter defined by Expression (E) or Expression (F). This filter is generated in advance by the filter generation unit 2 prior to the processing of the filtering unit 1.

  In addition, an observation signal from a circular microphone array arranged in a spherical shape is input to the filtering unit 1.

  The filtering unit 1 obtains a signal by performing space-time convolution of the input filter and the input observation signal. The obtained signal is output to the signal generator 3.

  The signal generation unit 3 transmits a plurality of head signals based on the time signal corresponding to each speaker position of the spherical speaker array used to obtain the HRTF of the signal obtained by the filtering unit 1 and the reproduced sound from the spherical speaker array. A binaural signal in the time domain is generated by adding up the time signal corresponding to each speaker position of the spherical speaker array of the function in the time direction and adding the result by the number of speakers constituting the spherical speaker array.

  The processing of the filtering unit 1 is only an example. The filtering unit 1 may generate a binaural signal by the other filter processing or convolution processing described in the section <Technical background>.

That is, for example, the filtering unit 1 may generate a binaural signal by performing the process defined by Equation (9) using ~ P ^rcv (R _m , m) as the spherical harmonic spectrum of the observed signal.

  Thus, by considering a plurality of head related transfer functions, a highly accurate binaural signal can be generated.

[function]
Each function used in the above description will be described.

N is the first real Hankel function H _n ⁽¹⁾ (•) and n-order Bessel function J _n (•) are defined as follows. Γ (z) is a gamma function, and Y _n (z) is a Neumann function.

The nth-order first-type sphere Hankel function h ⁽¹⁾ _n (•) and the nth-order sphere Bessel function j _n (•) are defined as follows.

Legendre 陪 function P ^m _n (•), defined as follows: P _n (·) represents a Legendre polynomial.

The spherical harmonic function Y _n ^m is defined by the following equation.

[Modification]
The filtering process in the temporal frequency domain or the spatiotemporal frequency domain may be performed in the spatiotemporal domain. In other words, the filtering process may be performed by filtering using a temporal convolution and a spatial convolution.

  The binaural signal generation device can be realized by a computer. In this case, the processing content of each part of this apparatus is described by a program. Then, by executing this program on a computer, each unit in this apparatus is realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. In this embodiment, these apparatuses are configured by executing a predetermined program on a computer. However, at least a part of these processing contents may be realized by hardware.

  The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit of the present invention.

1 Filtering Unit 2 Filter Generation Unit 3 Signal Generation Unit

Claims (6)

A filter obtained by matching a desired sound field and a synthesized sound field in consideration of a plurality of head related transfer functions based on reproduced sound from a circular speaker array arranged in a circle, and a circle arranged in a circle Including a filtering unit that generates a binaural signal by convolving the observation signals from the cylindrical microphone array in a spatio-temporal direction,
j is an imaginary number, k is a wave number, J _m is an m-order Bessel function, H _m ⁽¹⁾ 'is a derivative of the m-th order first-class Hankel function H _m ⁽¹⁾ , and one of the listeners The position of the ear in the cylindrical coordinate system is (R _L , φ _L , 0), ~ G ^L is the head related transfer function expressed in the cylindrical harmonic function area, and ~ G is the transfer expressed in the cylindrical harmonic function area of each speaker _Where ~ G is measured on the circumference of radius R _ref , R _m is the radius of the circular microphone array, R _s is the radius of the circular speaker array,
The filter is a filter obtained by inverse Fourier transform of F ^L which is defined by the following formula (A) or Formula (B),

Binaural signal generator. The binaural signal generation device according to claim 1,
~ P ^rcv (R _m , m) as the cylindrical harmonic spectrum of the observed signal
When the listener changes the direction by φ _rot , the filtering unit convolves ~ P ^{r ot} (R _m , m) defined by the following equation (C) with the filter instead of the observed signal. ,

If the above listener moves to the position (R _t, φ _t), said filtering unit is defined by the formula (D) below in place of the observed signal _{^{- P trans (R m, m}} ) and the Convolve with filter,

Binaural signal generator. A signal is obtained by convolving the filter obtained by matching the desired sound field with the synthesized sound field in consideration of the spherical speaker array arranged in a spherical shape and the observation signal from the spherical microphone array arranged in a spherical shape. A filtering unit;
A time signal corresponding to each speaker position of the spherical speaker array of the obtained signal and a time corresponding to each speaker position of the spherical speaker array of a plurality of head-related transfer functions based on reproduced sound from the spherical speaker array A signal generation unit that generates a binaural signal in the time domain by adding the convolution of the signal in the time direction by the number of speakers constituting the spherical speaker array;
Including
j is an imaginary number, k is a wave number, n and m are the orders of the spherical harmonic function Y _n ^m , and the position r _s of the spherical speaker array in the spherical coordinate system is represented by r _s = (R _s , θ _s , φ _s ) , H _n ⁽¹⁾ 'is the derivative of the nth- order first-class spherical Hankel function h _n ⁽¹⁾ , ~ G is the transfer function expressed in the cylindrical harmonic function region of the speaker constituting the spherical speaker array, and ~ G is measured on the circumference of radius R _ref , R _m is the radius of the spherical microphone array, R _s is the radius of the spherical speaker array,
The filter is F ^L which is defined by the following formula (E) or Formula (F),

Binaural signal generator. A filter obtained by matching a desired sound field and a synthesized sound field in consideration of a plurality of head-related transfer functions based on reproduced sound from circular speaker arrays arranged in a circular shape, and a circular shape; It includes a filtering step that generates a binaural signal by convolving the observation signals from the arranged circular microphone array in the spatio-temporal direction, where j is an imaginary number, k is a wave number, J _m is an m-th order Bessel function, and H _m and ⁽¹⁾ 'the derivative of m-th order first kind Hankel function H _m ^(1), and the position in the cylindrical coordinate system of one of the ears of the listener and _{(R L, φ L, 0} ), ~ G Let ^L be the head-related transfer function expressed in the cylindrical harmonic function area, ~ G be the transfer function expressed in the cylindrical harmonic function area of each speaker, ~ G be measured on the circumference of the radius R _ref , R _m was the radius of the circular microphone array, the circular Supikaa the R _s As the radius of the over,
The filter is a filter obtained by inverse Fourier transform of F ^L which is defined by the following formula (A) or Formula (B),

Binaural signal generation method. Filtering space-time convolves the filter obtained by matching the desired sound field with the synthesized sound field in consideration of the spherical speaker array arranged in a spherical shape and the observation signal from the spherical microphone array arranged in a spherical shape. A filtering step to obtain a signal by
The signal generation unit includes the spherical signals of a plurality of head-related transfer functions based on time signals corresponding to the respective speaker positions of the spherical speaker array of the obtained signals and reproduced sounds from the spherical speaker arrays arranged in the spherical shape. A signal generation step of generating a binaural signal in the time domain by adding the time signal corresponding to each speaker position of the speaker array in the time direction and adding the amount corresponding to the number of speakers constituting the spherical speaker array;
Where j is an imaginary number, k is a wave number, n, m is the order of the spherical harmonic function Y _n ^m , and the position r _s of the spherical speaker array in the spherical coordinate system is expressed as r _s = (R _s , θ _s , φ _s ), h _n ⁽¹⁾ 'is the derivative of the nth- order first-class spherical Hankel function h _n ⁽¹⁾ , and ~ G is the transfer function expressed in the cylindrical harmonic function region of the speaker constituting the spherical speaker array And ~ G is measured on the circumference of radius R _ref , R _m is the radius of the spherical microphone array, and R _s is the radius of the spherical speaker array,
The filter is F ^L which is defined by the following formula (E) or Formula (F),

Binaural signal generation method.   A program for causing a computer to function as each unit of the binaural signal generation device according to any one of claims 1 to 3.

Images