JP6569945B2 - Binaural sound generator, microphone array, binaural sound generation method, program - Google Patents

Description

  The present invention relates to a binaural sound generation technique, and more particularly to a technique for generating binaural sound from a signal collected using a microphone array having a predetermined three-dimensional shape.

  In recent years, with the widespread use of omnidirectional cameras, extensive research has been conducted to virtually generate sound corresponding to images overlooked by users. One of them is an omnidirectional video / audio viewing system (Non-Patent Document 1). An omnidirectional image is an image taken with an omnidirectional camera. As a result, the user can view the video as if it were in the shooting location.

  In the omnidirectional video / audio viewing system, the user convolves a HRTF (Head-Related Transfer Function) with a local sound source signal group estimated in a plurality of regions (specifically, regions divided by specific angle widths). It can generate and output binaural sound corresponding to the overlooked video. In this system, the user acquires the head direction in real time by wearing an HMD (Head Mounted Display) with a gyro sensor. And the binaural sound corresponding to the image | video which the user overlooks is produced | generated in real time by switching HRTF convolved with each local sound source signal according to the acquired head direction. The generated binaural sound is heard using earphones or headphones.

  The HMD may be as simple as a combination of one Fresnel lens and a smartphone. By using a smart phone, content distributed over the network can be easily viewed.

  Hereinafter, sound generation in the omnidirectional video / audio viewing system (a binaural sound generation system corresponding to the omnidirectional video) will be described.

Assume that an array composed of M (M is an integer of 1 or more) microphones is installed and observed in a sound field in which K (K is an integer of 1 or more) sound sources exist. The k-th (1 ≦ k ≦ K) sound source signal is _represented by S _{k, ω, τ} , the m-th (1 ≦ m ≦ M) observation signal is represented by X _{m, ω, τ} , and the transfer characteristic therebetween is represented by A _{m, k, When ω} , the observation signal group x _{ω, τ} is modeled by the following equation.

Here, ω and τ represent a frequency index and a frame time index, respectively. Also,

T represents transposition, and N _{m, ω, τ} represents background noise included in the m-th observed signal.

The generation of binaural sound b _{ω, τ} = [B _{ω, τ} ^(Left) , B _{ω, τ} ^(Right) ] ^T corresponding to the video that the user is looking over will be described. The head direction (polar coordinate expression) of the user at the frame time τ is expressed as Ψ _τ = [Ψ _τ ^(Hor) , Ψ _τ ^(Ver) ] ^T. Assuming that the directivity of the sound source and background noise can be ignored, by convolving the HRTF between the direction of the user's head and each sound source into each sound source signal, the binaural sound b _{ω, τ} can be output. This is shown in FIG.

Here, H _{k, Ψτ, ω} ^(Left) and H _{k, Ψτ, ω} ^(Right) are the HRTF between the kth sound source and the user's left ear, and the HRTF between the kth sound source and the user's right ear. Represent each.

Considering that the HRTF does not change dramatically with the difference in the position of adjacent sound sources, even if the sound source group in the local region is regarded as one sound source signal (hereinafter referred to as a local sound source signal), the user's It is considered that the sound image localization is not greatly affected. Therefore, in the omnidirectional video / audio viewing system, the direction Θ _j = [Θ _j ^(Hor) , Θ _j ^(Ver) ] ^T (j = 1,..., L) is not extracted from individual sound source signals. A method of collecting sounds by direction for estimating a local sound source signal group in a group of L areas having an angular width as a main axis (hereinafter also referred to as a local area Θ _j for simplicity) is adopted. This is shown in FIG. For example, the local sound source signal Z _{Θ3, ω, τ} in FIG. 2 corresponds to the third sound source signal S _{3, ω, τ} in FIG. 1 and the fourth sound source signal S _{4, ω, τ.} Yes. A specific method of collecting sound by direction will be described later.

Direction Θ _j = [Θ _j ^(Hor) , Θ _j ^(Ver) ] A region having an angular width with ^T as the main axis and a sound source group coming from other regions are separated, and a local sound source signal Z _{Θj, ω, τ} (j = 1,..., L), the binaural sound b _{ω, τ} corresponding to the video overlooked by the user is virtually generated by the following equation.

Here, H _{Θj, Ψτ, ω} ^(Left) and H _{Θj, Ψτ, ω} ^(Right) are the HRTF between the principal axis direction of the jth region and the left ear of the user, the principal axis direction of the jth region and the user's HRTF between right ears is represented respectively. It is generally known that the HRTF changes depending on the reverberation time of the sound field, the personality of the physical structure of the head and both ears, and the distance between the sound source and the listener. Assuming that the influence of can be ignored, H _{Θj, Ψτ, ω} ^(Left) and H _{Θj, Ψτ, ω} ^(Right) are simplified. This simplified H _{Θj, Ψτ, ω} ^(Left) and H _{Θj, Ψτ, ω} ^(Right) are recorded in advance by installing HATS (Head and Torso Simulators) under low reverberation and discretely arranging speakers. It is obtained by selecting the HRTF in the direction closest to the selected database.

Excitation signal group s _omega, binaural sound from the _tau b _omega, the overall processing flow for generating the _tau shown in Fig. The re-synthesis process in FIG. 3 corresponds to the generation of binaural sound using Expressions (9) and (10). At that time, the head direction of the user acquired by the HMD is input (the user control in FIG. 3 corresponds).

Next, the sound collection by direction for collecting the local sound source signal group z _{ω, τ} = [Z _{Θ1, ω, τ} ,..., Z _{ΘL, ω, τ} ] ^T from the observed signal group x _{ω, τ} will be described. The omnidirectional video / audio viewing system uses direction-specific sound collection by a sound source enhancement method based on local PSD (Power Spectral Density) estimation.

  Here, the reason for using sound collection by direction instead of sound collection by sound source in the omnidirectional video and audio viewing system will be described. In an application where a signal group separated so as to correspond to a video that the user looks around is localized and re-synthesized, it is considered unnecessary to forcibly separate sound source groups at close positions. This is because the characteristics of the HRTF between the sound source group and the listener do not change greatly, and thus the sound image localization of the listener is not greatly affected. Rather, if it is assumed that the sound source moves from moment to moment, it is preferable that a local sound source signal group corresponding to a group of regions divided as uniformly as possible can be generated.

Pre-enhance sound coming from the region with the direction Θ _j as the main axis by applying beam forming to the observation signal group x _{ω, τ} or receiving sound using a super-directional microphone such as a shotgun microphone The signal obtained is _defined as Y _{Θj, ω, τ} (j = 1,..., L). Also, the pre-enhancement signal group _{y ω, τ = [Y Θ1} , ω, τ, ..., Y ΘL, ω, τ] denoted ^T. The process of generating the pre-enhanced signal group _{yω, τ} is the directivity forming process in FIG.

When the sound source signal is assumed to be mutually _{uncorrelated,} Y Θj, ω, PSDφ YΘj of _{tau, omega} is modeled by the following equation.

Here, <•> represents the expected value calculation, D _{Θj, k, ω} represents the average sensitivity of the jth beamforming / received sound for the kth sound source _, and φ _{Sk, ω} represents the PSD of the kth sound source. .

Assuming that the relationship of Equation (11) also holds for the relationship between the local sound source signal group z _{ω, τ} and the pre-enhanced signal group y _{ω, τ} , φ _{YΘj, ω} is expressed by the following equation.

_Here, D Θj, Θi, average sensitivity of the j-th beam forming / sound receiving the _omega is the direction theta _i for regions with the _spindle, φ SΘi, _ω is the i-th local source signal PSD (local PSD) Represents. The relationship between L _{φSΘi, ω} and _{φYΘj, ω} is modeled by the following equation.

In order to estimate L local PSDφ _{SΘi, ω} , the inverse problem of equation (13) is solved. Here, if the local PSD is estimated for each frame in order to enhance the noise suppression performance, the inverse problem is formulated by the following equation.

As a practical problem, there arises a problem of controlling the stability of the number of local regions L and D _ω ⁻¹ where sparseness can be assumed. Since all elements of D _omega is the number of positive, depending on the conditions of the singular values of D _omega sometimes not obtained is stable solutions. Therefore, it is necessary to adjust the stabilization calculation manually. For example, an operation of adding a predetermined value to the diagonal term may be performed and adjusted as follows.

Here, ε is a stabilization coefficient, and a larger value enables more stable inverse matrix calculation.

When only the interference noise is mixed in the observation signal, the PSD of the target sound and the PSD of the noise may be obtained from Φ ^ _{S, ω, τ} calculated by the equation (14). Note that the PSD of the target sound and the PSD of the noise are necessary when generating a sound source enhancement filter.

  However, in actuality, incoherent (or diffusive) background noise exists in the observed signal as shown in Equation (1). In such a case, it is considered that a more accurate sound source enhancement filter can be generated by separately estimating the PSD of coherent noise and the PSD of background noise. One method for separately estimating the PSD of the coherent noise and the PSD of the background noise will be described below.

First, the PSD of background noise is removed from Φ ^ _{S, ω, τ} calculated by Expression (14). Since the background noise can be assumed to be uncorrelated with the target sound and the interference noise, it can be considered that the background noise can be approximated even if the addability in the power spectrum region is assumed. A sound source group in a local region in the i-th direction Θ _i is set as a target sound. At that time, local _{PSDφ SΘi, ω,} the background noise of _PSDφ BNTΘi present from _τ in _{it, ω,} is subtracted _τ. As a result, the PSD of the estimated target sound (having removed the influence of background noise) φ _{TSΘi, ω, τ} is obtained.

If the target sound _{PSDφTSΘi, ω, τ} is smaller than 0, it is set to 0. _Further, in order to calculate the background noise PSDφ _{BNTΘi, ω, τ} of the equation (16), it is assumed that the background noise has a strong temporal _steadiness (that is, it does not change dramatically according to the time), and is recursive. When the sudden component is removed by _{performing a} time smoothing process on _{φSΘi, ω, τ} by a typical update algorithm, Expression (17) is obtained.

Here, _βω is a constant for time smoothing. For example, it may be set to forget about 150 ms. _φ ^- _{SΘi, ω,} by holding the minimum value in the interval Τ of _tau, can be estimated _PSDφ BNTΘi of background noise (local area of i-th direction theta _i) target sound _{region, omega,} and _tau.

Similarly, the interference noise group _PSDφ ISΘi in the other region (local regions of the i-th direction theta _i) target sound _{region, ω,} PSDφ _BNIΘi similarly background noise and target sound to estimate the _{tau, omega , τ} is subtracted.

Here, α _{1 and ω} are weighting factors whose optimum values change according to the content. Also, PSDφ _{ISΘi, ω, τ} of the coherent noise group is floored to 0 when it is smaller than 0. The background noise PSDφ _{BNIΘi, ω, τ} in the equation (19) is calculated as follows.

A Wiener filter _{GΘj, ω, τ} for estimating the jth local sound source signal _{ZΘj, ω, τ} is generated.

Here, α _{2, ω} and α _{3, ω} are weighting factors.

The Wiener filter G _{Θj, ω, τ} after calculation using the equation (22) is shaped as follows.

Here, α _{4, ω} is a weighting coefficient. _{Thereafter, α 5, ω (0 ≦} α 5, ω <1) with _{a, α 5, ω ≦ G Θj} , ω, τ ≦ 1 become as G _{.theta.j, omega,} performs flooring processing _tau . The local sound source signal Z _{Θj, ω, τ} is calculated by the following equation.

The process of generating the local sound source signal group z _{ω, τ} by performing Wiener filtering on the pre-enhanced signal group y _{ω, τ} is the direction-specific sound collection process in FIG.

  Finally, a binaural sound generation system 900 that executes binaural sound generation processing in the omnidirectional video / audio viewing system will be described. FIG. 4 is a block diagram showing the configuration of the binaural sound generation system 900. As shown in FIG. 4, the binaural sound generation system 900 includes a sound collection device 905 and a resynthesis device 955. The sound collection device 905 includes M microphones 910-1 to 910 -M, M frequency domain conversion units 920-1 to 920 -M, L beam forming units 930-1 to 930 -L, A local PSD estimation unit 940 and a Wiener filtering unit 950 are included. The resynthesis device 955 includes an HRTF convolution unit 960.

The sound collection device 905 executes processing (sound source separation processing) for generating a local sound source signal group from the time domain observation signal group. Microphones 910-1 to 910 -M collect sound in a sound field where K sound sources are present, and generate time-domain observation signals. The frequency domain conversion units 920-1 to 920 -M convert the time domain observation signals to observation signals X _{m, ω, τ} (1 ≦ m ≦ M), respectively.

Beam forming sections 930-1 to 930 -L generate pre-enhanced signals Y _{Θj, ω, τ} (j = 1,..., L) from M observation signals (observation signal group). Instead of the microphones 910-1 to 910 -M, L = M may be used and sound may be collected using L directional microphones. In this case, since the signal Y _{Θj, ω, τ} (j = 1,..., L) _obtained by pre- _enhancing the signal collected using the directional microphone may be used, the beam forming units 930-1 to 930-L are unnecessary. Become.

The local PSD estimation unit 940 generates a target sound PSD, interference noise PSD, and background noise PSD using the pre-enhanced signal Y _{Θj, ω, τ} (j = 1,..., L). Specifically, the target sound PSD, the interference noise PSD, and the background noise PSD are generated using the equations (14), (16), (19), and (18).

The Wiener filtering unit 950 generates L Wiener filters using the target sound PSD, the interference noise PSD, and the background noise PSD, and pre-enhanced signals Y _{Θj, ω, τ} (j = 1,..., L ) To apply the Wiener filter G _{Θj, ω , τ} (j = 1,..., L) to generate the local sound source signal Z _{Θj, ω, τ} (j = 1,..., L). Specifically, the local sound source signal Z _{Θj, ω, τ} is generated using Expression (22), Expression (23), and Expression (24).

The re-synthesis device 955 executes a process (re-synthesis process) for generating a binaural sound from the local sound source signal group. The HRTF convolution unit 960 generates a binaural sound b _{ω, τ} from the local sound source signal Z _{Θj, ω, τ} (j = 1,..., L). Specifically, the listening signal (left) and the listening signal (right), which are binaural signals for listening, are generated using Equations (9) and (10).

  Note that the binaural sound generation system 900 can be configured by connecting the sound collection device 905 and the re-synthesis device 955 to a network such as the Internet. In this case, it goes without saying that the sound collection device 905 and the re-synthesis device 955 need to have means necessary for communication via the network. In addition, the sound collection device 905 and the resynthesis device 955 are provided with an encoding unit that encodes a local excitation signal group and a decoding unit that decodes encoded data obtained by encoding the local excitation signal group, respectively, so as to be suitable for transmission. May be.

  In the omnidirectional video / audio viewing system, binaural sound is generated after sound source separation processing, so there is no particular limitation on the arrangement of microphones for generating observation signals and the shape of the microphone array. On the other hand, research on binaural recording that directly obtains binaural sounds from observation signals by arranging microphones that generate observation signals in a microphone array of a special shape and collecting sound is also underway. Usually, in binaural recording, recording is performed using a microphone with an auricle such as HATS or a dummy head. On the other hand, Non-Patent Document 2 proposes a method for simply recording binaural sounds for images captured in a fixed direction without elaborately modeling the pinna. In Non-Patent Document 2, frequency-space characteristics that can serve as a clue to localize sound using a simple configuration in which a spherical microphone array is provided with a hemispherical depression and a microphone is installed there. It is confirmed that the pattern can be obtained.

Kenta Niwa, Kuruma Koizumi, Kazunori Kobayashi, Takashi Uematsu, “Study on sound collection according to direction to generate binaural sound corresponding to omnidirectional video”, IEICE Technical Report EA2015-7, IEICE, 2015 July, vol.115, no.126, pp.33-38. Nakagiri Taishi, Yamamura Toshiki, Nishino Takanori, Naruse Osamu, Takeda Kazuya, “Study of Binaural Recording Using Spherical Microphone Baffle with Indentation”, Acoustical Society of Japan 2015 Spring Meeting 2-P-42, March 2015, pp.889-890.

  In the binaural sound generation system 900, since the optimum values of the array signal processing parameters (parameters of the sound collection device 905) necessary for adjusting the noise suppression amount and the like are different for each content, it is necessary to perform parameter adjustment work. was there. On the other hand, instead of adjusting to the optimum parameter for each content, it may be possible to adjust to a parameter that can be used universally for various contents, but there is content that deteriorates the sound collection performance by doing so. There were problems such as.

  Accordingly, an object of the present invention is to provide a binaural sound generation device that generates a binaural sound from an observation signal, which does not require adjustment of array signal processing parameters.

  One aspect of the present invention is a binaural sound generation device that generates a binaural sound from an observation signal picked up using a microphone array including M depressions (M is an integer of 3 or more) where microphones are installed, and n , K is an integer satisfying M = 2n + k (n ≧ 1, k = 0 or 1), and the shape of the three-dimensional shape of the microphone array viewed from above is a symmetrical figure, and the M depressions are , Provided on the side of the three-dimensional shape, of which 2n dents are provided so as to be paired at intervals of 180 degrees when the three-dimensional shape is viewed from above, and at least one microphone is provided in the dent. And an interpolating and synthesizing unit that generates the binaural sound by interpolating and synthesizing the observation signals.

  According to the present invention, sound collection is performed using a microphone array in which microphones are installed in depressions provided at positions that are paired at intervals of 180 degrees when viewed from above into a three-dimensional shape having a symmetrical shape when viewed from above. By generating a binaural sound from the observed signal by interpolation synthesis, it becomes possible to generate a binaural sound from the observed signal without adjusting the parameters of the array signal processing.

The figure which shows the image of the production | generation of the binaural sound according to the head direction using the sound-collection according to sound source. The figure which shows the image of the production | generation of the binaural sound according to the head direction using the sound collection according to direction. The figure which shows the production | generation processing flow of the binaural sound in a omnidirectional video-audio viewing system. 1 is a block diagram showing a configuration of a binaural sound generation system 900. FIG. The block diagram which shows the structure of the binaural sound production | generation apparatus 400. FIG. 5 is a flowchart showing the operation of the binaural sound generating apparatus 400. The figure which shows an example of the three-dimensional shape of the microphone array 410. FIG. The figure which shows an example of the three-dimensional shape of the microphone array 410. FIG. The figure which shows a mode that the omnidirectional video production | generation camera was incorporated in the microphone array 410. FIG. The figure which shows an example of the installation position of the microphone of the microphone array. The figure which shows the mode of selection of the microphone in a horizontal surface. The figure which shows the graph of the weighting coefficient of each microphone in a horizontal surface. The figure which shows the graph of the weight of a horizontal direction and an elevation angle direction.

  Hereinafter, embodiments of the present invention will be described in detail. In addition, the same number is attached | subjected to the structure part which has the same function, and duplication description is abbreviate | omitted.

  Non-Patent Document 2 shows that instead of an elaborate pinna model, the sound reception signal contains information useful for sound source localization simply by installing a microphone in a hemispherical depression provided in a spherical microphone array. Experimentally shown. Therefore, a simple method of installing a microphone in a hollow provided in this simple three-dimensional shape is extended to omnidirectional sound collection, and a binaural sound corresponding to the omnidirectional image is virtually generated from the sound collection signal Will be described.

  Hereinafter, the binaural sound generation device 400 will be described with reference to FIGS. FIG. 5 is a block diagram illustrating a configuration of the binaural sound generation device 400. FIG. 6 is a flowchart showing the operation of the binaural sound generator 400. As shown in FIG. 5, the binaural sound generation device 400 includes a microphone array 410 and an interpolation / synthesis unit 420. The microphone array 410 has a predetermined three-dimensional shape with a recess at a predetermined position, and M microphones 910-1 to 910 -M are installed in the recess.

  The microphone array 410 picks up sound in a sound field in which K sound sources are present, and generates M time domain observation signals (S410). An example of the three-dimensional shape of the microphone array 410 is a sphere or a cylinder. Moreover, the shape close | similar to a sphere and a cylinder may be sufficient instead of a exact sphere and a cylinder. Considering that the head is swung from side to side when viewing the omnidirectional image, the shape of the three-dimensional shape seen from above is typically a point-symmetric figure such as a circle, or a figure with symmetry. .

  Moreover, a hollow is comprised by denting the side surface of these solid shapes at intervals of 90 degrees. This depression is a simple model of the auricle. The shape of the recess may be a simple shape such as a hemisphere. Further, the depressions are not limited to 90 degree intervals. Considering that the arrangement of human ears is symmetric when viewed from above, if left and right microphone pairs (pairs spaced 180 degrees) can be placed symmetrically on the sides, for example, 60 degrees and 30 degrees The depressions may be formed at any angle such as an angular interval (generally, n is an integer of 2 or more and an interval of 180 / n degrees). Note that the sound collection performance is improved when the interval is narrower than 90 degrees. In addition, if the microphones can be installed symmetrically as a left and right pair, the intervals at which the recesses are installed do not have to be exactly uniform intervals such as 180 / n degrees.

  Considering the motion of shaking the head left and right and the arrangement of human ears, the best sound collection performance is achieved by placing the left and right pairs of microphones symmetrically in a three-dimensional shape symmetrical from above. If one microphone is installed in each recess, four, six, and twelve microphones are installed at 90 degrees, 60 degrees, and 30 degrees, respectively. However, since a virtual binaural sound for listening can be generated by interpolation synthesis, it is not always necessary to install two microphones that are paired left and right symmetrically. For example, a configuration may be adopted in which one microphone is installed for each of three depressions among four depressions provided at intervals of 90 degrees. Further, the remaining one does not actually have to be a depression.

  In consideration of the front and back face orientation (ear orientation), two microphones may be installed in each recess (see FIG. 7C). By installing in this way, binaural sound with higher sound quality can be re-synthesized.

  Furthermore, considering the operation of shaking the neck up and down, it is preferable to install microphones on the upper and lower surfaces of the three-dimensional shape. In addition, when a microphone is installed on the upper surface or the lower surface, no depression is necessary. This is because it is not necessary to model the pinna. In this way, even by installing a microphone in the elevation direction, binaural sound with higher sound quality can be re-synthesized.

  An example of a three-dimensional shape having such depressions is shown in FIGS. FIGS. 7A and 8A are views of the three-dimensional shape of the microphone array as viewed from above and below. FIGS. 7B and 8B are views of the three-dimensional shape of the microphone array as viewed from the front (back) and the side. FIGS. 7C and 8C are views showing the shape of the depression and the sound receiving position where the microphone is installed. In FIGS. 7A to 7C and FIGS. 8A to 8C, a broken semicircle or a solid circle indicates a depression, and a small black dot indicates a sound receiving position. The sound receiving position may be shifted 30 degrees forward and backward, left and right, etc. on a horizontal plane as shown in the left diagram of FIG. FIG. 8D shows how the three-dimensional top and bottom shapes shown in FIGS. 8A and 8B are generated. The solid line portions in FIG. 8D are the shapes of the three-dimensional upper and lower surfaces.

  As can be seen from FIGS. 7 and 8, it can be seen that the figure is symmetrical in the vertical and horizontal directions when viewed from above and from the side. It can also be seen that the three-dimensional shape of the microphone array is configured by combining cylinders and spheres. The solid shapes of FIGS. 7 and 8 are based on a cylinder with a diameter of 12 cm, but considering that this solid shape simulates the head shape, it is better to be close to a sphere with a diameter of about 16 cm. Note that the upper limit of the width of the three-dimensional shape as viewed from above (corresponding to the diameter in the case of a circle) is about 25 cm in consideration of transmission delay. The lower limit is about 5 cm in consideration of downsizing the microphone array. That is, the width is preferably 5 cm or more and 25 cm or less.

  As can be seen from FIGS. 7C and 8C, two microphones are installed in each recess. This is because a binaural sound is generated according to the direction of the front and back faces as described above.

  Moreover, in order to provide not only horizontal localization but also elevation angle localization, microphones may be installed on the upper and lower surfaces of a three-dimensional shape. Only one microphone may be installed on the upper surface or the lower surface. Of course, one microphone may be provided on each of the upper and lower surfaces. 7 and 8, three microphones are installed on the upper and lower surfaces. Since it is not necessary to simulate the auricle, when the microphones are installed on the upper surface and the lower surface, the depression is not necessary as described above.

  Note that the omnidirectional video generation camera may be built in the microphone array 410 as shown in FIG. 9, for example, or may be installed at a location different from the microphone array 410.

The interpolating / combining unit 420 interpolates and synthesizes M time domain observation signals x _{m, t} (1 ≦ m ≦ M), and binaural sounds b _t ^(Left) and b _t in the time domain, which are virtual signals for listening. ^(Right) is generated (S420). Specifically, interpolation synthesis is performed using Expression (25) and Expression (26).

Here, w _{m, Ψτ} ^(Left) and w _{m, Ψτ} ^(Right) are weighting factors that vary depending on the head direction Ψτ and the microphone index m.

  Hereinafter, the design of the weighting coefficient used for the interpolation will be described with reference to FIGS. Here, a description will be given using a microphone array 410 having four depressions in the horizontal direction every 90 degrees and one microphone on each of the upper and lower surfaces. The microphone array 410 receives sound using a total of ten microphones.

  FIG. 10 is a diagram showing a microphone installation position (sound receiving position) using a microphone index m (hereinafter referred to as Mic (m)). As shown in FIG. 10, the direction (Azimuth = 0 °) of the microphone array is the direction in which the depressions with Mic (1) and Mic (2) (thick semicircles in the figure) are present. Further, a horizontal plane passing through the center of the depression is defined as a reference horizontal plane (Elevation = 0 °). It is assumed that each one microphone installed on the upper surface and the lower surface is installed in Mic (10) and Mic (13). That is, no microphone is installed in Mic (9), Mic (11), Mic (12), and Mic (14).

  FIG. 11 shows how a microphone is selected on a horizontal plane. The direction in which the eyes are facing (the direction indicated by the arrow) corresponds to the head direction of the user. Note that the positions of Mic (1) to Mic (8) are not changed. For example, in the leftmost diagram in the upper stage, the eyes are oriented in the direction of 0 °, and at this time, observation is performed using Mic (4) and Mic (7).

  FIG. 12 shows a graph of the weight coefficient of each microphone on the horizontal plane (lateral movement). According to the graph, the weight coefficient of each microphone is set. For example, when viewing the graph of Mic (1) (upper left graph), the weighting factor (Weight of the graph) monotonically increases from 0 to 1 at -180 ° to -90 °, and the weighting factor at -90 ° to 0 °. Is monotonically decreased from 1 to 0, and the weighting coefficient is set to be zero between 0 ° and 180 °.

  FIG. 13 shows a graph of weighting coefficients (weights used for weighted addition of signals received by the upper and lower microphones) in the horizontal direction and the elevation angle direction. Each weighting factor is set according to the graph as in the horizontal direction. When the three graphs are combined, the sum of the weighting coefficient in the horizontal direction (Horizontal Signal) and the weighting coefficient in the elevation direction (Vertical Signal) of Mic (10) and Mic (13) becomes 1 at -90 ° to 90 °. It turns out that it is set up to become.

  Note that the weighting factor in the horizontal direction (FIG. 12) and the weighting factor in the elevation direction (FIG. 13) for providing a sense of orientation in the vertical direction need not be designed strictly. It is only necessary to set the weighting coefficient so as to have a corresponding relationship according to the head direction.

  A method of performing interpolation synthesis using a horizontal weighting factor and an elevation weighting factor will be described. First, signals received by eight microphones at the horizontal angle in the head direction are synthesized according to a preset weighting factor. Next, the final synthesized binaural sound is obtained by weighting and adding the signal synthesized earlier according to the elevation angle in the head direction and the signal received by the two microphones in the vertical direction.

Since the weights w _{m, Ψτ} ^(Left) and w _{m, Ψτ} ^(Right) for the m-th microphone when the head direction is Ψτ can be calculated as described above, a virtual sound source separation process is not performed. Binaural sounds can be generated.

  In the present embodiment, an M channel signal is observed using the microphone array 410 and mixed down (interpolation synthesis) by the interpolation synthesis unit 420 to generate a virtual binaural sound. This eliminates the need for adjustment of the latent parameter group, which is present in the sound source enhancement method employed in the binaural sound generation system 900 and that changes the optimum value for the type of sound to be recorded.

<Supplementary note>
The apparatus of the present invention includes, for example, a single hardware entity as an input unit to which a keyboard or the like can be connected, an output unit to which a liquid crystal display or the like can be connected, and a communication device (for example, a communication cable) capable of communicating outside the hardware entity. Can be connected to a communication unit, a CPU (Central Processing Unit, may include a cache memory or a register), a RAM or ROM that is a memory, an external storage device that is a hard disk, and an input unit, an output unit, or a communication unit thereof , A CPU, a RAM, a ROM, and a bus connected so that data can be exchanged between the external storage devices. If necessary, the hardware entity may be provided with a device (drive) that can read and write a recording medium such as a CD-ROM. A physical entity having such hardware resources includes a general-purpose computer.

  The external storage device of the hardware entity stores a program necessary for realizing the above functions and data necessary for processing the program (not limited to the external storage device, for example, reading a program) It may be stored in a ROM that is a dedicated storage device). Data obtained by the processing of these programs is appropriately stored in a RAM or an external storage device.

  In the hardware entity, each program stored in an external storage device (or ROM or the like) and data necessary for processing each program are read into a memory as necessary, and are interpreted and executed by a CPU as appropriate. . As a result, the CPU realizes a predetermined function (respective component requirements expressed as the above-described unit, unit, etc.).

  The present invention is not limited to the above-described embodiment, and can be appropriately changed without departing from the spirit of the present invention. In addition, the processing described in the above embodiment may be executed not only in time series according to the order of description but also in parallel or individually as required by the processing capability of the apparatus that executes the processing. .

  As described above, when the processing functions in the hardware entity (the apparatus of the present invention) described in the above embodiments are realized by a computer, the processing contents of the functions that the hardware entity should have are described by a program. Then, by executing this program on a computer, the processing functions in the hardware entity are realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used. Specifically, for example, as a magnetic recording device, a hard disk device, a flexible disk, a magnetic tape or the like, and as an optical disk, a DVD (Digital Versatile Disc), a DVD-RAM (Random Access Memory), a CD-ROM (Compact Disc Read Only). Memory), CD-R (Recordable) / RW (ReWritable), etc., magneto-optical recording medium, MO (Magneto-Optical disc), etc., semiconductor memory, EEP-ROM (Electronically Erasable and Programmable-Read Only Memory), etc. Can be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing the process, the computer reads a program stored in its own recording medium and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from a portable recording medium and execute processing according to the program, and the program is transferred from the server computer to the computer. Each time, the processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program in this embodiment includes information that is used for processing by an electronic computer and that conforms to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer).

  In this embodiment, a hardware entity is configured by executing a predetermined program on a computer. However, at least a part of these processing contents may be realized by hardware.

Claims (8)

The binaural sound corresponding to the video that the user overlooks in the omnidirectional video / audio viewing system from the observation signal collected using the microphone array having M depressions (M is an integer of 3 or more) where the microphone is installed. A binaural sound generating device for generating,
n and k are integers satisfying M = 2n + k (n ≧ 1, k = 0 or 1),
The shape of the three-dimensional shape of the microphone array viewed from above is a figure having symmetry,
The M depressions are provided on the side surfaces of the three-dimensional shape, of which 2n depressions are provided so as to be paired at intervals of 180 degrees when the three-dimensional shape is viewed from above.
At least one microphone is installed in the depression,
When there is a dent corresponding to the ear in the user's head direction, a weighting factor greater than 0 is applied only to the observation signal collected by the microphone installed in the dent and the ear in the user's head direction. A binaural sound generating device including: an interpolating / synthesizing unit that generates the binaural sound by interpolating and synthesizing the observation signals by giving a weighting coefficient larger than 0 to all the observation signals when there is no corresponding depression . Microphone provided with M (M is an integer of 3 or more) depressions for installing microphones that collect observation signals used to generate binaural sounds corresponding to images overlooked by the user in the omnidirectional video / audio viewing system An array,
n and k are integers satisfying M = 2n + k (n ≧ 1, k = 0 or 1),
The shape of the three-dimensional shape of the microphone array viewed from above is a figure having symmetry,
The M depressions are provided on the side surfaces of the three-dimensional shape, of which 2n depressions are provided so as to be paired at intervals of 180 degrees when the three-dimensional shape is viewed from above.
At least one microphone is installed in the depression ,
The binaural sound has a weight coefficient greater than 0 for only the observation signal collected by the microphone installed in the depression when there is a depression corresponding to the ear in the head direction of the user. A microphone array generated by interpolating and synthesizing the observation signals by giving a weighting coefficient larger than 0 to all the observation signals when there is no depression corresponding to the ear in the direction .   The microphone array according to claim 2,
  x _{m, t} (1 ≦ m ≦ M) is the observed signal, b _t ^(Left) , B _t ^(Right) Is the binaural sound,
  The binaural sound is generated by interpolating with the following equation.
(However, w _{m, Ψτ} ^(Left) , W _{m, Ψτ} ^(Right) Is a weighting coefficient that varies depending on the user's head direction Ψτ and the microphone index m at the frame time τ)
  A microphone array characterized by that. The microphone array according to claim 3 ,
The microphone array, wherein the shape viewed from above is a circle. The microphone array according to claim 3 or 4 ,
K satisfies k = 0,
A microphone array, wherein two microphones are installed in each of the 2n depressions. A microphone array according to any one of claims 3 to 5 ,
A microphone array, wherein at least one microphone is provided on each of the upper surface or the lower surface or the upper and lower surfaces of the three-dimensional shape. The binaural sound corresponding to the video that the user overlooks in the omnidirectional video / audio viewing system from the observation signal collected using the microphone array having M depressions (M is an integer of 3 or more) where the microphone is installed. A binaural sound generating method in a binaural sound generating device for generating,
n and k are integers satisfying M = 2n + k (n ≧ 1, k = 0 or 1),
The shape of the three-dimensional shape of the microphone array viewed from above is a figure having symmetry,
The M depressions are provided on the side surfaces of the three-dimensional shape, of which 2n depressions are provided so as to be paired at intervals of 180 degrees when the three-dimensional shape is viewed from above.
At least one microphone is installed in the depression,
When there is a dent corresponding to the ear in the user's head direction, a weighting factor greater than 0 is applied only to the observation signal collected by the microphone installed in the dent and the ear in the user's head direction. An interpolating and synthesizing step of generating the binaural sound by interpolating and synthesizing the observed signal by giving a weighting coefficient greater than 0 to all the observed signals when there is no corresponding depression .   A program for causing a computer to function as the binaural sound generating device according to claim 1.