JPH07248255A - Method and apparatus for forming stereophonic image - Google Patents

Abstract

PURPOSE:To provide a stereophonic image forming apparatus and method which allows the elimination of time and labor for transferring a head transmission function by reducing interpolation calculation while enabling the following of the movement of the position of a sound image. CONSTITUTION:A sound data inputted undergoes a head transmission function processing with a plurality of head transmission function processors 13a, 13b... and 13n in parallel. The sound data subjected to the head transmission function processing is stored temporarily into a sound data memory 16. When the position of a sound image is detected with a sound image position detector 11, the sound data subjected to the head transmission function processing stored temporarily into the sound data memory 16 is interpolated in the sound image with a sound image interpolator 17 based on the results of the detection and is outputted as sound image output 100.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a stereoscopic sound image generating apparatus and a stereoscopic sound image generating method for generating a stereoscopic sound image used for virtual reality, that is, virtual reality.

[0002]

2. Description of the Related Art Generally, a sound from a sound source directly reaches the ear or a reflected sound reflected on a wall reaches both ears after being diffracted and reflected by the head and auricle. A person perceives a sound propagation direction because of a difference in sound pressure intensity between the ears and a difference in arrival time of the sound to the ears, that is, a phase difference. The sound pressure difference and the phase difference until the sound from the sound source reaches the eardrum are measured in advance for each frequency, and the head related transfer function, which is the transfer function thereof, is applied to the waveform generated by the sound source. It is possible to reproduce the waveform just before the eardrum.

In virtual reality, the sound image changes according to the movement of the sound source or the movement of the person. Since the absolute position of the sound source is usually given as a function of time, the relative position of the sound image is determined from the position and angle of the human head and the absolute position of the sound source.

A conventional stereoscopic sound image generating apparatus will be described below with reference to FIG.

The three-dimensional sound image generation device has a sound source input means 10 such as a microphone for inputting a sound from a sound source, and a sound image position detection device 11 for detecting the position of the sound image. A sound data storage device 12 for storing the generated sound data is connected. A plurality of head related transfer function processing devices 13a, 13b, ..., 13n for performing head related transfer function processing of input sound data are connected in parallel to the storage device 12. The head related transfer function processing device 13a is configured to perform head related transfer function processing on the direct wave, and the head related transfer function processing devices 13b, ..., 13n are configured to perform head related transfer function processing on the reflected wave, respectively. ing. A head-related transfer function interpolating device 14 for obtaining a head-related transfer function from position information of the detected sound image is connected to the sound image position detecting device 11, and the interpolating device 14 includes the processing devices 13a, 13b,
..., 13n. Processors 13a, 13
b, ..., 13n include processing devices 13a, 13b ,.
A voice synthesizing device 15 for synthesizing the 3n result and outputting a sound image output 100 is connected.

Next, the operation of the conventional example will be described.

The sound from the sound source is input by the sound source input means 10 and stored in the sound data recording device 12. The relative position of the sound image is detected by the sound image position detection device 11 from the position of the sound image and the position and angle of the person's head. From the position information of the detected sound image, the HRTF is HRTF interpolation device 1
4 is required. Since it is impossible to prepare the head related transfer functions in all directions, the head related transfer functions in the necessary directions are obtained by interpolation. Each processing device 13a, 13
.., 13n read the sound data from the sound data stored in the sound data recording device 12 for each of the direct waves and the reflected waves in consideration of the arrival time difference, and the head related transfer function is operated. The result is synthesized by the speech synthesizer 15,
It is output as a sound image output 100.

As described above, the sound image from the required direction is generated.

[0009]

It has been known that the time delay from the detection of the relative position of the sound image to the generation of a sound corresponding thereto makes the localization of the sound unstable, so this time delay needs to be made small. There is. The cause of this time delay is due to the interval of sound image position detection, the time required to interpolate the head related transfer function, and the time to transfer the interpolation to the head related transfer function processing device. Considering an implementation method of the head-related transfer function processing device, the head-related transfer function is defined in the frequency domain, and the sound from the sound source is Fourier-transformed to perform complex multiplication for each of the N frequency components, Inverse Fourier transform is performed. If the head related transfer function changes, theoretically,
It can be calculated in the order of og (N), but in reality, it is difficult to handle it when the head related transfer function changes.

As a method of realizing the head related transfer function processing device, a method of convoluting the impulse response obtained by inverse Fourier transforming the head related transfer function and the sound from the sound source is considered,
The calculation amount is on the order of N per sound. The convolution operation can be calculated in a predetermined time regardless of the length N by using an FIR filter or an IIR filter. However, when the coefficient of the convolution operation is changed, that is, when the head related transfer function is changed, it takes time of the order of N.

Even when the impulse response is approximated by a shorter FIR at the sacrifice of the localization and the head-related transfer function processing device is realized by the convolution operation of the impulse response and the sound from the sound source, the number of taps of the FIR is the head. It must be determined in consideration of the labor of calculation and transfer of transfer function interpolation.

An object of the present invention is to provide a stereoscopic sound image generating apparatus and a stereoscopic sound image generating method capable of reducing the calculation of interpolation, omitting the trouble of transferring the head related transfer function and smoothly following the movement of the sound image position. To aim.

[0013]

According to the present invention, the above-mentioned object is to perform head related transfer function processing on input sound data which is connected in parallel to the sound input means and the sound source input means. A plurality of head related transfer function processing means, a sound image position detecting means for detecting a position of a sound image, and sound image interpolation of sound data processed by the plurality of head related transfer function processing means based on a detection result by the sound image position detecting means. This is achieved by a stereoscopic sound image generating device including a sound image interpolating means.

It is preferable that the three-dimensional sound image generation device includes sound data storage means for temporarily storing the sound data processed by the plurality of head related transfer function processing means.

According to the present invention, the above-mentioned object is to perform head related transfer function processing in parallel on input sound data, detect the position of a sound image, and to detect the plurality of head related transfer functions based on the detection result. This is achieved by a stereoscopic sound image generation method in which processed sound data is sound image interpolated.

[0016]

According to the three-dimensional sound image generating apparatus of the present invention, the input sound data is subjected to head-related transfer function processing in parallel by a plurality of head-related transfer function processing means, and the position of the sound image is detected by the sound image position detecting means. The sound image subjected to the head related transfer function processing based on the detection result is subjected to the sound image interpolation by the sound image interpolating means, so that the trouble of transferring the head related transfer function to the head related transfer function processing device is omitted, and the interpolation Calculation is reduced. As a result, it becomes possible to correspond to various processing methods such as the processing by IIR filter and FFT device other than the FIR filter as the processing method corresponding to the action of the head related transfer function, and follow the relative movement of the sound image. Easier to do.

The three-dimensional sound image generating apparatus of the present invention preferably comprises sound data storage means for temporarily storing the sound data processed by the plurality of head related transfer function processing means. As a result, various sound data processing methods are possible, and it becomes easier to follow the relative movement of the sound image.

According to the three-dimensional sound image generation method of the present invention, sound data subjected to head related transfer function processing in parallel with input sound data to detect the position of the sound image, and subjected to head related transfer function processing based on the detection result. Sound image is interpolated. The labor of transferring the head related transfer function can be omitted, and the calculation of interpolation can be reduced. This allows
The calculation processing method corresponding to the function of the head related transfer function is FIR
In addition to filters, it is possible to support various processing methods such as processing by IIR filters and FFT devices,
It becomes easy to follow the relative movement of the sound image.

In the three-dimensional sound image generation method of the present invention, it is preferable to temporarily store the sound data subjected to head-related transfer function processing in parallel.
This enables various sound data processing methods,
It becomes easier to follow the relative movement of the sound image.

[0020]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to FIG.

The stereoscopic sound image generating apparatus of this embodiment has a sound source input means 10 such as a microphone for inputting a sound from a sound source, and a sound image position detecting apparatus 11 as a sound image position detecting means for detecting the position of the sound image. There is. The sound source input means 10 includes a head related transfer function processing device 13a as a plurality of head related transfer function processing means for performing head related transfer function processing of input sound data,
, 13n are connected in parallel, the processing device 13a is configured to perform a head-related transfer function process on the direct wave, and the processing devices 13b, ..., 13n respectively transmit the reflected wave via the head. It is configured to process functions. A sound data storage unit 16 as sound data storage means for temporarily storing the sound data processed by the processing devices 13a, 13b, ..., 13n is connected to the processing devices 13a, 13b ,. . In these sound data storage units 16, a sound image interpolating device 17 as a sound image interpolating unit that outputs a sound image output 100 by performing sound image interpolation on the sound data stored in the storage unit 16 based on the detection result by the detecting device 11. It is connected.

The sound image is localized by the time difference between both ears, that is, the phase difference and the level difference for each frequency. This is obtained from the time difference between the left and right head related transfer functions and the level difference for each frequency. When the head-related transfer function is interpolated, it is necessary to interpolate the time difference and the level difference for each frequency. The time difference τ of the head-related transfer function H (ω) is obtained by the theoretical expression when the head is approximated to a sphere, or the time that gives the maximum value of the impulse response h (t) obtained by inverse Fourier transforming the head-related transfer function. In the Fourier transform of h (t-τ) from which the time difference is subtracted, H (ω) exp (-jωτ) represents the level difference for each frequency. Here, the head-related transfer function has a constant distance, and points (θi
, Ζ j). Head-related transfer functions from directions θ1 and θ2 with the same elevation angle are H1 (ω), H2
If (ω) and the time differences are τ1 and τ2, and θ = (1-λ) θ1 + λθ2 (0 ≦ λ ≦ 1), the time difference τ becomes τ = (1-λ) τ1 + λτ2. The level difference for each frequency is (1-λ) H1 (ω) exp (-jωτ1) + λH2
It can be approximated by (ω) exp (-jωτ2). Therefore, the interpolation of the head related transfer function is: {(1-λ) H1 (ω) exp (-jωτ1) + λH2
(Ω) exp (-jωτ2)} * exp (jω {(1-
λ) τ1 + λτ2}) = (1-λ) H1 (ω) exp (-jωλ (τ1 -τ
2)) + λH2 (ω) exp (-jω (1-λ) (τ1−
τ2)) can be approximated. The inverse Fourier transformed impulse response h (t) is h (t) = (1−λ) h1 (t−λ (τ1−τ2)) + λ
It is given by h2 (t- (1- [lambda]) ([tau] 1- [tau] 2)). Here, Σ is a total sum of 0 to ∞ regarding the variable i. Arbitrary signals x (t) and h1 (t), h2
If the convolution with (t) is y1 (t) and y2 (t), then y1 (t) = h1 (t) * x (t) = Σh1 (i) * x
(T-i) y2 (t) = h2 (t) * x (t) = Σh2 (i) * x
(T−i), and the convolution y (t) with h (t) is y (t) = h (t) * x (t−i) = Σ {(1−λ) h1 (i−λ ( τ1 −τ2)) + λh2
(I− (1−λ) (τ1−τ2))} * x (t−i) = (1−λ) Σh1 (i−λ (τ1−τ2)) * x (t
−i) + λ Σh2 (i− (1-λ) (τ1−τ2))}
* X (t-i) = (1-?) Y1 (t-? (? 1-? 2)) +? Y2 (t
− (1-λ) (τ1 −τ2)). Here, y1 and y2 are sound images from a certain direction. This formula shows that the sound image in the desired angular direction can be created by interpolating the sound image instead of interpolating the head related transfer function. The interpolation of the head-related transfer function depends on the length, but the interpolation of the sound image requires only a few calculations and the processing becomes light.

Letting z1 (t) and z2 (t) be the convolutions of an arbitrary signal (x) t and the level difference for each frequency, y1 (t) = z1 (t-λτ1) y2 ( t) = z2 (t- [lambda] [tau] 2) y (t) = (1- [lambda]) z1 (t- (1- [lambda]) [tau] 1- [lambda] [tau].
2)) + [lambda] z2 (t- (1- [lambda]) [tau] 1- [lambda] [tau] 2), and the convolution result with the function representing the level difference for each frequency can be used instead of the head related transfer function.

Even when the sound images are not located on the same plane, the convolution result with the function representing the level difference for each frequency of the horizontal angle θi and elevation angle ζj is zij (t), and the time difference is τij.
If (t), the sound image at θ = (1−λ) θ1 + λθ2 (0 ≦ λ ≦ 1) ζ = (1-μ) ζ1 + μζ2 (0 ≦ μ ≦ 1) is the same as the above, and y ( t) = (1−λ) (1−μ) z11 (t−τ) + (1
−λ) μz12 (t−τ) + λ (1-μ) z21
(T-τ) + λμz22 (t-τ) τ = (1-λ) (1-μ) τ11 + (1-λ) μτ12 +
λ (1-μ) τ21 + λμτ22.

If the reflection component is obtained by a mirror image, the loudness Li of the sound and the time Ti to reach it are determined to be the highest depending on the distance, the reflectance is Ai, and when i = 0, the direct wave is generated. In this case, y (t) = Σh (t) * (AiLix (t−T)) = ΣAiLi (1−λi) (1−μi) z11 (t−τi) + (1−λi) μz12 (t−) τi) + λi (1-μi) z21
(T- [tau] i) + [lambda] i [mu] iz22 (t- [tau] i) [tau] i = Ti + (1- [lambda] i) (1- [mu] i) [tau] i11 + (1- [lambda]
i) μiτi12 + λi (1-μi) τi21 + λiμiτi22.

By obtaining the acoustic signals from a plurality of directions by the above equation, the acoustic signals in the angular direction can be created by interpolating the acoustic signals.

Next, the operation of this embodiment will be described.

Head related transfer function processing devices 13a, 13b,
, 13n generate a reproduction signal by causing a head-related transfer function representing the diffraction effect of the head and auricle from the arrival direction of the sound wave to act on the original sound. It is assumed that the head-related transfer function is measured at points (θi, ζj) where the distance is constant, the horizontal angle is constant, and the elevation angle is constant, and it is assumed that there is a processor corresponding to all the head-related transfer functions. The processing device 13a, 1
Sound signals from the respective directions, which are the outputs of 3b, ..., 13n, are temporarily stored in the sound data storage device 16. From the information of the sound image position detecting device 11 for detecting the relative position of the sound image from the position of the sound image and the position of the head, the horizontal angle θ, and the elevation angle ζ, the sound image interpolating device 17 determines θ = (1−λ) θ1 + λθ2. (0 ≦ λ ≦ 1) ζ = (1−μ) ζ1 + μζ2 Four head related transfer functions (θi, ζj) satisfying (0 ≦ μ ≦ 1) and the time difference τij are determined, and the loudness of the sound is determined by the distance. The length L and the time T until the arrival are determined.

When τ = T + (1-λ) (1-μ) τ11 + (1-λ) μτ12 + λ (1-μ) τ21 + λμτ22, zij (t-τ) is calculated from the sound data storage device 16.
Sound data corresponding to y (t) = L {(1-λ) (1-μ) z11 (t-τ) +
(1-λ) μz12 (t-τ) + λ (1-μ) z21 (t-
τ) + λμz 22 (t-τ)} determines the direct wave.

When the reflection of a wall or floor is taken into consideration, it results in a mirror image position of the sound source wall or floor. Therefore, y (t) = ΣAiLi {(1-λi) (1- μi) z11 (t
−τi) + (1-λi) μz12 (t−τi) + λi (1-μ
i) z21 (t-τi) + λiμi z22 (t-τi)} τi = Ti + (1-λi) (1-μi) τi11 + (1-λ
i) μiτi12 + λi (1-μi) τi21 + λiμiτi22, and the sound image output 100 is determined.

As described above, in the conventional apparatus, the head-related transfer function had to be reset in the head-related transfer processing apparatus every time the relative position of the sound image changed, whereas in the present embodiment, the head-related transfer function was changed. Has the advantage that it can be set only in the initial setting in the head related transfer function processing device.

According to the present embodiment, since the head related transfer function does not change during the processing, it is possible to use an apparatus for exclusive use of FIR filter such as A100 manufactured by Inmos Co., Ltd., and a FIR filter of sufficient length can be used in real time. This facilitates the design of a stereoscopic sound image generation device that can be processed. Further IIR filter,
FFT device for Fourier transform, NT for number theory transform
It is replaced with a three-dimensional sound image generation device using a T device. Note that the IIR filter requires a small number of taps when performing the same approximation, but since it includes an infinite element component,
The head related transfer function must be changed every time the relative position of the sound source changes. In this embodiment, since the head related transfer function does not change during the processing, the IIR filter can be used in the stereoscopic sound image generating apparatus of the invention. FFT device or NT
The T-device can shorten the total processing time by collectively calculating the convolution calculation for a predetermined time period. In this case, it is necessary that the head related transfer coefficient does not change within the predetermined time period. In this embodiment, since the head related transfer function does not change during the processing, the input sound can be taken in first for the time required for the arithmetic processing. Therefore, the FFT device or the NTT device can be used for the stereoscopic sound image generation apparatus of this embodiment.

[0033]

According to the three-dimensional sound image generating device of the first aspect,
The input sound data is subjected to head-related transfer function processing in parallel, the position of the sound image is detected, and the sound data subjected to head-related transfer function processing based on the detection result is configured to be sound image interpolated. The head-related transfer function does not change during processing, and the labor of transferring the head-related transfer function to the head-related transfer function processing device can be omitted. As a result, the head related transfer function processing and the calculation of interpolation are separated, and the calculation of interpolation can be reduced. The calculation processing method corresponding to the action of the head related transfer function is I except the FIR filter.
It is possible to deal with various processing methods such as processing by an IR filter and an FFT device, and it is possible to smoothly follow the relative movement of the sound image.

According to the three-dimensional sound image generating apparatus of the second aspect, various sound data processing methods are possible, and it becomes easier to follow the relative movement of the sound image.

According to the three-dimensional sound image generation method of the third aspect, the head-related transfer function processing is performed in parallel with the input sound data to detect the position of the sound image, and the sound subjected to the head-related transfer function processing based on the detection result. Since the data is interpolated by the sound image, the labor of transferring the head related transfer function can be omitted and the calculation of the interpolation can be reduced. As a result, it becomes possible to correspond to various processing methods such as the processing by IIR filter and FFT device other than the FIR filter as the processing method corresponding to the action of the head related transfer function, and follow the relative movement of the sound image. Easier to do.

According to the four-dimensional sound image generating method of the fourth aspect, various sound data processing methods are possible, and it becomes easier to follow the relative movement of the sound image.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of an embodiment of a three-dimensional sound image generation device of the present invention.

FIG. 2 is a block diagram showing a configuration of a conventional stereoscopic sound image generation device.

[Explanation of symbols]

 10 Sound Source Input Means 11 Sound Image Position Detection Device 13a to 13n Head Transfer Function Processing Device 16 Sound Data Storage Device 17 Sound Image Interpolation Device

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification number Office reference number FI technical display location H03H 17/08 8842-5J H04S 1/00 K 8421-5H 7/00 F 8421-5H

Claims (4)

[Claims] 1. A sound expression input means, a plurality of head related transfer function processing means connected in parallel to the sound source input means for performing head related transfer function processing on input sound data, and detecting a position of a sound image. A stereoscopic sound image generation device comprising: a sound image position detecting means for performing sound image position detection means; and a sound image interpolation means for performing sound image interpolation on the sound data processed by the plurality of head related transfer function processing means based on the detection result by the sound image position detecting means. 2. The stereoscopic sound image generating apparatus according to claim 1, further comprising a sound data storage unit that temporarily stores sound data processed by the plurality of head-related transfer function processing units. 3. Head-related transfer function processing is performed in parallel on the input sound data to detect the position of the sound image, and based on the detection result of the position of the sound image, the sound data subjected to the head-related transfer function processing in parallel is detected. A three-dimensional sound image generation method that interpolates sound images. 4. The stereoscopic sound image generation method according to claim 3, wherein the sound data subjected to the head related transfer function processing is temporarily stored in parallel.