KR20170135604A - A method and an apparatus for processing an audio signal - Google Patents

Abstract

The present invention relates to a method for rendering a DRR matched audio signal by using a recorded signal, and more particularly, to a method and an apparatus for processing an audio signal for immersive and interactive audio rendering.

Description

TECHNICAL FIELD [0001] The present invention relates to an audio signal processing method and apparatus,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a signal processing method and apparatus for effectively reproducing an audio signal, and more particularly, to a signal processing method and apparatus for effectively reproducing an audio signal for implementing an immersive binaural rendering for a portable device including an HMD (Head Mounted Display) To a signal processing method and apparatus.

Binaural rendering techniques are essential for immersive audio listening in HMDs. At this time, in the mobile device with the limitation of the computation amount and the power consumption, not only the computation amount and power consumption burden due to the increase of the object to be rendered or the channel, but also the problem of the individual maladjustment due to the HRTF Artifacts due to insufficient number of images (insufficient spatial resolution), performance degradation due to head tracking lag, and inconvenience.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems, and has an object to solve problems such as spatial discrepancy and sound image localization distortion in the Cinematic 360 VR.

According to an embodiment of the present invention, a binaural signal processing method and apparatus for solving the above problems can be provided.

<< Key Ideas >>

1. It is very important to record the audio signal to provide a more immersive experience in the Cinematic VR production process. When recording only individual sound objects, it is difficult to reflect the acoustic characteristics of the space, and when recording only ambience, it is difficult to recognize the exact position of the sound object. In order to solve this problem, there is a method of simultaneous recording of a sound object and an ambience in a production process and simultaneously providing a clear sound image position and a spatial feeling through post-processing.

2. To reproduce effective immersive spatial audio in Cinematic 360 VR, you can record and process individual sound objects and ambience together. In this case, the sound object and the ambience are played together at the processing stage, and there is a possibility that a difference in the energy ratio between the object and the ambience may cause a mix different from the acoustic characteristic of the actual space.

3. The microphone input signal used to clearly record the sound object contains only the direct sound component that does not include the reverberation of the space, while the microphone used to record the ambience includes direct sound, early reflections, late reverberation, It is possible to increase the immersion feeling in the VR by reflecting the acoustic characteristics of the actual space by using the ambience microphone recording signal to extract the acoustic characteristics of the actually recorded space and using it in the processing step.

According to the embodiment of the present invention, when the Cinematic 360 VR is manufactured, more immersive rendering can be performed using the recorded sound object and the ambience signal.

Sound objects can be converted into HoA and FoA signals using location information.

The converted HoA and FoA can be binaurally rendered with the ambience sound converted to HoA and FoA form to perform immersive binaural rendering.

FIG. 1 is a block diagram of a process for acquiring, processing, and generating cinematic 360 VR audio to provide precise sounding and spatial sense.
2 is a result of the final perceptual evaluation of the final binaural rendering according to the acquisition signal in Cinematic 360VR audio.
3 is a block diagram of the HoA + object and the FoA conversion process for renderer compatibility.

As used herein, terms used in the present invention are selected from general terms that are widely used in the present invention while taking into account the functions of the present invention. However, these terms may vary depending on the intention of a person skilled in the art, custom or the emergence of new technology. Also, in certain cases, there may be a term arbitrarily selected by the applicant, and in this case, the meaning thereof will be described in the description of the corresponding invention. Therefore, it is intended that the terminology used herein should be interpreted relative to the actual meaning of the term, rather than the nomenclature, and its content throughout the specification.

The process of recording an audio signal is very important to provide a more immersive experience in the production process of Cinematic VR. When recording only individual sound objects, it is difficult to reflect the acoustic characteristics of the space, and when recording only ambience, it is difficult to recognize the exact position of the sound object. In order to solve this problem, there is a method of simultaneous recording of a sound object and an ambience in a production process and simultaneously providing a clear sound image position and a spatial feeling through post-processing. The process is shown in FIG. 1 as a block diagram.

, ... ,

및 앰비소닉 신호로 변환된다. 이 과정에서

, ... ,

는 최종 렌더러에 따라 그 형태가 변할 수 있다. 이 경우 format converter를 거쳐 나오는 신호의 조합, 또는 취득 방법에 따른 신호의 조합은 크게 아래와 같이 분류될 수 있다.In Fig. 1, obj ₁ , ..., obj _K are acoustic signals considered as individual objects during recording, and ambience sound is a sound field signal at the ambience microphone position generated by individual sound and recording space. To distinguish the flow of the signal, the sound object is represented by a solid line, and the signal corresponding to ambience is represented by a dotted line. According to the final binaural rendering method, individual objects and ambience recording signals are processed through a format converter process

, ...,

And Ambsonic signals. In this process

, ...,

May vary in shape depending on the final renderer. In this case, the combination of the signals coming through the format converter or the combination of the signals according to the acquisition method can be largely classified as follows.

1) Sound object + Higher Order Ambisonics (HoA) of Ambience

2) HoA of the sound object of 1) + HoA of Ambience = Mixed HoA

3) First Order Ambisonics of Sound Object (FoA) + FoA of Ambience = Mixed FoA

4) FoA of Ambience

For convenience of explanation, it is assumed that the number of sound objects is one, but the same method can be used even when there are a plurality of sound objects.

The above-mentioned classification 1) is obtained from the recorded signal pa using the microphone array, and it has the following relationship as shown in the following Equation 1.

(1)

및

는 마이크로폰 어레이의 개별 구성 마이크로폰의 수평각 및 수직각이다. 또한 Y는 해당 수평각, 수직각을 입력으로 하는 구면조화함수 (spherical harmonic functions) 이다. m과 n은 구면조화함수의 order와 degree를 의미한다. 위의 수학식 1을 Matrix 형태로 나타내면 수학식 2와 같이 나타낼 수 있고, 수학식 2에서 얻고자 하는 HoA of ambience 신호는 B이다.In Equation (1)

And

Are the horizontal and vertical angles of the individual configuration microphones of the microphone array. Y is a spherical harmonic function whose input is a horizontal angle and a vertical angle. m and n mean the order and degree of the spherical harmonic function. Equation (1) can be expressed in Matrix form as shown in Equation (2), and HoA of ambience signal in Equation (2) is B.

(2)

에 대한 구면조화 함수를 이용해 아래 수학식 3과 같이 표현할 수 있다.The HoA transform signal of the sound object in Classification 2)

Can be expressed as Equation (3) below using the spherical harmonic function for the following equation (3).

(3)

The position of the sound object is measured by using an external microphone or an external sensor installed at the reference point or by analyzing the signal of the microphone array.

Considering realistic conditions where infinite spherical harmonic order can not be used, the ambsonic signal approximates the order of the spherical harmonic function by truncating it to M orders. If the HoA signal for the sound object and the ambience is obtained by using Equations (2) and (3), the FoA signal can be obtained by removing the higher order components while leaving the 0th order and first order components. In this approximation process, spatial resolution is also degraded, and the smaller M is, the worse the deterioration becomes. For this reason, performance of binaural rendering is best for object-based rendering of sound objects and scene-based rendering of ambience signals. Based on the performance of binaural rendering based on classification 1), the results of cognitive subjective evaluation in case of scene-based rendering excluding classification 2), 3) and sound object are shown in FIG.

As shown in FIG. 2, when the classification 1) is used as a reference, the final rendering quality can be represented by classification 2)> classification 3), and the quality of scene-based rendering of only the FoA ambience signal is lowest. A typical example of a FoA-based renderer is YouTube Spatial Audio. Classification 1) Signals and Categorization Signals corresponding to 3) or 4) are capable of format conversion for renderer compatibility (eg YouTube Spatial Audio), as shown in FIG.

In Fig. 3, Binaural_1 signal is binaurally rendered by GAUDIO Renderer using signal classification 1), Binaural_2 signal is format converted by signal classification 3) or 4), and then GAURIO Renderer Binaural_3 is the binaural rendered signal by the other renderers after the format conversion, but Binaural_4 is the binaural rendered by the other renderer using signal classification 3) or 4) Lt; / RTI &gt; The solid line and the dotted line indicate what the original sound source is when the signal flow is viewed. The solid line represents the flow of the signal generated from the HoA + Object signal, and the dotted line represents the flow of the signal generated from the FoA signal.

[what is claimed here]:

One.

receives a first audio signal corresponding to hoa (foa)

receiving a second audio signal corresponding to the object

HoA receives the position information of the object on the sound scene (directly calculates or receives it by using an external sensor)

A method of generating a third signal obtained by synthesizing a first signal and a second signal using the object position information

2.

And the third signal is a hoa signal

3.

The third signal is converted into foa and reproduced (rendered)

4.

The process of generating and transmitting the first signal, the second signal, and the position information to the bit stream and generating the third signal is performed by the receiving terminal

5.

The method for generating the third signal may further comprise the steps of converting the first signal to foa, converting the second signal to foa,

6.

In generating the third signal, the first signal and the second signal are synthesized first by hoa and then converted into foa

Claims (1)

Method and apparatus for processing audio signal.

Images