KR102216657B1 - A method and an apparatus for processing an audio signal - Google Patents

Abstract

The present invention relates to an audio signal processing method and apparatus for implementing filtering on an input audio signal with a low computational amount.
To this end, the present invention provides the steps of receiving an input audio signal, wherein the input audio signal includes a plurality of subband signals including a first subband group and a second subband group; Receiving truncated subband filter coefficients for filtering each subband signal of the first subband group, at least one FFT by FFTing the truncated subband filter coefficients in units of preset blocks in the corresponding subband Obtaining filter coefficients; Performing FFT on a subband signal of the first subband group based on a predetermined subframe unit in a corresponding subband; Generating a filtered subframe by multiplying the IFF subframe and the FFT filter coefficient; IFFT the filtered subframe; And generating a filtered subband signal of the first subband group by overlap-adding the IFFTed at least one subframe. It provides an audio signal processing method and an audio signal processing apparatus comprising a.

Description

Audio signal processing method and apparatus {A METHOD AND AN APPARATUS FOR PROCESSING AN AUDIO SIGNAL}

The present invention relates to a signal processing method and apparatus for effectively reproducing an audio signal, and more particularly, to an audio signal processing method and apparatus for implementing filtering on an input audio signal with a low computational amount.

Binaural rendering for listening to a multichannel signal in stereo has a problem that requires a larger amount of computation as the length of the target filter increases. In particular, when using a BRIR (Binaural Room Impulse Response) filter that reflects the characteristics of a recording room, the length can reach 48,000 to 96,000 samples. Here, as in the 22.2 channel format, if the number of input channels increases, the amount of computation is enormous.

, 해당 채널의 좌, 우 BRIR 필터를 각각

,

, 출력 신호를

,

이라고 하면, 바이노럴 필터링(binaural filtering)은 다음과 같은 식으로 표현할 수 있다.input signal of the i-th channel

, The left and right BRIR filters of the corresponding channel

,

, The output signal

,

In terms of, binaural filtering can be expressed in the following equation.

이며, *는 콘볼루션(convolution)을 의미한다. 위의 시간-도메인 콘볼루션은 일반적으로 고속 퓨리에 변환(Fast Fourier Transform, FFT)에 기반한 고속 콘볼루션(fast convolution)을 이용하여 수행된다. 고속 콘볼루션을 이용하여 바이노럴 렌더링을 수행하는 경우, 입력 채널수에 해당하는 횟수의 FFT와 출력 채널수에 해당하는 횟수의 역 고속 퓨리에 변환(Inverse FFT)을 수행해야 한다. 게다가 멀티채널 오디오 코덱과 같이 실시간 재생 환경에서의 경우 딜레이를 고려해야 하기 때문에 블록 단위(block-wise)의 고속 콘볼루션을 수행해야 하며, 이는 전체 길이에 대하여 단순히 고속 콘볼루션을 수행했을 때보다 더 많은 연산량을 소모할 수 있다.From here

And * means convolution. The above time-domain convolution is generally performed using fast convolution based on Fast Fourier Transform (FFT). When binaural rendering is performed using fast convolution, the FFT of the number of input channels and the inverse fast Fourier transform of the number of output channels must be performed. In addition, in the case of a real-time playback environment such as a multi-channel audio codec, since delay must be considered, block-wise high-speed convolution must be performed. It can consume the amount of computation.

However, most coding schemes are performed in the frequency domain, and in some coding schemes (for example, HE-AAC, USAC, etc.), the last step of the decoding process is performed in the QMF domain. Therefore, as shown in Equation 1 above, when binaural filtering is performed in the time domain, it is very inefficient because it additionally requires an operation for QMF synthesis as many as the number of channels. Therefore, there is an advantage when binaural rendering is performed directly in the QMF domain.

In the present invention, in reproducing a multi-channel or multi-object signal in stereo, a filtering process that requires a large amount of computation in binaural rendering to preserve a three-dimensional effect like the original signal is implemented with a very low computational amount while minimizing sound quality loss. It has a purpose for it.

In addition, an object of the present invention is to minimize the spread of distortion through a high-quality filter when there is distortion in the input signal itself.

In addition, the present invention has an object to implement a finite impulse response (FIR) filter having a very long length as a filter having a smaller length.

In addition, the present invention has an object of minimizing distortion of a portion damaged by a missing filter coefficient when performing filtering using a reduced FIR filter.

In order to solve the above problems, the present invention provides an audio signal processing method and an audio signal processing apparatus as follows.

First of all, the present invention provides the steps of receiving an input audio signal; Receiving truncated subband filter coefficients for filtering each subband signal of the input audio signal, wherein the truncated subband filter coefficient is a Binaural Room Impulse Response (BRIR) for binaural filtering of the input audio signal It is at least a part of the subband filter coefficients obtained from the filter coefficients, and the length of the truncated subband filter coefficients is determined based on filter order information obtained by at least partially using characteristic information extracted from the corresponding subband filter coefficients Wherein the truncated subband filter coefficient is composed of at least one FFT filter coefficient in which a Fast Fourier Transform (FFT) has been performed in a predetermined block unit in a corresponding subband; Performing a fast Fourier transform on the subband signal based on a predetermined subframe unit in a corresponding subband; Generating a filtered subframe by multiplying the fast Fourier transformed subframe and the FFT filter coefficient; Inverse fast Fourier transform of the filtered subframe; And generating a filtered subband signal by overlapping-adding the at least one subframe subjected to the inverse fast Fourier transform. It provides an audio signal processing method comprising a.

In addition, as an audio signal processing apparatus for performing binaural rendering of an input audio signal, the input audio signal includes a plurality of subband signals, respectively, and the audio signal processing apparatus directly applies to each of the subband signals. A high-speed convolution unit for rendering sound and early reflection sound parts, wherein the high-speed convolution unit receives an input audio signal; Receives truncated subband filter coefficients for filtering each subband signal of the input audio signal, wherein the truncated subband filter coefficient is a Binaural Room Impulse Response (BRIR) filter for binaural filtering of the input audio signal Is at least a part of the subband filter coefficients obtained from coefficients, and the length of the truncated subband filter coefficients is determined based on filter order information obtained at least partially using characteristic information extracted from the corresponding subband filter coefficients, and , The truncated subband filter coefficient is composed of at least one FFT filter coefficient on which a Fast Fourier Transform (FFT) has been performed in a predetermined block unit in the corresponding subband; Performing fast Fourier transform on the subband signal based on a predetermined subframe unit in a corresponding subband; Generating a filtered subframe by multiplying the fast Fourier transformed subframe and the FFT filter coefficient; Inverse fast Fourier transform of the filtered subframe; It provides an audio signal processing apparatus, characterized in that the inverse fast Fourier transformed at least one subframe is overlapped and added to generate a filtered subband signal.

According to another embodiment of the present invention, there is provided a method comprising: receiving an input audio signal; Receiving truncated subband filter coefficients for filtering each subband signal of the input audio signal, wherein the truncated subband filter coefficient is a Binaural Room Impulse Response (BRIR) for binaural filtering of the input audio signal It is at least a part of the subband filter coefficient obtained from the filter coefficient, and the length of the truncated subband filter coefficient is determined based on filter order information obtained by at least partially using the characteristic information extracted from the corresponding subband filter coefficient ; Obtaining at least one FFT filter coefficient by performing a Fast Fourier Transform (FFT) on the truncated subband filter coefficients in units of a predetermined block in a corresponding subband; Performing a fast Fourier transform on the subband signal based on a predetermined subframe unit in a corresponding subband; Generating a filtered subframe by multiplying the fast Fourier transformed subframe and the FFT filter coefficient; Inverse fast Fourier transform of the filtered subframe; And generating a filtered subband signal by overlapping-adding the at least one subframe subjected to the inverse fast Fourier transform. It provides an audio signal processing method comprising a.

In addition, as an audio signal processing apparatus for performing binaural rendering of an input audio signal, the input audio signal includes a plurality of subband signals, respectively, and the audio signal processing apparatus directly applies to each of the subband signals. A high-speed convolution unit for rendering sound and early reflection sound parts, wherein the high-speed convolution unit receives an input audio signal; Receives truncated subband filter coefficients for filtering each subband signal of the input audio signal, wherein the truncated subband filter coefficient is a Binaural Room Impulse Response (BRIR) filter for binaural filtering of the input audio signal Is at least a part of the subband filter coefficients obtained from coefficients, and the length of the truncated subband filter coefficients is determined based on filter order information obtained at least partially using characteristic information extracted from the corresponding subband filter coefficients, and ; Obtaining at least one FFT filter coefficient by performing Fast Fourier Transform (FFT) on the truncated subband filter coefficients in units of a predetermined block in a corresponding subband; Performing fast Fourier transform on the subband signal based on a predetermined subframe unit in a corresponding subband; Generating a filtered subframe by multiplying the fast Fourier transformed subframe and the FFT filter coefficient; Inverse fast Fourier transform of the filtered subframe; It provides an audio signal processing apparatus, characterized in that the inverse fast Fourier transformed at least one subframe is overlapped and added to generate a filtered subband signal.

In this case, the characteristic information includes reverberation time information of a corresponding subband filter coefficient, and the filter order information is characterized by having one value for each subband.

In addition, the length of at least one of the truncated subband filter coefficients is different from the length of the truncated subband filter coefficients of other subbands.

In addition, the preset block length and the preset sub-frame length have a power of 2 values.

In addition, the length of the preset subframe is characterized in that it is determined based on the length of the preset block in a corresponding subband.

According to an embodiment of the present invention, the performing of the fast Fourier transform includes: dividing the subband signal into the predetermined subframe unit; Generating a temporary sub-frame including a first half composed of the divided sub-frames and a second half composed of a zero-padded value; And fast Fourier transforming the generated temporary subframe.

According to another embodiment of the present invention, there is provided a method comprising: receiving at least one prototype filter coefficient for filtering each subband signal of an input audio signal; Converting the circular filter coefficients into a plurality of subband filter coefficients; Truncating each of the subband filter coefficients based on filter order information obtained by at least partially using characteristic information extracted from the corresponding subband filter coefficients, wherein at least one of the truncated subband filter coefficients has a different length Different from the length of the truncated subband filter coefficients of the band; And generating FFT filter coefficients by performing a Fast Fourier Transform (FFT) on the truncated subband filter coefficients in units of a predetermined block in a corresponding subband. It provides a method of generating a filter for an audio signal comprising a.

Further, as a parameterization unit for generating a filter of an audio signal, the parameterization unit is configured to receive at least one prototype filter coefficient for filtering each subband signal of the input audio signal; Transforming the circular filter coefficients into a plurality of subband filter coefficients; Each of the subband filter coefficients is truncated based on filter order information obtained by at least partially using characteristic information extracted from the corresponding subband filter coefficients, but at least one of the truncated subband filter coefficients has a different length Is different from the length of the truncated subband filter coefficients; A parameterization unit comprising generating FFT filter coefficients by performing Fast Fourier Transform (FFT) on the truncated subband filter coefficients in units of a predetermined block in a corresponding subband.

In this case, the characteristic information includes reverberation time information of a corresponding subband filter coefficient, and the filter order information is characterized by having one value for each subband.

In addition, the length of the preset block is determined as a smaller value among two times the reference filter length of the truncated subband filter coefficient and a preset maximum FFT size, and the reference filter length is a power of 2 of the filter order. It is characterized in that it represents either a true value or an approximate value of the shape.

In addition, when the reference filter length is N and the length of the preset block corresponding thereto is M, the M is a power of 2 and 2N = kM (where k is a natural number). .

According to an embodiment of the present invention, the generating of the FFT filter coefficient includes: dividing the truncated subband filter coefficient into half units of a preset block; Generating a temporary filter coefficient of the predetermined block unit by using the divided filter coefficients, the first half of the temporary filter coefficient is composed of the divided filter coefficients and the second half of the temporary filter coefficient is a zero-padded value Composed; And fast Fourier transforming the generated temporary filter.

In addition, the circular filter coefficient is characterized in that the BRIR filter coefficient in the time domain.

According to another embodiment of the present invention, the step of receiving an input audio signal, the input audio signal each includes a plurality of subband signals, and the plurality of subband signals have a low frequency based on a preset frequency band. Including the signal of the first subband group and the signal of the second subband group of high frequency; Receiving truncated subband filter coefficients for filtering each subband signal of the first subband group, wherein the truncated subband filter coefficient is a subband obtained from circular filter coefficients for filtering the input audio signal At least a portion of the filter coefficient, and the length of the truncated subband filter coefficient is determined based on filter order information obtained by at least partially using characteristic information extracted from the corresponding subband filter coefficient; Obtaining at least one FFT filter coefficient by performing a Fast Fourier Transform (FFT) on the truncated subband filter coefficients in units of a predetermined block in a corresponding subband; Performing a fast Fourier transform on a subband signal of the first subband group based on a predetermined subframe unit in a corresponding subband; Generating a filtered subframe by multiplying the fast Fourier transformed subframe and the FFT filter coefficient; Inverse fast Fourier transform of the filtered subframe; And generating a filtered subband signal by overlapping-adding the at least one subframe subjected to the inverse fast Fourier transform. It provides an audio signal processing method comprising a.

In addition, as an audio signal processing apparatus for performing filtering on an input audio signal, the input audio signal includes a plurality of subband signals, and the plurality of subband signals have a low frequency based on a preset frequency band. A high-speed convolution unit that includes a signal of a first subband group and a signal of a second subband group having a high frequency, and performs filtering on each subband signal of the first subband group; And a tap-delay line processing unit that performs filtering on each subband signal of the second subband group, wherein the fast convolution unit receives the input audio signal and receives each subband of the first subband group. Receive truncated subband filter coefficients for filtering a band signal, wherein the truncated subband filter coefficient is at least a part of a subband filter coefficient obtained from a circular filter coefficient for filtering the input audio signal, and the truncated The length of the subband filter coefficient is determined based on filter order information obtained using at least partially the characteristic information extracted from the corresponding subband filter coefficient, and the truncated subband filter coefficient is preset in the corresponding subband. Obtaining at least one FFT filter coefficient by performing a Fast Fourier Transform (FFT) in block units; A fast Fourier transform is performed on a subband signal of the first subband group based on a preset subframe unit in a corresponding subband, and a filtered subframe is multiplied by the fast Fourier transformed subframe and the FFT filter coefficient. It provides an audio signal processing apparatus comprising generating, inverse fast Fourier transforming the filtered subframe, and generating a filtered subband signal by overlap-adding at least one subframe subjected to the inverse fast Fourier transform. .

In this case, the audio signal processing method includes receiving at least one parameter corresponding to each subband signal of the second subband group, wherein the at least one parameter is the subband filter corresponding to each subband signal Extracted from coefficients; And performing tap-delay line filtering on the subband signals of the second subband group using the received parameters.

In addition, the tap-delay line processing unit receives at least one parameter corresponding to each subband signal of the second subband group, wherein the at least one parameter is the subband filter coefficient corresponding to each subband signal. And performing tap-delay line filtering on the subband signals of the second subband group using the received parameters.

In this case, the tap-delay line filtering may be one-tap-delay line filtering using the parameter.

According to an embodiment of the present invention, it is possible to significantly reduce the amount of computation while minimizing sound quality loss when performing binaural rendering for a multi-channel or multi-object signal.

According to an embodiment of the present invention, it is possible to perform high-quality binaural rendering of a multi-channel or multi-object audio signal, which was not possible in real-time processing in a conventional low-power device.

The present invention provides a method for efficiently performing filtering of various types of multimedia signals including audio signals with a low computational amount.

1 is a block diagram showing an audio signal decoder according to an embodiment of the present invention.
2 is a block diagram showing each configuration of a binaural renderer according to an embodiment of the present invention.
3 to 7 are diagrams showing various embodiments of an audio signal processing apparatus according to the present invention.
8 to 10 are diagrams illustrating a method of generating an FIR filter for binaural rendering according to an embodiment of the present invention.
11 to 14 are views showing various embodiments of the P-part rendering unit of the present invention.
15 and 16 are diagrams showing various embodiments of the QTDL processing of the present invention.
17 and 18 are diagrams showing an embodiment of an audio signal processing method using high-speed convolution in units of blocks.
19 is a diagram showing an embodiment of an audio signal processing process in a high-speed convolution unit of the present invention.

The terms used in the present specification have been selected as currently widely used general terms as possible while considering the functions of the present invention, but this may vary according to the intention, custom, or the emergence of new technologies of the skilled person in the art. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in the description of the corresponding invention. Therefore, it should be noted that the terms used in the present specification should be interpreted based on the actual meaning of the term and the entire contents of the present specification, not a simple name of the term.

1 is a block diagram showing an audio signal decoder according to an embodiment of the present invention. The audio signal decoder of the present invention includes a core decoder 10, a rendering unit 20, a mixer 30, and a post processing unit 40.

First, the core decoder 10 decodes a loudspeaker channel signal, a discrete object signal, an object downmix signal, and a pre-rendered signal. According to an embodiment, in the core decoder 10, a codec based on Unified Speech and Audio Coding (USAC) may be used. The core decoder 10 decodes the received bitstream and delivers it to the rendering unit 20.

The rendering unit 20 renders the signal decoded by the core decoder 10 using reproduction layout information. The rendering unit 20 may include a format converter 22, an object renderer 24, an OAM decoder 25, an SAOC decoder 26, and an HOA decoder 28. The rendering unit 20 performs rendering using any one of the above configurations according to the type of the decoded signal.

The format converter 22 converts the transmitted channel signal into an output speaker channel signal. That is, the format converter 22 performs conversion between the transmitted channel configuration and the speaker channel configuration to be reproduced. If the number of output speaker channels (e.g., 5.1 channels) is less than the number of transmitted channels (e.g., 22.2 channels) or the transmitted channel configuration and the channel configuration to be reproduced are different, the format converter 22 Downmix the signal. The audio signal decoder of the present invention may generate an optimal downmix matrix using a combination of an input channel signal and an output speaker channel signal, and perform downmixing using the matrix. According to an embodiment of the present invention, the channel signal processed by the format converter 22 may include a pre-rendered object signal. According to an embodiment, before encoding an audio signal, at least one object signal may be pre-rendered and mixed with a channel signal. The object signal mixed in this way may be converted into an output speaker channel signal by the format converter 22 together with the channel signal.

The object renderer 24 and the SAOC decoder 26 perform rendering on an object-based audio signal. Object-based audio signals may include individual object waveforms and parametric object waveforms. In the case of individual object waveforms, each object signal is provided to an encoder in a monophonic waveform, and the encoder transmits each object signal using single channel elements (SCEs). In the case of a parametric object waveform, a plurality of object signals are downmixed to at least one channel signal, and characteristics of each object and a relationship between them are expressed by a Spatial Audio Object Coding (SAOC) parameter. The object signals are downmixed and encoded with a core codec, and parametric information generated at this time is transmitted to the decoder together.

Meanwhile, when an individual object waveform or a parametric object waveform is transmitted to the audio signal decoder, compressed object metadata corresponding thereto may be transmitted together. Object metadata quantizes object properties in units of time and space to designate a position and a gain value of each object in a three-dimensional space. The OAM decoder 25 of the rendering unit 20 receives the compressed object metadata, decodes it, and transmits it to the object renderer 24 and/or the SAOC decoder 26.

The object renderer 24 renders each object signal according to a given reproduction format using object metadata. In this case, each object signal may be rendered to specific output channels based on object metadata. The SAOC decoder 26 restores an object/channel signal from the decoded SAOC transport channels and parametric information. The SAOC decoder 26 may generate an output audio signal based on reproduction layout information and object metadata. In this way, the object renderer 24 and the SAOC decoder 26 may render the object signal as a channel signal.

The HOA decoder 28 receives a Higher Order Ambisonics (HOA) signal and HOA side information, and decodes it. The HOA decoder 28 creates a sound scene by modeling a channel signal or an object signal using a separate equation. When a position in the space where the speaker is located in the generated sound scene is selected, rendering can be performed with a speaker channel signal.

Meanwhile, although not shown in FIG. 1, when an audio signal is transmitted to each component of the rendering unit 20, dynamic range control (DRC) may be performed as a preprocessing process. DRC limits the dynamic range of the reproduced audio signal to a certain level. Sounds that are less than a preset threshold are adjusted to be louder and sounds that are louder than a preset threshold are adjusted to be smaller.

The channel-based audio signal and object-based audio signal processed by the rendering unit 20 are transmitted to the mixer 30. The mixer 30 adjusts the delay between the channel-based waveform and the rendered object waveform, and sums them in units of samples. The audio signal summed by the mixer 30 is delivered to the post processing unit 40.

The post processing unit 40 includes a speaker renderer 100 and a binaural renderer 200. The speaker renderer 100 performs post processing for outputting a multi-channel and/or multi-object audio signal transmitted from the mixer 30. Such post processing may include dynamic range control (DRC), loudness normalization (LN), and peak limiter (PL).

The binaural renderer 200 generates a binaural downmix signal of a multi-channel and/or multi-object audio signal. The binaural downmix signal is a two-channel audio signal that allows each input channel/object signal to be represented by a virtual sound source located in three dimensions. The binaural renderer 200 may receive an audio signal supplied to the speaker renderer 100 as an input signal. Binaural rendering is performed based on a Binaural Room Impulse Response (BRIR) filter, and may be performed in a time domain or a QMF domain. According to an embodiment, as a post-processing process for binaural rendering, the above-described dynamic range control (DRC), volume normalization (LN), and peak limiting (PL) may be additionally performed.

2 is a block diagram illustrating each configuration of a binaural renderer according to an embodiment of the present invention. As shown, the binaural renderer 200 according to an embodiment of the present invention includes a BRIR parameterization unit 210, a high-speed convolution unit 230, a late reverberation generation unit 240, a QTDL processing unit 250, Mixer & combiner 260 may be included.

The binaural renderer 200 generates a 3D audio headphone signal (ie, a 3D audio 2 channel signal) by performing binaural rendering on various types of input signals. In this case, the input signal may be an audio signal including at least one of a channel signal (ie, a speaker channel signal), an object signal, and an HOA signal. According to another embodiment of the present invention, when the binaural renderer 200 includes a separate decoder, the input signal may be an encoded bitstream of the aforementioned audio signal. Binaural rendering converts a decoded input signal into a binaural downmix signal, so that a surround sound can be experienced when listening with headphones.

, 서브밴드 도메인에서의 시간 인덱스를 l이라고 하면, QMF 도메인에서의 바이노럴 렌더링은 다음과 같은 식으로 표현할 수 있다.According to an embodiment of the present invention, the binaural renderer 200 may perform binaural rendering of an input signal on a QMF domain. For example, the binaural renderer 200 may receive a multi-channel (N channels) signal in the QMF domain and perform binaural rendering on the multi-channel signal using a BRIR subband filter in the QMF domain. The k-th subband signal of the i-th channel that has passed through the QMF analysis filter bank

, If the time index in the subband domain is l, the binaural rendering in the QMF domain can be expressed as follows.

이며,

은 시간 도메인 BRIR 필터를 QMF 도메인의 서브밴드 필터로 변환한 것이다.here,

Is,

Is transformed from the time domain BRIR filter into a subband filter in the QMF domain.

That is, the binaural rendering may be performed by dividing the channel signal or object signal of the QMF domain into a plurality of subband signals, convolving each subband signal with a corresponding BRIR subband filter, and then summing it.

The BRIR parameterization unit 210 transforms and edits BRIR filter coefficients for binaural rendering in the QMF domain, and generates various parameters. First, the BRIR parameterization unit 210 receives time domain BRIR filter coefficients for multi-channels or multi-objects, and converts them into QMF domain BRIR filter coefficients. Here, the QMF domain BRIR filter coefficients include a plurality of subband filter coefficients respectively corresponding to a plurality of frequency bands. In the present invention, the subband filter coefficient indicates each BRIR filter coefficient of the QMF-transformed subband domain. In this specification, the subband filter coefficients may also be referred to as BRIR subband filter coefficients. The BRIR parameterization unit 210 may edit a plurality of BRIR subband filter coefficients of the QMF domain, respectively, and transfer the edited subband filter coefficients to the high-speed convolution unit 230 or the like. According to an embodiment of the present invention, the BRIR parameterization unit 210 may be included as a component of the binaural renderer 200 or may be provided as a separate device. According to an embodiment, a configuration including a high-speed convolution unit 230, a late reverberation generation unit 240, a QTDL processing unit 250, and a mixer & combiner 260 excluding the BRIR parameterization unit 210 It may be classified as a binaural rendering unit 220.

According to an embodiment, the BRIR parameterization unit 210 may receive a BRIR filter coefficient corresponding to at least one location in the virtual reproduction space as an input. Each position of the virtual reproduction space may correspond to a position of each speaker of the multi-channel system. According to an embodiment, each BRIR filter coefficient received by the BRIR parameterizer 210 may be directly matched to each channel or each object of an input signal of the binaural renderer 200. On the other hand, according to another embodiment of the present invention, each of the received BRIR filter coefficients may have a configuration independent of the input signal of the binaural renderer 200. That is, at least some of the BRIR filter coefficients received by the BRIR parameterization unit 210 may not be directly matched to the input signal of the binaural renderer 200, and the number of received BRIR filter coefficients is the channel and/or the input signal. Alternatively, it may be smaller or larger than the total number of objects.

According to an embodiment of the present invention, the BRIR parameterization unit 210 transforms and edits the BRIR filter coefficients corresponding to each channel or object of the input signal of the binaural renderer 200 so that the binaural rendering unit 220 Can be delivered. The corresponding BRIR filter coefficient may be a matching BRIR or a fallback BRIR for each channel or object. BRIR matching may be determined according to whether there is a BRIR filter coefficient targeting the position of each channel or each object in the virtual reproduction space. In this case, the location information of each channel (or object) may be obtained from an input parameter signaling a channel configuration. If there is a BRIR filter coefficient targeting at least one of each channel of an input signal or a location of each object, the corresponding BRIR filter coefficient may be a matching BRIR of the input signal. However, if there is no BRIR filter coefficient targeting the location of a specific channel or object, the BRIR parameterization unit 210 falls back the BRIR filter coefficient targeting the location most similar to the corresponding channel or object. It can be provided by BRIR.

First, when there is a BRIR filter coefficient having an elevation and azimuth deviation within a desired position (a specific channel or object) and a preset range, a corresponding BRIR filter coefficient may be selected. For example, a BRIR filter coefficient with an elevation equal to the desired location and an azimuth deviation within +/- 20° can be selected. If there is no corresponding BRIR filter coefficient, a BRIR filter coefficient having a minimum geometric distance from the desired position among a set of BRIR filter coefficients may be selected. That is, a BRIR filter coefficient that minimizes the geometric distance between the location of the corresponding BRIR and the desired location may be selected. Here, the location of the BRIR indicates the location of the speaker corresponding to the corresponding BRIR filter coefficient. Also, the geometric distance between the two positions may be defined as a value obtained by summing the absolute value of the elevation deviation of the two positions and the absolute value of the azimuth deviation.

Meanwhile, according to another embodiment of the present invention, the BRIR parameterization unit 210 may transform and edit all of the received BRIR filter coefficients and transfer them to the binaural rendering unit 220. In this case, the selection of BRIR filter coefficients (or edited BRIR filter coefficients) corresponding to each channel or object of the input signal may be performed by the binaural rendering unit 220.

The binaural rendering unit 220 includes a high-speed convolution unit 230, a late reverberation generator 240, and a QTDL processing unit 250, and includes a multi-channel and/or multi-object signal. Receive. In this specification, an input signal including a multi-channel and/or multi-object signal will be referred to as a multi-audio signal. In FIG. 2, the binaural rendering unit 220 is shown to receive a multichannel signal in the QMF domain according to an embodiment, but the input signal of the binaural rendering unit 220 includes a time domain multichannel signal and a multichannel signal. Object signals and the like may be included. In addition, when the binaural rendering unit 220 additionally includes a separate decoder, the input signal may be an encoded bitstream of the multi-audio signal. In addition, in the present specification, the present invention is described based on a case of performing BRIR rendering on a multi-audio signal, but the present invention is not limited thereto. That is, the features provided by the present invention may be applied to other types of rendering filters other than BRIR, and may also be applied to an audio signal of a single channel or a single object instead of a multi-audio signal.

The high-speed convolution unit 230 performs high-speed convolution between the input signal and the BRIR filter to process direct sound and early reflection on the input signal. To this end, the high-speed convolution unit 230 may perform high-speed convolution using a truncated BRIR. The truncated BRIR includes a plurality of subband filter coefficients that are cut dependently on each subband frequency, and is generated by the BRIR parameterization unit 210. In this case, the length of each truncated subband filter coefficient is determined dependently on the frequency of the corresponding subband. The fast convolution unit 230 may perform variable order filtering in the frequency domain by using truncated subband filter coefficients having different lengths according to the subbands. That is, high-speed convolution between the QMF domain subband audio signal and the truncated subband filters of the QMF domain corresponding thereto for each frequency band may be performed. In the present specification, a direct sound and early reflection (D&E) part may be referred to as an F (front)-part.

The late reverberation generator 240 generates a late reverberation signal for the input signal. The late reverberation signal represents the direct sound generated by the high-speed convolution unit 230 and the output signal after the initial reflection sound. The late reverberation generator 240 may process the input signal based on reverberation time information determined from each subband filter coefficient transmitted from the BRIR parameterization unit 210. According to an embodiment of the present invention, the late reverberation generator 240 may generate a mono or stereo downmix signal for an input audio signal, and perform a late reverberation process on the generated downmix signal. In the present specification, the Late Reverberation (LR) part may be referred to as a P (parametric)-part.

A QMF domain tapped delay line (QTDL) processing unit 250 processes a signal of a high frequency band among input audio signals. The QTDL processing unit 250 receives at least one parameter corresponding to each subband signal of a high frequency band from the BRIR parameterization unit 210, and performs tap-delay line filtering in the QMF domain using the received parameter. . According to an embodiment of the present invention, the binaural renderer 200 separates the input audio signal into a low frequency band signal and a high frequency band signal based on a preset constant or a preset frequency band, and the low frequency band signal is In the convolution unit 230 and the late reverberation generation unit 240, the high frequency band signal may be processed by the QTDL processing unit 250, respectively.

The high-speed convolution unit 230, the late reverberation generation unit 240, and the QTDL processing unit 250 respectively output two-channel QMF domain subband signals. The mixer & combiner 260 combines the output signal of the high-speed convolution unit 230, the output signal of the late reverberation generator 240, and the output signal of the QTDL processing unit 250 to perform mixing. In this case, the combination of the output signals is performed separately for the left and right output signals of the two channels. The binaural renderer 200 performs QMF synthesis of the combined output signal to generate a final output audio signal in the time domain.

Hereinafter, various embodiments of the high-speed convolution unit 230, the late reverberation generation unit 240, the QTDL processing unit 250, and combinations thereof of FIG. 2 will be described in detail with reference to each drawing.

3 to 7 show various embodiments of an audio signal processing apparatus according to the present invention. In the present invention, the audio signal processing apparatus may refer to the binaural renderer 200 or the binaural rendering unit 220 shown in FIG. 2 in a narrow sense. However, in the present invention, the audio signal processing apparatus may refer to the audio signal decoder of FIG. 1 including a binaural renderer in a broad sense. Each binaural renderer illustrated in FIGS. 3 to 7 may represent only a partial configuration of the binaural renderer 200 illustrated in FIG. 2 for convenience of description. In addition, in the following specification, an embodiment of a multi-channel input signal may be mainly described, but unless otherwise noted, the channel, multi-channel, and multi-channel input signals each include an object, a multi-object, and a multi-object input signal. Can be used as a concept. In addition, the multi-channel input signal may be used as a concept including a HOA decoded and rendered signal.

3 shows a binaural renderer 200A according to an embodiment of the present invention. When the binaural rendering using BRIR is generalized, it is M-to-O processing to obtain O output signals for a multichannel input signal having M channels. Binaural filtering can be viewed as filtering using filter coefficients corresponding to each input channel and output channel in this process. In FIG. 3, the original filter set H refers to transfer functions from the speaker position of each channel signal to the positions of the left and right ears. Among these transfer functions, measurements measured in a general listening space, that is, a space with reverberation, is called Binaural Room Impulse Response (BRIR). On the other hand, what was measured in the anechoic chamber so that there is no effect of the regeneration space is called Head Related Impulse Response (HRIR), and the transfer function for this is called Head Related Transfer Function (HRTF). Therefore, unlike HRTF, BRIR contains not only direction information, but also information about the reproduction space. According to an embodiment, the BRIR may be replaced by using an HRTF and an artificial reverberator. In the present specification, binaural rendering using BRIR is described, but the present invention is not limited thereto, and the same or corresponding method can be applied to binaural rendering using various types of FIR filters including HRIR and HRTF. In addition, the present invention is applicable not only to binaural rendering of an audio signal, but also to various types of filtering operations of an input signal. Meanwhile, the BRIR may have a sample length of 96K as described above, and multi-channel binaural rendering is performed using M*O different filters, thus requiring a high computational processing process.

According to an embodiment of the present invention, the BRIR parameterization unit 210 may generate modified filter coefficients from the original filter set H to optimize the amount of computation. The BRIR parameterization unit 210 separates the original filter coefficient into an F (front)-part coefficient and a P (parametric)-part coefficient. Here, the F-part represents the direct sound and early reflection (D&E) parts, and the P-part represents the late reverberation (LR) part. For example, the original filter coefficient having a 96K sample length may be divided into an F-part cut only up to the previous 4K sample and a P-part corresponding to the remaining 92K samples.

The binaural rendering unit 220 receives F-part coefficients and P-part coefficients from the BRIR parameterization unit 210, respectively, and renders a multi-channel input signal by using them. According to an embodiment of the present invention, the high-speed convolution unit 230 shown in FIG. 2 renders a multi-audio signal using the F-part coefficient received from the BRIR parameterization unit 210, and a late reverberation generation unit 240 ) May render the multi-audio signal using the P-part coefficient received from the BRIR parameterization unit 210. That is, the high-speed convolution unit 230 and the late reverberation generation unit 240 may correspond to the F-part rendering unit and the P-part rendering unit of the present invention, respectively. According to an embodiment, F-part rendering (binaural rendering using F-part coefficients) is implemented with a conventional Finite Impulse Response (FIR) filter, and P-part rendering (binaural rendering using P-part coefficients) Rendering) can be implemented in a parametric way. On the other hand, the complexity-quality control input provided by the user or the control system may be used to determine information generated by the F-part and/or the P-part.

4 is a binaural renderer 200B according to another embodiment of the present invention, and shows a more detailed method of implementing F-part rendering. For convenience of explanation, the P-part rendering unit is omitted in FIG. 4. In addition, although FIG. 4 shows a filter implemented in the QMF domain, the present invention is not limited thereto and can be applied to all subband processing in other domains.

Referring to FIG. 4, F-part rendering may be performed by the high-speed convolution unit 230 on the QMF domain. For rendering on the QMF domain, the QMF analysis unit 222 includes time domain input signals x0, x1, ... x_M-1 to QMF domain signals X0, X1, ... Convert to X_M-1. At this time, the input signals x0, x1, ... x_M-1 may be a multi-channel audio signal, for example, a channel signal corresponding to a 22.2 channel speaker. A total of 64 subbands may be used for the QMF domain, but the present invention is not limited thereto. Meanwhile, according to an embodiment of the present invention, the QMF analysis unit 222 may be omitted from the binaural renderer 200B. In the case of HE-AAC or USAC using SBR (Spectral Band Replication), processing is performed in the QMF domain, so the binaural renderer 200B directly performs QMF domain signals X0, X1,… without QMF analysis. X_M-1 can be received as an input. Accordingly, when the QMF domain signal is received as a direct input as described above, the QMF used in the binaural renderer according to the present invention is characterized in that it is the same as the QMF used in the previous processing unit (for example, SBR). The QMF synthesizing unit 244 QMF synthesizes the two-channel left and right signals Y_L and Y_R on which binaural rendering has been performed to generate two-channel output audio signals yL and yR in the time domain.

5 to 7 illustrate embodiments of binaural renderers 200C, 200D, and 200E respectively performing F-part rendering and P-part rendering. In the embodiments of FIGS. 5 to 7, the F-part rendering is performed by the high-speed convolution unit 230 on the QMF domain, and the P-part rendering is performed by the late reverberation generation unit 240 on the QMF domain or time domain. do. In the embodiments of FIGS. 5 to 7, detailed descriptions of parts overlapping with the embodiments of the previous drawings will be omitted.

Referring to FIG. 5, the binaural renderer 200C may perform both F-part rendering and P-part rendering in the QMF domain. That is, the QMF analysis unit 222 of the binaural renderer 200C is the time domain input signal x0, x1, ... x_M-1 to QMF domain signals X0, X1, ... It is converted into X_M-1 and transmitted to the high-speed convolution unit 230 and the late reverberation generation unit 240, respectively. The high-speed convolution unit 230 and the late reverberation generation unit 240 are QMF domain signals X0, X1, ... X_M-1 is rendered to generate output signals Y_L, Y_R and Y_Lp and Y_Rp of two channels, respectively. In this case, the high-speed convolution unit 230 and the late reverberation generator 240 may perform rendering using the F-part filter coefficients and the P-part filter coefficients respectively received from the BRIR parameterization unit 210. The output signals Y_L and Y_R of the F-part rendering and the output signals Y_Lp and Y_Rp of the P-part rendering are combined for each of the left and right channels in the mixer & combiner 260 and transmitted to the QMF synthesis unit 224. The QMF synthesizing unit 224 QMF synthesizes the input two-channel left and right signals to generate two-channel output audio signals yL and yR in the time domain.

Referring to FIG. 6, the binaural renderer 200D may perform F-part rendering in the QMF domain and P-part rendering in the time domain, respectively. The QMF analysis unit 222 of the binaural renderer 200D converts the time domain input signal to QMF and transmits it to the high-speed convolution unit 230. The high-speed convolution unit 230 generates two-channel output signals Y_L and Y_R by F-part rendering the QMF domain signal. The QMF synthesis unit 224 converts the output signal of F-part rendering into a time domain output signal and transmits it to the mixer & combiner 260. Meanwhile, the late reverberation generation unit 240 directly receives the time domain input signal and performs P-part rendering. The output signals yLp and yRp of the P-part rendering are transmitted to the mixer & combiner 260. The mixer & combiner 260 generates two-channel output audio signals yL and yR in the time domain by combining the F-part rendering output signal and the P-part rendering output signal in the time domain, respectively.

In the embodiments of FIGS. 5 and 6, F-part rendering and P-part rendering are performed in parallel, respectively, whereas according to the embodiment of FIG. 7, the binaural renderer 200E includes F-part rendering and Each P-part rendering can be performed sequentially. That is, the high-speed convolution unit 230 F-parts rendering the QMF-converted input signal, and the F-part-rendered two-channel signals Y_L and Y_R are converted into time domain signals in the QMF synthesis unit 224 and then reverberated. It may be transmitted to the generator 240. The late reverberation generation unit 240 generates two-channel output audio signals yL and yR in the time domain by performing P-part rendering on the input two-channel signal.

5 to 7 illustrate an embodiment of performing F-part rendering and P-part rendering, respectively, and binaural rendering may be performed by combining or modifying the embodiments of each drawing. For example, in each embodiment, the binaural renderer may individually perform P-part rendering for each of the input multi-audio signals, but downmixes the input signal into a 2-channel left/right signal or a mono signal and then down P-part rendering can also be performed on the mixed signal.

<Variable Order Filtering in Frequency-domain, VOFF>

8 to 10 illustrate a method of generating an FIR filter for binaural rendering according to an embodiment of the present invention. According to an embodiment of the present invention, for binaural rendering in the QMF domain, an FIR filter converted into a plurality of subband filters in the QMF domain may be used. In this case, subband filters cut dependently on each subband frequency may be used for F-part rendering. That is, the high-speed convolution unit of the binaural renderer may perform variable-order filtering in the QMF domain by using truncated subband filters having different lengths according to subbands. The filter generation embodiment of FIGS. 8 to 10 described below may be performed by the BRIR parameterization unit 210 of FIG. 2.

8 shows an embodiment of a length according to each QMF band of a QMF domain filter used for binaural rendering. In the embodiment of FIG. 8, the FIR filter is converted into I QMF subband filters, and Fi denotes a truncated subband filter of QMF subband i. A total of 64 subbands may be used for the QMF domain, but the present invention is not limited thereto. In addition, N represents the length (number of taps) of the original subband filter, and the lengths of the cut subband filter are expressed as N1, N2, and N3, respectively. Here, lengths N, N1, N2, and N3 represent the number of taps in the down-sampled QMF domain.

According to an embodiment of the present invention, a truncated subband filter having a different length (N1, N2, N3) according to each subband may be used for F-part rendering. In this case, the cut subband filter is a filter of a front part cut from the original subband filter, and may also be referred to as a front subband filter. Further, a rear portion of the original subband filter after truncation may be referred to as a rear subband filter and may be used for P-part rendering.

In the case of rendering using a BRIR filter, the filter order (i.e., filter length) for each subband is the parameters extracted from the original BRIR filter, such as reverberation time (RT) information for each subband filter, and EDC (Energy Decay Curve) value, energy decay time information, and the like. The reverberation time may vary depending on the frequency due to the acoustic characteristics of different levels of attenuation in the air for each frequency and the degree of sound absorption according to the material of the wall and ceiling. In general, the lower the frequency signal, the longer the reverberation time. If the reverberation time is long, it means that a lot of information remains behind the FIR filter. Therefore, it is preferable to properly transmit the reverberation information by cutting the filter long and using it. Accordingly, the length of each truncated subband filter of the present invention is determined based at least in part on characteristic information (eg, reverberation time information) extracted from the corresponding subband filter.

The length of the cut subband filter may be determined according to various embodiments. First, according to an embodiment, each subband is classified into a plurality of groups, and a length of each truncated subband filter may be determined according to the classified group. According to the example of FIG. 8, each subband may be classified into three zones (Zone 1, Zone 2, and Zone 3), and the truncated subband filters of Zone 1 corresponding to a low frequency are Zone 1 corresponding to a high frequency. It may have a longer filter order (ie, filter length) than the truncated subband filters of Zone 2 and Zone 3. In addition, as it goes to the high frequency region, the filter order of the truncated subband filter in the region may gradually decrease.

According to another embodiment of the present invention, the length of each truncated subband filter may be independently and variably determined for each subband according to characteristic information of the original subband filter. The length of each truncated subband filter is determined based on the truncated length determined in the corresponding subband, and is not affected by the length of the truncated subband filters of neighboring or other subbands. For example, the length of part or all of the truncated subband filters in Zone 2 may be longer than the length of at least one truncated subband filter in Zone 1.

According to another embodiment of the present invention, frequency domain variable order filtering may be performed only on some of subbands classified into a plurality of groups. That is, truncated subband filters having different lengths may be generated only for subbands belonging to some of the classified at least two groups. According to an embodiment, the group in which the truncated subband filter is generated may be a subband group (for example, Zone 1) classified as a low frequency band based on a preset constant or a preset frequency band. For example, when the sampling frequency of the original BRIR filter is 48 kHz, the original BRIR filter can be converted into a total of 64 QMF subband filters (I = 64). At this time, a truncated subband filter may be generated only for a total of 32 subbands having an index of 0 to 31 in the order of a low frequency band, that is, a subband corresponding to a 0 to 12 kHz band, which is half of the entire 0 to 24 kHz band. . In this case, according to an embodiment of the present invention, the length of the truncated subband filter of the subband of index 0 is longer than the length of the truncated subband filter of the subband of index 31.

The length of the truncated filter may be determined based on additional information acquired by the audio signal processing apparatus, such as a decoder's complexity, a complexity level (profile), or required quality information. The complexity may be determined according to a hardware resource of the audio signal processing apparatus or may be determined according to a value directly input by the user. The quality may be determined according to a user's request, or may be determined with reference to a value transmitted through the bitstream or other information included in the bitstream. In addition, the quality may be determined according to a value obtained by estimating the quality of the transmitted audio signal. For example, the higher the bit rate, the higher the quality may be regarded. In this case, the length of each truncated subband filter may increase proportionally according to complexity and quality, or may change at different rates for each band. In addition, the length of each truncated subband filter may be determined by a corresponding size unit, such as a multiple of a power of 2, in order to obtain an additional gain by high-speed processing such as FFT to be described later. On the other hand, when the determined length of the cut subband filter is longer than the total length of the actual subband filter, the length of the cut subband filter may be adjusted to the length of the actual subband filter.

The BRIR parameterization unit generates truncated subband filter coefficients (F-part coefficients) corresponding to each of the truncated subband filters determined according to the above-described embodiment, and transfers them to the high-speed convolution unit. The fast convolution unit performs frequency domain variable-order filtering on each subband signal of the multi-audio signal using the truncated subband filter coefficients.

9 shows another embodiment of the length of each QMF band of a QMF domain filter used for binaural rendering. In the embodiment of FIG. 9, overlapping descriptions of parts that are the same as or corresponding to the embodiment of FIG. 8 will be omitted.

In the embodiment of FIG. 9, Fi represents a truncated subband filter (front subband filter) used for rendering the F-part of the QMF subband i, and Pi represents a rear subband filter used for rendering the P-part of the QMF subband i. Represents a band filter. N denotes the length (number of taps) of the original subband filter, and NiF and NiP denote the lengths of the front subband filter and the rear subband filter of subband i, respectively. As mentioned above, NiF and NiP represent the number of taps in the down sampled QMF domain.

According to the embodiment of FIG. 9, lengths of not only the front subband filter but also the rear subband filter may be determined based on parameters extracted from the original subband filter. That is, the lengths of the front and rear subband filters of each subband are determined based at least in part on characteristic information extracted from the corresponding subband filter. For example, the length of the front subband filter may be determined based on first reverberation time information of the subband filter, and the length of the rear subband filter may be determined based on second reverberation time information. That is, the front subband filter is a filter of the front part cut based on the first reverberation time information in the original subband filter, and the rear subband filter is a section after the front subband filter, and is between the first and second reverberation times. It can be a filter at the end corresponding to the section of. According to an embodiment, the first reverberation time information may be RT20, and the second reverberation time information may be RT60, but the present invention is not limited thereto.

Within the second reverberation time, there is a part that changes from the early reflection sound part to the late reverberation part. That is, there is a point at which the transition from a section having a deterministic characteristic to a section having a stochastic characteristic exists, and this point is called a mixing time from the perspective of the BRIR of the entire band. In the case of a section before the mixing time, information providing direction is mainly present for each location, which is unique for each channel. On the other hand, since the late reverberation part has common characteristics for each channel, it may be efficient to process a plurality of channels at once. Therefore, by estimating the mixing time for each subband, high-speed convolution is performed through F-part rendering before the mixing time, and processing that reflects the common characteristics of each channel is performed through P-part rendering after the mixing time. have.

However, estimating the mixing time may cause errors due to bias from a perceptual point of view. Therefore, rather than estimating an accurate mixing time and dividing it into an F-part and a P-part based on a corresponding boundary, it is superior in terms of quality to perform high-speed convolution with the length of the F-part as long as possible. Accordingly, the length of the F-part, that is, the length of the front subband filter may be longer or shorter than the length corresponding to the mixing time according to the complexity-quality control.

In addition to this, in addition to the method of cutting as described above to reduce the length of each subband filter, when the frequency response of a specific subband is monotonic, modeling that reduces the filter of the corresponding subband to a lower order is possible. As a representative method, there is an FIR filter modeling using frequency sampling, and a filter that is minimized in terms of least squares can be designed.

According to an embodiment of the present invention, the length of the front subband filter and/or the rear subband filter for each subband may have the same value for each channel of the corresponding subband. Measurement errors may exist in the BRIR, and error factors such as bias may exist in estimating the reverberation time. Accordingly, in order to reduce this effect, the length of the filter may be determined based on a correlation between channels or between subbands. According to an embodiment, the BRIR parameterization unit extracts first characteristic information (for example, first reverberation time information) from subband filters corresponding to each channel of the same subband, and combines the extracted first characteristic information. Accordingly, one filter order information (or first cutoff point information) for a corresponding subband may be obtained. The front subband filters for each channel of the corresponding subband may be determined to have the same length based on the obtained filter order information (or first cutting point information). Likewise, the BRIR parameterization unit extracts second characteristic information (for example, second reverberation time information) from subband filters corresponding to each channel of the same subband, and combines the extracted second characteristic information to obtain the corresponding subband. It is possible to obtain second cutting point information that is commonly applied to the rear subband filter corresponding to each channel of. Here, the front subband filter is a filter of the front part cut based on the first cutoff point information in the original subband filter, and the rear subband filter is a section after the front subband filter, between the first cutoff point and the second cutoff point. Can be a filter at the end corresponding to the section of

Meanwhile, according to another embodiment of the present invention, only F-part processing may be performed on subbands of a specific subband group. At this time, if processing is performed using only the filter up to the first cut point for the subband, the user can perceive it by the energy difference of the processed filter compared to when processing is performed using the entire subband filter. A level of distortion can occur. In order to prevent such distortion, energy compensation may be performed for a region not used for processing in a corresponding subband filter, that is, a region after the first cutting point. The energy compensation can be performed by dividing the filter power up to the first cutting point of the subband filter by the F-part coefficient (front subband filter coefficient) and multiplying the energy of the desired region, such as the total power of the subband filter. Do. Thus, the energy of the F-part coefficient can be adjusted to be equal to the energy of the entire subband filter. In addition, although the P-part coefficient is transmitted from the BRIR parameterization unit, the binaural rendering unit may not perform P-part processing based on the complexity-quality control. In this case, the binaural rendering unit may perform the energy compensation for the F-part coefficient using the P-part coefficient.

In the F-part processing according to the above-described methods, filter coefficients of truncated subband filters having different lengths for each subband are obtained from one time domain filter (ie, a proto-type filter). That is, since one time-domain filter is converted into a plurality of QMF subband filters and the lengths of filters corresponding to each subband are varied, each truncated subband filter is obtained from one circular filter.

The BRIR parameterization unit generates front subband filter coefficients (F-part coefficients) corresponding to each front subband filter determined according to the above-described embodiment, and transfers the coefficients to the high-speed convolution unit. The fast convolution unit performs frequency domain variable-order filtering on each subband signal of the multi-audio signal using the received front subband filter coefficients. In addition, the BRIR parameterization unit may generate rear subband filter coefficients (P-part coefficients) corresponding to each rear subband filter determined according to the above-described embodiment, and transfer the coefficients to the late reverberation generation unit. The late reverberation generator may perform reverberation processing on each subband signal by using the received rear subband filter coefficients. According to an embodiment of the present invention, the BRIR parameterization unit may generate a downmix subband filter coefficient (downmix P-part coefficient) by combining rear subband filter coefficients for each channel, and transfer the result to the late reverberation generation unit. As will be described later, the late reverberation generator may generate two-channel left and right subband reverberation signals using the received downmix subband filter coefficients.

10 shows another embodiment of a method of generating an FIR filter used for binaural rendering. In the embodiment of FIG. 10, duplicate descriptions of parts that are the same as or corresponding to the embodiments of FIGS. 8 and 9 will be omitted.

Referring to FIG. 10, a plurality of QMF-transformed subband filters are classified into a plurality of groups, and different processing may be applied to each classified group. For example, a plurality of subbands are classified into a low frequency first subband group (Zone 1) and a high frequency second subband group (Zone 2) based on a preset frequency band (QMF band i). Can be. In this case, F-part rendering may be performed on the input subband signals of the first subband group, and QTDL processing to be described later may be performed on the input subband signals of the second subband group.

Accordingly, the BRIR parameterization unit generates front subband filter coefficients for each subband of the first subband group, and transfers them to the high-speed convolution unit. The fast convolution unit performs F-part rendering on the subband signals of the first subband group using the received front subband filter coefficients. Depending on the embodiment, the P-part rendering of the subband signals of the first subband group may be additionally performed by the late reverberation generator. In addition, the BRIR parameterization unit obtains at least one parameter from each subband filter coefficient of the second subband group and transmits the obtained to the QTDL processing unit. The QTDL processing unit performs tap-delay line filtering on each subband signal of the second subband group, as described later, using the acquired parameters. According to an embodiment of the present invention, a preset frequency (QMF band i) for classifying the first subband group and the second subband group may be determined based on a predetermined constant value, or the bit of the transmitted audio input signal It may be determined according to the thermal characteristics. For example, in the case of an audio signal using SBR, the second subband group may be set to correspond to the SBR band.

According to another embodiment of the present invention, the plurality of subbands may be classified into three subband groups based on a preset first frequency band (QMF band i) and a second frequency band (QMF band j). That is, the plurality of subbands is a first subband group (Zone 1) that is a low frequency region less than or equal to the first frequency band, and a second subband group that is an intermediate frequency region that is greater than or equal to the second frequency band. It may be classified into a band group (Zone 2) and a third subband group (Zone 3), which is a higher frequency region than the second frequency band. For example, when a total of 64 QMF subbands (subband indices 0 to 63) are classified into the three subband groups, the first subband group includes a total of 32 subbands having indices from 0 to 31, The second subband group may include a total of 16 subbands having indices from 32 to 47, and the third subband group may include subbands having indices from 48 to 63. Here, the subband index has a lower value as the subband frequency decreases.

According to an embodiment of the present invention, binaural rendering may be performed only on subband signals of the first subband group and the second subband group. That is, as described above, F-part rendering and P-part rendering may be performed on the subband signals of the first subband group, and QTDL processing may be performed on the subband signals of the second subband group. I can. Also, binaural rendering may not be performed on the subband signals of the third subband group. Meanwhile, information on the maximum frequency band for performing binaural rendering (Kproc=48) and information on the frequency band for performing convolution (Kconv=32) may be predetermined values, or are determined by the BRIR parameterization unit. It can be delivered to the inoral rendering unit. At this time, the first frequency band (QMF band i) is set to a subband of index Kconv-1, and the second frequency band (QMF band j) is set to subband of index Kproc-1. Meanwhile, values of the maximum frequency band information (Kproc) and the frequency band information (Kconv) for performing convolution may vary depending on the sampling frequency of the original BRIR input and the sampling frequency of the input audio signal.

<Late reverberation rendering>

Next, various embodiments of P-part rendering according to the present invention will be described with reference to FIGS. 11 to 14. That is, various embodiments of the late reverberation generation unit 240 of FIG. 2 performing P-part rendering in the QMF domain will be described with reference to FIGS. 11 to 14. In the embodiments of FIGS. 11 to 14, it is assumed that a multichannel input signal is received as a subband signal in the QMF domain. Accordingly, processing of each of the components of FIGS. 11 to 14, that is, the decorrelator 241, the subband filtering unit 242, the IC matching unit 243, the downmix unit 244, and the energy attenuation matching unit 246 May be performed for each QMF subband. In the embodiments of FIGS. 11 to 14, detailed descriptions of parts that overlap with the embodiments of the previous drawings will be omitted.

In the above-described embodiments of FIGS. 8 to 10, Pi (P1, P2, P3, …) corresponding to the P-part corresponds to the rear part of each subband filter removed by frequency variable cutting, and is typically used for late reverberation. It contains information about. According to the complexity-quality control, the length of the P-part may be defined as the entire filter after the cut point of each subband filter, or may be defined as a smaller length by referring to the second reverberation time information of the corresponding subband filter. have.

P-part rendering may be performed independently for each channel, or may be performed for downmixed channels. In addition, the P-part rendering may be applied through different processing for each subband group or for each subband, or may be applied with the same processing for all subbands. At this time, processing applicable to the P-part includes energy reduction compensation for the input signal, tap-delay line filtering, processing using an IIR (Infinite Impulse Response) filter, processing using an artificial reverberator, and FIIC (Frequency -Independent interaural coherence (FDIC) compensation, frequency-dependent interaural coherence (FDIC) compensation, etc. may be included.

Meanwhile, for parametric processing of the P-part, it is important to preserve two characteristics, namely, energy decay relief (EDR) and frequency-dependent interaural coherence (FDIC). First, when the P-part is observed from the energy point of view, it can be seen that the EDR is the same or similar for each channel. Since each channel has a common EDR, it is reasonable from an energy point of view to downmix all channels to one or two channels and then perform P-part rendering on the downmixed channels. At this time, the P-part rendering operation, which requires M convolutions for M channels, is reduced to M-to-O downmix and one (or two) convolution, resulting in a significant gain of computation. Can provide.

Next, it is necessary to compensate for the FDIC in P-part rendering. There are several methods of estimating FDIC, but the following equation can be used.

는 임펄스 응답

의 STFT(Short Time Fourier Transform) 계수, n은 시간 인덱스, i는 주파수 인덱스, k는 프레임 인덱스, m은 출력 채널 인덱스(L, R)를 나타낸다. 또한, 분자의 함수

는 입력 x의 실수 값을 출력하고,

는 x의 복소 켤레(complex conjugate) 값을 나타낸다. 상기 수식에서 분자 부분은 실수 값 대신 절대값을 취하는 함수로 치환될 수도 있다.here,

Is the impulse response

STFT (Short Time Fourier Transform) coefficient, n is a time index, i is a frequency index, k is a frame index, m is an output channel index (L, R). Also, the function of the molecule

Outputs the real value of the input x,

Represents the complex conjugate value of x. In the above equation, the numerator part may be substituted with a function that takes an absolute value instead of a real value.

Meanwhile, in the present invention, since binaural rendering is performed in the QMF domain, FDIC may be defined by the following equation.

는 BRIR의 서브밴드 필터를 나타낸다.Where i is the subband index, k is the time index in the subband,

Represents the subband filter of BRIR.

The FDIC of the late reverberation part is a parameter mainly affected by the position of the two microphones when BRIR is recorded, and is not affected by the position of the speaker, that is, direction and distance. Assuming that the listener's head is a sphere, BRIR's theoretical FDIC (IC _ideal ) can satisfy the following equation.

Here, r is the distance between the two ears of the listener, that is, the distance between the two microphones, and k is the frequency index.

When analyzing the FDIC using the BRIR of multiple channels, it can be seen that the initial reflection sound mainly included in the F-part is very different for each channel. In other words, the FDIC of the F-part changes very differently for each channel. On the other hand, in the case of the high frequency band, the FDIC changes very significantly, but this is because a large measurement error occurs due to the characteristic of the high frequency band signal in which the energy rapidly attenuates, and when the average for each channel is taken, the FDIC converges to almost zero. . On the other hand, even in the case of the P-part, a difference in FDIC for each channel occurs due to a measurement error, but it can be seen that the average convergence to the sync function shown in Equation (5). According to an embodiment of the present invention, a late reverberation generator for P-part rendering may be implemented based on the above-described characteristics.

11 shows a late reverberation generation unit 240A according to an embodiment of the present invention. According to the embodiment of FIG. 11, the late reverberation generation unit 240A may include a subband filtering unit 242 and downmix units 244a and 244b.

The subband filtering unit 242 uses the P-part coefficients to provide multi-channel input signals X0, X1, ... , X_M-1 is filtered for each subband. The P-part coefficient is received from the BRIR parameterization unit (not shown) as described above, and may include coefficients of rear subband filters having different lengths for each subband. The subband filtering unit 242 performs high-speed convolution between the QMF domain subband signal and the corresponding rear subband filter of the QMF domain for each frequency. In this case, the length of the rear subband filter may be determined based on RT60 as described above, but may be set to a value larger or smaller than RT60 according to complexity-quality control.

The multi-channel input signals are respectively left channel signals X_L0, X_L1, ... by the subband filtering unit 242. , X_L_M-1 and right channel signals X_R0, X_R1,… , Rendered as X_R_M-1. The downmixing units 244a and 244b downmix the rendered plurality of left channel signals and the plurality of right channel signals for each left and right channel, respectively, to generate two channel left and right output signals Y_Lp and Y_Rp.

12 shows a late reverberation generation unit 240B according to another embodiment of the present invention. According to the embodiment of FIG. 12, the late reverberation generation unit 240B includes a decorrelator 241, an IC matching unit 243, downmix units 244a and 244b, and energy attenuation matching units 246a and 246b. can do. In addition, for processing by the late reverberation generation unit 240B, the BRIR parameterization unit (not shown) may include an IC estimation unit 213 and a downmix subband filter generation unit 216.

According to the embodiment of FIG. 12, the late reverberation generation unit 240B may reduce the amount of computation by using the same energy decay characteristics for each channel of the late reverberation part. That is, the late reverberation generation unit 240B performs decoding and interaural coherence (IC) adjustment for each multi-channel signal, and downmixes the adjusted input signal and decoding signal for each channel into left and right channel signals. Afterwards, the energy attenuation of the downmixed signal is compensated to generate 2 channel left and right output signals. More specifically, the decorrelator 241 includes each of the multi-channel input signals X0, X1, ... , Decorrelation signals D0, D1,… for X_M-1 , D_M-1 is created. The decorrelator 241 is a kind of preprocessor for adjusting coherence between both ears, and a phase randomizer may be used, and the phase of the input signal in units of 90 degrees for efficiency You can also change it.

라고 했을 때, IC 매칭이 수행된 좌, 우 채널 신호 X_L, X_R은 다음 수식과 같이 표현될 수 있다.Meanwhile, the IC estimating unit 213 of the BRIR parameterization unit (not shown) estimates the IC value and transmits it to the binaural rendering unit (not shown). The binaural rendering unit may store the received IC value in the memory 255 and transfer it to the IC matching unit 243. The IC matching unit 243 may directly receive an IC value from the BRIR parameterization unit or may obtain an IC value previously stored in the memory 255. The input signals and decorrelation signals for each channel are left channel signals X_L0, X_L1, ... from the IC matching unit 243. , X_L_M-1 and right channel signals X_R0, X_R1,… , Rendered as X_R_M-1. The IC matching unit 243 performs weighted summation between the decorrelation signal and the original input signal for each channel by referring to the IC value, and adjusts the coherence between the two channel signals through this. In this case, since the input signal for each channel is a signal in the subband domain, the aforementioned FDIC can be matched. The original channel signal is X, the decorrelation channel signal is D, and the IC of the subband is

In the case of, the left and right channel signals X_L and X_R on which IC matching has been performed may be expressed as follows.

(East order of abdomen)

, k번째 채널에서의 N번째 샘플까지의 절단된 BRIR을

, N번째 샘플 이후의 절단된 부분의 에너지를 보상한 다운믹스 서브밴드 필터 계수를

라고 한다면,

는 다음과 같은 수식을 이용하여 구할 수 있다.The downmixing units 244a and 244b downmix the plurality of left channel signals and the plurality of right channel signals rendered through IC matching for left and right channels, respectively, and generate two-channel left and right rendering signals. Next, the energy attenuation matching units 246a and 246b generate 2 channel left and right output signals Y_Lp and Y_Rp by reflecting energy attenuation of the 2 channel left and right rendering signals, respectively. The energy attenuation matching units 246a and 246b perform energy attenuation matching by using the downmix subband filter coefficients obtained from the downmix subband filter generator 216. The downmix subband filter coefficients are generated by a combination of rear subband filter coefficients for each channel of the subband. For example, the downmix subband filter coefficient may include a subband filter coefficient obtained by taking a root of an average of square amplitude responses of rear subband filter coefficients for each channel for a corresponding subband. Accordingly, the downmix subband filter coefficient reflects the energy reduction characteristic of the late reverberation part for the corresponding subband signal. The downmix subband filter coefficient may include a downmix subband filter coefficient downmixed to mono or stereo according to an embodiment, and, like FDIC, directly received from the BRIR parameterization unit or from a value previously stored in the memory 225 Can be obtained. BRIR in which the F-part in the kth channel among M channels is cut

, the truncated BRIR up to the Nth sample in the kth channel

, The downmix subband filter coefficient compensated for the energy of the truncated portion after the Nth sample

If you say,

Can be obtained using the following equation.

here,

13 shows a late reverberation generation unit 240C according to another embodiment of the present invention. Each configuration of the late reverberation generation unit 240C of FIG. 13 may be the same as that of the late reverberation generation unit 240B described in the embodiment of FIG. 12, and a data processing order between each component may be partially different.

According to the embodiment of FIG. 13, the late reverberation generation unit 240C may further reduce the amount of calculation by using the same FDIC for each channel of the late reverberation part. That is, the late reverberation generation unit 240C downmixes each multi-channel signal into left and right channel signals, adjusts the IC of the downmixed left and right channel signals, and then adjusts the energy for the adjusted left and right channel signals. By compensating for attenuation, two-channel left and right output signals can be generated.

More specifically, the decorrelator 241 includes each of the multi-channel input signals X0, X1, ... , Decorrelation signals D0, D1,… for X_M-1 , Create D_M-1. Next, the downmixing units 244a and 244b downmix the multi-channel input signal and the decorrelation signal, respectively, to generate two-channel downmix signals X_DMX and D_DMX. The IC matching unit 243 weights the two-channel downmix signal by referring to the IC value, and adjusts the coherence between the two channel signals through this. The energy attenuation compensation units 246a and 246b perform energy compensation for each of the left and right channel signals X_L and X_R on which IC matching is performed by the IC matching unit 243, and thus output signals X_Lp and Y_Rp of the two channels. Create In this case, the energy compensation information used for energy compensation may include a downmix subband filter coefficient for each subband.

14 shows a late reverberation generation unit 240D according to another embodiment of the present invention. Each configuration of the late reverberation generation unit 240D of FIG. 14 may be the same as that of the late reverberation generation units 240B and 240C described in the embodiments of FIGS. 12 and 13, but has a more simplified feature.

First, the down-mix unit 244 includes multi-channel input signals X0, X1, ... , X_M-1 is downmixed for each subband to generate a mono downmix signal (ie, a mono subband signal) X_DMX. The energy attenuation matching unit 246 reflects the energy attenuation of the generated mono downmix signal. In this case, a downmix subband filter coefficient for each subband may be used to reflect energy attenuation. Next, the decorrelator 241 generates a decorrelation signal D_DMX of the mono downmix signal in which the energy attenuation is reflected. The IC matching unit 243 adds weights of the mono downmix signal reflecting the energy attenuation and the decorrelation signal by referring to the FDIC value, thereby generating the left and right output signals Y_Lp and Y_Rp of two channels. According to the embodiment of FIG. 14, since energy attenuation matching is performed only once for the mono downmix signal X_DMX, the amount of computation can be further saved.

<QTDL processing of high frequency band>

Next, various embodiments of the QTDL processing of the present invention will be described with reference to FIGS. 15 and 16. That is, various embodiments of the QTDL processing unit 250 of FIG. 2 that performs QTDL processing in the QMF domain will be described with reference to FIGS. 15 and 16. 15 and 16, it is assumed that the multi-channel input signal is received as a subband signal in the QMF domain. Accordingly, in the embodiment of FIGS. 15 and 16, the tap-delay line filter and the one-tap-delay line filter may perform processing for each QMF subband. In addition, QTDL processing may be performed only on input signals of a high frequency band classified based on a preset constant or a preset frequency band as described above. If SBR (Spectral Band Replication) is applied to the input audio signal, the high frequency band may correspond to the SBR band. In the embodiments of FIGS. 15 and 16, detailed descriptions of parts that overlap with the embodiments of the previous drawings will be omitted.

SBR (Spectral Band Replication), which is used for efficient encoding of high frequency bands, is a tool to secure the bandwidth as much as the original signal by expanding the narrowed band width by discarding the high frequency band signal during low bit rate encoding. . In this case, the high frequency band is generated by using information of the low frequency band transmitted after being encoded and additional information of the high frequency band signal transmitted from the encoder. However, high frequency components generated by using SBR may cause distortion due to incorrect harmonic generation. In addition, the SBR band is a high frequency band, and as described above, the reverberation time of the corresponding frequency band is very short. That is, the BRIR subband filter of the SBR band has little effective information and has a fast attenuation rate. Therefore, in the BRIR rendering for a high frequency band similar to the SBR band, rendering using a small number of effective taps rather than performing convolution may be very effective in terms of the amount of operation compared to the quality of sound quality.

15 shows a QTDL processing unit 250A according to an embodiment of the present invention. According to the embodiment of FIG. 15, the QTDL processing unit 250A uses a tap-delay line filter to provide multi-channel input signals X0, X1, ... , Performs filtering for each subband for X_M-1. The tap-delay line filter performs convolution of only a few preset taps for each channel signal. In this case, the number of taps used may be determined based on a parameter directly extracted from the BRIR subband filter coefficients corresponding to the corresponding subband signal. The parameter includes delay information for each tap to be used in the tap-delay line filter and gain information corresponding thereto.

The number of taps used in the tap-delay line filter may be determined by complexity-quality control. The QTDL processing unit 250A receives a set of parameters (gain information, delay information) corresponding to the number of taps for each channel and subband from the BRIR parameterization unit based on the predetermined number of taps. In this case, the received parameter set is extracted from BRIR subband filter coefficients corresponding to the corresponding subband signal, and may be determined according to various embodiments. For example, among a plurality of peaks of the corresponding BRIR subband filter coefficient, a set of parameters for each of the peaks extracted by the predetermined number of taps in the order of absolute value, real value, or imaginary value may be received. have. At this time, the delay information of each parameter indicates the position information of the corresponding peak, and has an integer value in units of samples in the QMF domain. Also, the gain information is determined based on the size of the peak corresponding to the corresponding delay information. In this case, a corresponding peak value in the subband filter coefficient may be used as the gain information, but a weight value of the corresponding peak after energy compensation for all subband filter coefficients is performed may be used. The gain information is obtained by using a real weight and an imaginary weight for a corresponding peak together, and thus has a complex value.

The plurality of channel signals filtered by the tap-delay line filter are summed into two channel left and right output signals Y_L and Y_R for each subband. Meanwhile, parameters used in each tap-delay line filter of the QTDL processing unit 250A may be stored in a memory during an initialization process of binaural rendering, and QTDL processing may be performed without an additional operation for parameter extraction.

16 shows a QTDL processing unit 250B according to another embodiment of the present invention. According to the embodiment of FIG. 16, the QTDL processing unit 250B uses a one-tap-delay line filter to provide multi-channel input signals X0, X1, ... , Performs filtering for each subband for X_M-1. The one-tap-delay line filter can be understood as performing convolution in only one tap for each channel signal. The tap used at this time may be determined based on a parameter directly extracted from the BRIR subband filter coefficients corresponding to the corresponding subband signal. As described above, the parameter includes delay information extracted from the BRIR subband filter coefficient and gain information corresponding thereto.

In Fig. 16, L_0, L_1, ... L_M-1 represents the delay for BRIR from the M channels to the left ear, R_0, R_1, ... , R_M-1 represents a delay for BRIR from M channels to the right ear, respectively. In this case, the delay information indicates position information on the maximum peak in the order of absolute value size, real value size, or imaginary value size among corresponding BRIR subband filter coefficients. In addition, in FIG. 16, G_L_0, G_L_1, ... , G_L_M-1 denotes a gain corresponding to each delay information of the left channel, G_R_0, G_R_1, ... , G_R_M-1 denotes a gain corresponding to each delay information of the right channel, respectively. As described above, each gain information is determined based on the size of a peak corresponding to the corresponding delay information. In this case, a corresponding peak value in the subband filter coefficients itself may be used as the gain information, but a weight value of the corresponding peak after energy compensation for all subband filter coefficients is performed may be used. The gain information is obtained by using a real weight and an imaginary weight for a corresponding peak together, and thus has a complex value.

As in the embodiment of FIG. 15, a plurality of channel signals filtered by a one-tap-delay line filter are summed into two channel left and right output signals Y_L and Y_R for each subband. In addition, parameters used in each one-tap-delay line filter of the QTDL processing unit 250B may be stored in memory during the initialization process of binaural rendering, and QTDL processing may be performed without an additional operation for parameter extraction. have.

<Block-wise Fast Convolution>

17 to 19 illustrate an audio signal processing method using block-wise high-speed convolution according to an embodiment of the present invention. In the embodiments of FIGS. 17 to 19, detailed descriptions of parts that overlap with the embodiments of the previous drawings will be omitted.

According to an embodiment of the present invention, for optimal binaural rendering in terms of efficiency and performance, high-speed convolution in units of preset blocks may be performed. The high-speed convolution based on FFT has a feature that the larger the FFT size, the smaller the amount of computation, but the overall processing delay increases and the memory usage increases. If a BRIR having a length of 1 second is high-speed convolution with an FFT size of twice the length of the corresponding length, although it is efficient from the viewpoint of the amount of computation, a delay of 1 second occurs, and the corresponding buffer and processing memory Will need. The audio signal processing method having a long delay time is not suitable for applications for real-time data processing. Since the smallest unit that can be decoded in the audio signal processing apparatus is a frame, it is preferable to perform high-speed convolution in block units with a size corresponding to the frame unit in the binaural rendering.

17 shows an embodiment of an audio signal processing method using high-speed convolution in units of blocks. Like the above-described embodiment, in the embodiment of FIG. 17, the circular FIR filter is converted into I subband filters, and Fi denotes a truncated subband filter of subband i. Each subband (Band 0 to Band I-1) may represent a subband in the frequency domain, that is, a QMF subband. A total of 64 subbands may be used for the QMF domain, but the present invention is not limited thereto. In addition, N represents the length (number of taps) of the original subband filter, and the lengths of the cut subband filter are expressed as N1, N2, and N3, respectively. That is, the length of the truncated subband filter coefficient of subband i included in Zone 1 is N1, the length of the truncated subband filter coefficient of subband i included in Zone 2 is N2, and Zone 3 is included. The length of the truncated subband filter coefficient of subband i has a value of N3. Here, lengths N, N1, N2, and N3 represent the number of taps in the down-sampled QMF domain. As described above, the length of the truncated subband filter may be independently determined for each subband group (Zone 1, Zone 2, Zone 3), as shown in FIG. 17, but may be independently determined for each subband. .

Referring to FIG. 17, the BRIR parameterization unit (or binaural rendering unit) of the present invention performs a fast Fourier transform by using a truncated subband filter coefficient in a predetermined block unit in a corresponding subband (or subband group). To generate FFT filter coefficients. In this case, the preset block length M_i in each subband i is determined based on the preset maximum FFT size L. More specifically, the length (M_i) of a preset block in subband i may be expressed by the following equation.

Here, L is the preset maximum FFT size, and N_i is the reference filter length of the truncated subband filter coefficient.

That is, the preset block length M_i may be determined as a smaller value of twice the reference filter length N_i of the truncated subband filter coefficient and the preset maximum FFT size L. If, as in Zone 1 and Zone 2 of FIG. 17, twice the reference filter length (N_i) of the truncated subband filter coefficient is greater than or equal to (or greater than) the maximum FFT size (L), The preset block length (M_i) is determined by the maximum FFT size (L). However, as in Zone 3 of FIG. 17, when the value twice the reference filter length (N_i) of the truncated subband filter coefficient is less than (or less than or equal to) the maximum FFT size (L), The length M_i is determined as a value twice the reference filter length N_i. As will be described later, since the truncated subband filter coefficients are expanded to twice the length through zero-padding and then fast Fourier transform is performed, the length (M_i) of the block for fast Fourier transform is the reference filter length (N_i) It may be determined based on a comparison result between the twice value of and a preset maximum FFT size (L).

Here, the reference filter length N_i represents either a true value or an approximate value in the form of a power of 2 of the filter order (ie, the length of the truncated subband filter coefficient) in the corresponding subband. That is, if the filter order of subband i is a power of 2, the filter order is used as the reference filter length (N_i) in subband i. If the filter order is not a power of 2, the power of the filter order is 2 The raised value or the lowered value in the form of a square is used as the reference filter length (N_i). For example, since N3, the filter order of subband I-1 of Zone 3, is not a power of 2, the approximate value N3' of the power of 2 is used as the reference filter length (N_I-1) of the subband. I can. At this time, since the double value of the reference filter length N3' is smaller than the maximum FFT size (L), the preset block length (M_I-1) in subband I-1 will be set to twice the value of N3'. I can. Meanwhile, according to an embodiment of the present invention, both the preset block length M_i and the reference filter length N_i may be a power of 2 values.

In this way, when the block length M_i in each subband is determined, fast Fourier transform is performed on the subband filter coefficients truncated in units of the determined block. More specifically, the BRIR parameterization unit divides the truncated subband filter coefficient in units of half (M_i/2) of a preset block. The area at the boundary of the dotted line of the F-part illustrated in FIG. 17 represents subband filter coefficients divided into half units of a preset block. Next, the BRIR parameterization unit generates temporary filter coefficients in a preset block unit (M_i) by using each of the divided filter coefficients. In this case, the first half of the temporary filter coefficient is composed of divided filter coefficients, and the second half is composed of zero-padded values. Through this, a temporary filter coefficient having a preset block length M_i is generated using a filter coefficient having a half length M_i/2 of the preset block. Next, the BRIR parameterization unit generates FFT filter coefficients by fast Fourier transform of the generated temporary filter coefficients. The generated FFT filter coefficients may be used for high-speed convolution of the input audio signal in units of blocks. That is, the high-speed convolution unit of the binaural renderer can perform high-speed convolution by multiplying the generated FFT filter coefficients and the corresponding multi-audio signals in sub-frame units (e.g., multiplying complex numbers) as described later. have.

As described above, according to an embodiment of the present invention, the BRIR parameterization unit performs a fast Fourier transform on the subband filter coefficients truncated in blocks of independently determined lengths for each subband (or for each subband group), Can be created. Accordingly, fast convolution may be performed using a different number of blocks for each subband (or for each subband group). In this case, the number of blocks k i in subband _i may satisfy the following equation.

Where k _i is a natural number.

That is, the number of blocks (k _i ) in subband _i may be determined as a value obtained by dividing a value twice the reference filter length (N_i) in the subband by a preset block length (M_i).

18 shows another embodiment of an audio signal processing method using high-speed convolution in units of blocks. In the embodiment of FIG. 18, duplicate descriptions of parts that are the same as or correspond to the embodiment of FIG. 10 or 17 are omitted.

Referring to FIG. 18, a plurality of subbands in the frequency domain include a first subband group (Zone 1) having a low frequency based on a preset frequency band (QMF band i), and a second subband group having a high frequency ( It can be classified as Zone 2). Alternatively, the plurality of subbands are three subband groups, that is, a first subband group (Zone 1) and a second subband group based on a preset first frequency band (QMF band i) and a second frequency band (QMF band j). It may be classified into a subband group (Zone 2) and a third subband group (Zone 3). In this case, F-part rendering using fast convolution in block units may be performed on the input subband signals of the first subband group, and QTDL processing may be performed on the input subband signals of the second subband group. In addition, rendering may not be performed on the subband signals of the third subband group.

Accordingly, according to an embodiment of the present invention, the above-described process of generating FFT filter coefficients in units of blocks may be limitedly performed on the front subband filters Fi of the first subband group. Meanwhile, as described above, according to an embodiment, the P-part rendering of the subband signals of the first subband group may be performed by the late reverberation generator. According to an embodiment, the late reverberation generator may also perform P-part rendering in units of a predetermined block. To this end, the BRIR parameterization unit may generate FFT filter coefficients in a predetermined block unit corresponding to each of the rear subband filters Pi of the first subband group. Although not shown in FIG. 18, the BRIR parameterization unit converts coefficients of each rear subband filter (Pi) or downmix subband filter (downmix P-part coefficient) into a preset block unit by fast Fourier transform and at least one FFT filter Can generate coefficients. The generated FFT filter coefficients are transmitted to the late reverberation generator and may be used for P-part rendering of the input audio signal. That is, the late reverberation generator may perform P-part rendering by multiplying the obtained FFT filter coefficients and the subband signals of the first subband group corresponding thereto by complex numbers in units of subframes.

In addition, as described above, the BRIR parameterization unit obtains at least one parameter from each subband filter coefficient of the second subband group and transmits it to the QTDL processing unit. As described above, the QTDL processing unit performs tap-delay line filtering on each subband signal of the second subband group using the acquired parameters. Meanwhile, according to an additional embodiment of the present invention, the BRIR parameterization unit may generate at least one FFT filter coefficient by performing a fast Fourier transform of a predetermined block unit on the acquired parameter. The BRIR parameterization unit transfers the FFT filter coefficients corresponding to each subband of the second subband group to the QTDL processing unit. The QTDL processing unit may perform filtering by multiplying the obtained FFT filter coefficients and the subband signals of the second subband group corresponding thereto by complex numbers in units of subframes.

The process of generating the FFT filter coefficients described above in FIGS. 17 and 18 may be performed by the BRIR parameterization unit included in the binaural renderer. However, the present invention is not limited thereto, and may be performed in a BRIR parameterization unit separate from the binaural rendering unit. In this case, the BRIR parameterization unit transfers the truncated subband filter coefficients to the binaural rendering unit in the form of a block-based FFT filter coefficient. That is, the truncated subband filter coefficients transmitted from the BRIR parameterization unit to the binaural rendering unit are composed of at least one FFT filter coefficient in which fast Fourier transform is performed in block units.

In addition, in the above-described embodiment, it has been described that the FFT filter coefficient generation process using the fast Fourier transform in block units is performed by the BRIR parameterization unit, but the present invention is not limited thereto. That is, according to another embodiment of the present invention, the above-described FFT filter coefficient generation process may be performed in the binaural rendering unit. The BRIR parameterization unit transmits the truncated subband filter coefficient obtained by truncating the BRIR subband filter coefficient to the binaural rendering unit. The binaural rendering unit may receive the truncated subband filter coefficients from the BRIR parameterization unit, and generate at least one FFT filter coefficient by fast Fourier transforming the truncated subband filter coefficients in units of a preset block.

19 shows an embodiment of an audio signal processing process in the high-speed convolution unit of the present invention. According to the embodiment of FIG. 19, the high-speed convolution unit of the present invention may filter an input audio signal by performing high-speed convolution in units of blocks.

First, the fast convolution unit acquires at least one FFT filter coefficient constituting a truncated subband filter coefficient for filtering each subband signal. To this end, the high-speed convolution unit may receive the FFT filter coefficient from the BRIR parameterization unit. According to another embodiment of the present invention, the high-speed convolution unit (or a binaural rendering unit including the same) receives the truncated subband filter coefficients from the BRIR parameterization unit, and sets the truncated subband filter coefficients to a preset block. FFT filter coefficients can be generated by fast Fourier transform in units. According to the above-described embodiment, the length M_i of a preset block in each subband is determined, and the number of FFT filter coefficients corresponding to the number k _i of blocks in the subband (FFT coef. 1 to FFT coef. k) is determined. _i ) is obtained.

Meanwhile, the fast convolution unit performs fast Fourier transform on each subband signal of the input audio signal based on a preset subframe unit in the corresponding subband. To this end, the fast convolution unit divides the subband signal into preset subframe units. In order to perform block-wise fast convolution between the input audio signal and the truncated subband filter coefficients, the length of the subframe is determined based on the preset block length M_i in the corresponding subband. According to an embodiment of the present invention, since each divided subframe is extended to twice the length through zero-padding and then fast Fourier transform is performed, the length of the subframe is half the length of a preset block (M_i/2 ) Can be determined. According to an embodiment of the present invention, the length of the subframe may be set to have a power of 2 value. Next, the high-speed convolution unit generates temporary subframes each having a length twice the length of the subframe (ie, length M_i) by using the divided subframes (subframe 1 to subframe K _i ). At this time, the first half of the temporary subframe is composed of divided subframes, and the second half is composed of zero-padded values. The fast convolution unit generates an FFT subframe by fast Fourier transforming the generated temporary subframe.

The fast convolution unit generates a filtered subframe by multiplying the fast Fourier transformed subframe (ie, FFT subframe) and the FFT filter coefficients. The complex multiplier (CMPY) of the fast convolution unit may generate a filtered subframe by performing complex multiplication between the FFT subframe and the FFT filter coefficient. Next, the fast convolution unit generates a fast convoluted subframe by inverse fast Fourier transforming each filtered subframe. The fast convolution unit overlap-adds at least one fast conv. subframe converted from the inverse fast Fourier transform to generate a filtered subband signal. The filtered subband signal may constitute an output audio signal in a corresponding subband. According to an embodiment, in a step before or after the inverse fast Fourier transform, subframes for each channel of the same subband may be summed into subframes for two output channels.

In addition, in order to minimize the computational amount of the inverse fast Fourier transform, the FFT filter coefficient after the first FFT filter coefficient of the corresponding subband, that is, the FFT coef. The filtered subframe obtained by performing complex multiplication with m (m is 2 or more and k _i or less) is stored in the memory (buffer), and is summed up when subframes after the current subframe are processed, and then reversed. Fast Fourier transform can be performed. For example, the filtered subframe obtained through complex multiplication between the first FFT subframe (FFT subframe 1) and the second FFT filter coefficient (FFT coef. 2) is stored in the buffer and then corresponding to the second subframe. At the time point, the filtered subframe obtained through complex multiplication between the second FFT subframe 2 and the first FFT filter coefficient 1 is added, and the inverse fast Fourier transform is performed on the summed subframe. Can be done. Similarly, the filtered subframe obtained through complex multiplication between the first FFT subframe (FFT subframe 1) and the third FFT filter coefficient (FFT coef. 3), the second FFT subframe (FFT subframe 2) and the second FFT Filtered subframes obtained through complex multiplication between filter coefficients (FFT coef. 2) may be stored in each buffer. The filtered subframe stored in the buffer is the filtered subframe obtained through complex multiplication between the third FFT subframe 3 and the first FFT filter coefficient 1 at a time corresponding to the third subframe. The frame is summed and the inverse fast Fourier transform may be performed on the summed sub-frame.

As another embodiment of the present invention, the length of the subframe may have a value smaller than the half length (M_i/2) of the preset block. In this case, each sub-frame may be extended to a preset block length M_i through zero-padding, and then fast Fourier transform may be performed. In addition, in the case of overlap-adding a filtered subframe generated using a complex multiplier (CMPY) of the high-speed convolution unit, the overlap interval is not the length of the subframe, but half the length of the preset block (M_i/2). Can be performed on the basis of.

In the above, the present invention has been described through specific embodiments, but those skilled in the art can modify and change without departing from the spirit and scope of the present invention. That is, although the present invention has been described with respect to an embodiment of binaural rendering for a multi-audio signal, the present invention can be equally applied and extended to various multimedia signals including video signals as well as audio signals. Therefore, what can be easily inferred by a person belonging to the technical field to which the present invention belongs from the detailed description and examples of the present invention is interpreted as belonging to the scope of the present invention.

10: core decoder 20: rendering unit
30: mixer 40: post processing unit
100: speaker renderer 200: binaural renderer
210: BRIR parameterization unit 220: binaural rendering unit
230: high-speed convolution unit 240: late reverberation generation unit
250: QTDL processing unit 260: Mixer & combiner

Claims (10)

Receiving an input audio signal, wherein each of the input audio signals includes a plurality of subband signals, and the plurality of subband signals are signals of a first subband group having a low frequency and a high frequency based on a preset frequency band. Including the signal of the second subband group;
Receiving truncated subband filter coefficients for filtering each subband signal of the first subband group, wherein the truncated subband filter coefficient is a subband obtained from circular filter coefficients for filtering the input audio signal It is at least a part of the filter coefficient, and the length of the truncated subband filter coefficient is determined based on filter order information obtained by at least partially using reverberation time information extracted from the corresponding subband filter coefficient, and the filter order information Is determined as a variable value in the frequency domain according to the reverberation time information of each subband;
Obtaining at least one FFT filter coefficient by performing a Fast Fourier Transform (FFT) on the truncated subband filter coefficients in a predetermined block unit in a corresponding subband, wherein the preset block length is a corresponding subband Is determined to be a smaller value of two times the reference filter length at and a preset maximum FFT size, and the reference filter length represents either a true value or an approximate value in the form of a power of 2 of the filter order in a corresponding subband;
Performing a fast Fourier transform on a subband signal of the first subband group based on a predetermined subframe unit in a corresponding subband;
Generating a filtered subframe by multiplying the fast Fourier transformed subframe and the FFT filter coefficient;
Inverse fast Fourier transform of the filtered subframe; And
Generating a filtered subband signal of the first subband group by overlap-adding at least one subframe subjected to the inverse fast Fourier transform; Audio signal processing method comprising a. The method of claim 1,
And the filter order information has one value for each subband. The method of claim 1,
The audio signal processing method, wherein the length of the at least one truncated subband filter coefficient is different from the length of the truncated subband filter coefficient of another subband. The method of claim 1,
Receiving at least one parameter corresponding to each subband signal of the second subband group, the at least one parameter being extracted from the subband filter coefficients corresponding to each subband signal; And
And performing tap-delay line filtering on the subband signals of the second subband group by using the received parameters. The method of claim 4,
The tap-delay line filtering is an audio signal processing method, characterized in that the one-tap-delay line filtering using the parameter. An audio signal processing apparatus for performing filtering on an input audio signal, wherein each of the input audio signals includes a plurality of subband signals, and the plurality of subband signals are first low-frequency signals based on a preset frequency band. Including the signal of the subband group and the signal of the second subband group of high frequency,
A high-speed convolution unit that performs filtering on each subband signal of the first subband group; And
A tap-delay line processing unit that performs filtering on each subband signal of the second subband group,
The high-speed convolution unit,
Receive the input audio signal,
Receives truncated subband filter coefficients for filtering each subband signal of the first subband group, wherein the truncated subband filter coefficient is a subband filter obtained from circular filter coefficients for filtering the input audio signal It is at least a part of the coefficient, and the length of the truncated subband filter coefficient is determined based on filter order information obtained at least partially using reverberation time information extracted from the corresponding subband filter coefficient, and the filter order information is It is determined as a variable value in the frequency domain according to the reverberation time information of each subband,
At least one FFT filter coefficient is obtained by performing a Fast Fourier Transform (FFT) on the truncated subband filter coefficient in a predetermined block unit in a corresponding subband, and the preset block length is in the corresponding subband. It is determined as a smaller value of twice the reference filter length of and a preset maximum FFT size, and the reference filter length represents either a true value or an approximate value in the form of a power of 2 of the filter order in a corresponding subband,
Fast Fourier transform is performed on the subband signal of the first subband group based on a preset subframe unit in the subband,
Generate a filtered subframe by multiplying the fast Fourier transformed subframe and the FFT filter coefficient,
Inverse fast Fourier transform of the filtered subframe,
And generating a filtered subband signal of the first subband group by overlap-adding at least one subframe subjected to the inverse fast Fourier transform. The method of claim 6,
The audio signal processing apparatus, wherein the filter order information has one value for each subband. The method of claim 6,
The audio signal processing apparatus, wherein the length of the at least one truncated subband filter coefficient is different from the length of the truncated subband filter coefficient of another subband. The method of claim 6,
The tap-delay line processing unit,
Receiving at least one parameter corresponding to each subband signal of the second subband group, wherein the at least one parameter is extracted from the subband filter coefficients corresponding to each subband signal,
And performing tap-delay line filtering on the subband signals of the second subband group using the received parameter. The method of claim 9,
The tap-delay line filtering is an audio signal processing apparatus, characterized in that the one-tap-delay line filtering using the parameter.

Images