RU2406164C2 - Signal coding/decoding device and method - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: invention relates to a coding/decoding method and device, more specifically to a coding/decoding device which processes an audio signal in order to create three-dimensional (3D) sound effects. The decoding method involves extraction of a step-down mixing 3D signal and spatial information from an input bit stream; removal of 3D effects from the step-down mixing 3D signal by performing 3D reproduction operations on the said 3D signal; and generation of a multichannel signal using the spatial information and step-down mixing signal obtained from said removal.

EFFECT: possibility of efficiently coding mutichannel signals with 3D effects and additively restore and playback audio signals with optimum audio quality in accordance with characteristics of the playback medium.

11 cl, 16 dwg

Description

Technical field

The present invention relates to an encoding / decoding method and an encoding / decoding device, and more particularly, to an encoding / decoding device that can process an audio signal so that three-dimensional (3D) sound effects can be generated, and an encoding / decoding method using an encoding device / decoding.

State of the art

The encoding device down-mixes the multi-channel signal into a signal with fewer channels and transmits the down-mix signal to the decoding device. The decoding apparatus then restores the multi-channel signal from the down-mix signal and generates the restored multi-channel signal using three or more speakers, for example, 5.1-channel speakers.

Multi-channel signals can be played back by dual-channel speakers, such as headphones. In this case, in order to make the user feel as if the sounds output from the two-channel speakers were reproduced from three or more sound sources, it is necessary to develop three-dimensional (3D) processing methods that can encode or decode multi-channel signals so that three-dimensional effects can be created .

Disclosure of invention

Technical problem

The present invention provides an encoding / decoding apparatus and an encoding / decoding method that can generate multi-channel signals in various playback environments, efficiently processing signals with 3D effects.

Technical solution

According to an aspect of the present invention, there is provided a decoding method for reconstructing a multi-channel signal, the decoding method including extracting a three-dimensional (3D) downmix signal and spatial information from an input bit stream, removing 3D effects from a 3D downmix signal by performing a 3D reproduction operation on a 3D signal down-mix, and multi-channel signal generation using spatial information and a down-mix signal, we get through removal.

According to another aspect of the present invention, there is provided a decoding method for reconstructing a multi-channel signal, the decoding method comprising extracting a 3D down-mix signal and spatial information from an input bit stream, generating a multi-channel signal using a 3D down-mix signal and spatial information, and removing 3D effects from a multi-channel signal by performing 3D playback on the multi-channel signal.

According to another aspect of the present invention, there is provided an encoding method for encoding a multi-channel signal with multiple channels, the encoding method comprising encoding the multi-channel signal into a downmix signal with fewer channels, generating spatial information regarding the plurality of channels, generating a 3D downmix signal by performing a 3D operation playback over the downmix signal, and generating a bit stream including a 3D signal downmix and the spatial information.

According to another aspect of the present invention, there is provided an encoding method for encoding a multi-channel signal with multiple channels, the encoding method including performing a 3D playback operation on a multi-channel signal, encoding a multi-channel signal obtained by a 3D playback operation into a 3D downmix signal with fewer channels, generating spatial information regarding a plurality of channels; and generating a bit stream including a 3D downmix signal tion and spatial information.

According to another aspect of the present invention, there is provided a decoding apparatus for reconstructing a multi-channel signal, wherein the decoding apparatus includes a bit decompression unit that extracts a 3D encoded downmix signal and spatial information from an input bit stream, a downmix decoder that decodes an encoded 3D downmix signal, a 3D reproducing unit that removes 3D effects from a decoded 3D down-mix signal obtained Nogo decoding performed by the decoder downmix by performing 3D playback of 3D decoded downmix signal and multi-channel decoder which generates a multichannel signal using the spatial information and the downmix signal obtained by removing the formed 3D playback unit.

According to another aspect of the present invention, there is provided a decoding apparatus for reconstructing a multi-channel signal, wherein the decoding apparatus includes a bit decompression unit that extracts a 3D encoded downmix signal and spatial information from an input bit stream, a downmix decoder that decodes a 3D encoded downmix signal, a multi-channel decoder that generates a multi-channel signal using spatial information and 3D s nal downmix signal obtained by decoding, the decoder downmix and 3D playback unit that removes 3D effects from the multi-channel signal by performing 3D playback operation on a multichannel signal.

According to another aspect of the present invention, there is provided an encoding device for encoding a multi-channel signal with multiple channels, the encoding device including a multi-channel encoder that encodes the multi-channel signal into a downmix signal with fewer channels and generates spatial information regarding the plurality of channels, a 3D rendering unit that generates a 3D down-mix signal by performing a 3D playback operation on the down-mix signal tion, the encoder downmix, which encodes 3D down-mix signal, and a block of bits arrangement, which generates a bitstream comprising an encoded 3D down-mix signal and spatial information.

According to another aspect of the present invention, there is provided an encoding device for encoding a multi-channel signal with multiple channels, the encoding device including a 3D playback unit that performs a 3D playback operation on a multi-channel signal, a multi-channel encoder that encodes a multi-channel signal obtained by a 3D playback operation into three-dimensional a downmix signal with fewer channels, and generates spatial information regarding a plurality of channels nalov, a down-mix encoder that encodes a 3D down-mix signal, and a bit composition unit that generates a bit stream including an encoded 3D down-mix signal and spatial information.

According to another aspect of the present invention, there is provided a bitstream including a data field that includes information regarding a 3D downmix signal, a filter information field that includes filter information identifying a filter used to generate a 3D downmix signal, a first field a header, which includes information indicating whether the filter information field contains filter information, a second header field, which includes information indicating whether the filter information field contains filter coefficients or inverse filter coefficients for the filter, and a spatial information field that includes spatial information regarding the plurality of channels.

According to another aspect of the present invention, there is provided a computer readable recording medium having a computer program for performing any of the above decoding methods and the above encoding methods.

Beneficial effects

According to the present invention, it is possible to efficiently encode multi-channel signals with 3D effects and adaptively restore and reproduce audio signals with optimal sound quality according to the characteristics of the playback environment.

Brief Description of the Drawings

FIG. 1 is a block diagram of an encoding / decoding apparatus according to an embodiment of the present invention;

FIG. 2 is a block diagram of an encoding device according to an embodiment of the present invention;

FIG. 3 is a block diagram of a decoding apparatus according to an embodiment of the present invention;

FIG. 4 is a block diagram of an encoding device according to another embodiment of the present invention;

FIG. 5 is a block diagram of a decoding apparatus according to another embodiment of the present invention;

FIG. 6 is a block diagram of a decoding apparatus according to another embodiment of the present invention;

FIG. 7 is a block diagram of a three-dimensional (3D) reproducing apparatus according to an embodiment of the present invention;

FIG. 8-11 illustrate bit streams according to embodiments of the present invention;

FIG. 12 is a block diagram of an encoding / decoding apparatus for processing an arbitrary downmix signal according to an embodiment of the present invention;

FIG. 13 is a block diagram of an arbitrary down-mix signal reproduction / compensation 3D unit according to an embodiment of the present invention;

FIG. 14 is a block diagram of a decoding apparatus for processing a compatible downmix signal according to an embodiment of the present invention;

FIG. 15 is a block diagram of a 3D down-mix compatibility playback / processing unit according to an embodiment of the present invention; and

FIG. 16 is a block diagram of a decoding apparatus for compensating cross-coupling according to an embodiment of the present invention.

The best mode of carrying out the invention

The present invention will be further described more fully with reference to the accompanying drawings, in which exemplary embodiments of the invention are presented.

In FIG. 1 is a block diagram of an encoding / decoding apparatus according to an embodiment of the present invention. According to FIG. 1, coding unit 100 includes a multi-channel encoder 110, a 3D rendering unit 120, a down-mix encoder 130, and a bitmap unit 140.

The multi-channel encoder 110 downmixes a multi-channel multi-channel signal into a down-mix signal, such as a stereo signal or a mono signal, and generates spatial information regarding the channels of the multi-channel signal. Spatial information is needed to reconstruct a multi-channel signal from a downmix signal.

Examples of spatial information include a channel level difference (CLD), which indicates the difference between the energy levels of a channel pair, a channel prediction coefficient (CRC), which is a prediction coefficient used to generate a 3-channel signal based on a 2-channel signal, inter-channel correlation ( ICC), which indicates a correlation between a pair of channels, and a channel time difference (CTD), which is a time interval between a pair of channels.

The 3D reproduction unit 120 generates a 3D downmix signal based on the downmix signal. The 3D down-mix signal can be a 2-channel signal with three or more directions and can thus be played back by 2-channel speakers, such as headphones with 3D effects. In other words, the 3D down-mix signal can be reproduced by 2-channel speakers so that the user can perceive as if the 3D down-mix signal was reproduced from a sound source with three or more channels. The direction of the sound source can be determined based on at least one of the difference between the intensities of the two sounds respectively coming to both ears, the time interval between the two sounds and the phase difference of the two sounds. Therefore, the 3D reproducing unit 120 can convert the downmix signal to a 3D downmix signal based on how people can determine the 3D location of the sound source with their hearing organs.

The 3D reproduction unit 120 may generate a 3D downmix signal by filtering the downmix signal using a filter. In this case, filter related information, for example, a filter coefficient, may be input to the 3D rendering unit 120 by an external source. The 3D reproducing unit 120 may use the spatial information provided by the multi-channel encoder 110 to generate a 3D downmix signal based on the downmix signal. More specifically, the 3D reproducing unit 120 can convert the downmix signal to a 3D downmix signal by converting the downmix signal into an imaginary multi-channel signal using spatial information and filtering the imaginary multi-channel signal.

The 3D reproduction unit 120 may generate a 3D downmix signal by filtering the downmix signal using a head-related transfer function filter (HRTF).

HRTF is a transfer function that describes the transmission of sound waves between a sound source at an arbitrary location and the eardrum, and returns a value that changes according to the direction and height of the sound source. If a signal with no directivity is filtered using HRTF, then the signal can be listened as if it was reproduced from a specific direction.

The 3D reproduction unit 120 may perform 3D reproduction in the frequency domain, for example, the discrete Fourier transform (DFT) region, the fast Fourier transform (FFT) region. In this case, the 3D rendering unit 120 may perform DFT or FFT before the 3D playback operation, or may perform the inverse DFT (ODPF) or the inverse FFT (OBPF) after the 3D playback operation.

The 3D reproduction unit 120 may perform a 3D reproduction operation in the quadrature mirror filter (QMF) / hybrid region. In this case, the 3D rendering unit 120 may perform QMF / hybrid analysis and synthesis operations before or after the 3D playback operation.

The 3D reproduction unit 120 may perform 3D reproduction in the time domain. The 3D playback unit 120 may determine in which area the 3D playback operation is to be performed according to the desired sound quality and the operational capabilities of the encoding / decoding device.

The down-mix encoder 130 encodes the down-mix signal output by the multi-channel encoder 110, or the down-mix 3D signal output by the 3D reproducing unit 120. The downmix encoder 130 may encode the downmix signal output by the multi-channel encoder 110, or the 3D downmix signal output by the 3D playback unit 120 using the audio coding method, the advanced audio coding (AAC) method, the MPEG 3 (MP3) method, or the coding method with using bit arithmetic (BSAC).

The downmix encoder 130 may encode a non-3D downmix signal or a 3D downmix signal. In this case, the encoded non-3D down-mix signal and the encoded 3D down-mix signal can both be included in the bit stream to be transmitted.

The bitmap unit 140 generates a bitstream based on spatial information and either an encoded non-3D downmix signal or an encoded 3D downmix signal.

The bitstream generated by the bitmap unit 140 may include spatial information, downmix identification information indicating whether the downmix signal included in the bitstream is a non-3D downmix signal or a 3D downmix signal, and information identifying a filter used by 3D rendering unit 120 (e.g., HRTF coefficient information).

In other words, the bitstream generated by the bitmap unit 140 may include at least one of a non-3D downmix signal that has not yet undergone 3D processing, and a 3D encoder downmix signal that is obtained by the 3D processing operation performed by the encoding device and downmix identification information identifying the type of downmix signal included in the bit stream.

It can be determined which of the non-3D downmix signal and the 3D encoder downmix signal should be included in the bit stream generated by the bit mapper 140, as selected by the user or according to the functionality of the encoding / decoding apparatus illustrated in FIG. 1, and the characteristics of the playback environment.

The HRTF coefficient information may include the coefficients of the inverse HRTF function used by the 3D rendering unit 120. The HRTF coefficient information may only include a brief HRTF coefficient information used by the 3D rendering unit 120, for example, envelope information of HRTF coefficients. If the bitstream including the inverse HRTF coefficients is transmitted to the decoding device, the decoding device does not need to perform the HRTF coefficient conversion operation, and thus, the computation amount of the decoding device can be reduced.

The bitstream generated by the bitmap unit 140 may also include information regarding the energy change in the signal due to HRTF filtering, that is, information regarding the difference between the energy of the signal to be filtered and the energy of the signal that has been filtered, or the energy ratio the signal to be filtered, and the energy of the signal that has been filtered.

The bitstream generated by the bitmap unit 140 may also include information indicating whether it includes HRTF coefficients. If the coefficients of the HRTF function are included in the bit stream generated by the bitmap unit 140, the bit stream may also include information indicating whether it includes the coefficients of the HRTF function used by the 3D rendering unit 120 or the coefficients of the inverse HRTF function.

According to FIG. 1, the first decoding unit 200 includes a bit decompression unit 210, a downmix decoder 220, a 3D playback unit 230, and a multi-channel decoder 240.

The bit decompression unit 210 receives the input bit stream from the encoding unit 100 and extracts the encoded down-mix signal and spatial information from the input bit stream. The downmix decoder 220 decodes the encoded downmix signal. The downmix decoder 220 may decode the encoded downmix signal using an audio decoding method such as an AAC method, an MP3 method, or a BSAC method.

As described above, the encoded downmix signal extracted from the input bitstream may be an encoded non-3D downmix signal or an encoded 3D encoder downmix signal. Information indicating whether the encoded downmix signal extracted from the input bitstream is an encoded non-3D downmix signal or a encoded 3D encoder downmix signal may be included in the input bitstream.

If the encoded downmix signal extracted from the input bitstream is a 3D encoder downmix signal, then the encoded downmix signal can be simply reproduced by being decoded by the downmix decoder 220.

On the other hand, if the encoded downmix signal extracted from the input bit stream is a non-3D downmix signal, then the encoded downmix signal can be decoded by the downmix decoder 220, and the downmix signal obtained by decoding can be converted to 3D a decoder down-mix signal by a 3D playback operation performed by the third playback unit 233. The 3D decoder down-mix signal can be simply played back.

The 3D reproducing unit 230 comprises a first reproducing unit 231, a second reproducing unit 232 and a third reproducing unit 233. The first reproducing unit 231 generates a downmix signal by performing a 3D reproduction operation on a 3D encoder downmix signal provided by the downmix decoder 220. For example, the first reproduction unit 231 may generate a non-3D down-mix signal by removing 3D effects from the 3D encoder down-mix signal. 3D effects of the 3D encoder downmix signal cannot be completely removed by the first reproduction unit 231. In this case, the down-mix signal output by the first reproducing unit 231 may have some 3D effects.

The first reproducing unit 231 may convert the 3D downmix signal provided by the downmix decoder 220 into a downmix signal with 3D effects removed from it using a filter inverse to the filter used by the 3D reproducing unit 120 of the encoding unit 100. Information regarding a filter used by the 3D reproducing unit 120, or a filter inverse to the filter used by the 3D reproducing unit 120, may be included in the input bitstream.

The filter used by the 3D rendering unit 120 may be an HRTF filter. In this case, the HRTF coefficients used by the coding unit 100, or the HRTF inverse function coefficients, may also be included in the input bit stream. If the HRTF coefficients used by the coding unit 100 are included in the input bitstream, the HRTF coefficients may be inverted, and the inverse transform results may be used during the 3D playback operation performed by the first playback unit 231. If the coefficients of the inverse HRTF function used by the encoding unit 100 are included in the input bit stream, they can simply be used during the 3D playback operation performed by the first playback unit 231 without undergoing any inverse conversion operation. In this case, the computational volume of the first decoding apparatus 100 can be reduced.

The input bit stream may also include filter information (for example, information indicating whether the HRTF coefficients used by the encoding unit 100 are included in the input bit stream) and information indicating whether the filter information has been inverted.

Multichannel decoder 240 generates a 3D multichannel signal with three or more channels based on a downmix signal with remote 3D effects and spatial information extracted from the input bit stream.

The second playback unit 232 may generate a 3D downmix signal with 3D effects by performing a 3D reproduction operation on the downmix signal with 3D effects removed. In other words, the first reproducing unit 231 removes 3D effects from the 3D encoder downmix signal provided by the downmix decoder 220. Thereafter, the second reproducing unit 232 may generate a combined 3D downmix signal with 3D effects desired for the first decoding apparatus 200 by performing a 3D reproducing operation on the downmix signal obtained by deletion performed by the first reproducing unit 231 using a filter of the first decoding apparatus 200 .

The first decoding device 200 may include a playback unit in which two or more first, second and third playback units 231, 232 and 233 are integrated that perform the same operations.

The bitstream generated by the encoding unit 100 may be input to the second decoding device 300, which has a structure different from that of the first decoding device 200. The second decoding device 300 may generate a 3D down-mix signal based on the down-mix signal included in the input bit stream supplied thereto.

More specifically, the second decoding device 300 includes an decompression unit 310, a downmix decoder 320, and a 3D reproducer 330 330. The bit decompression unit 310 receives an input bit stream from an encoding unit 100 and extracts the encoded downmix signal and spatial information from the input bit stream . The downmix decoder 320 decodes the encoded downmix signal. The 3D reproducing unit 330 performs a 3D reproducing operation on the decoded down-mix signal so that the decoded down-mix signal can be converted to a 3D down-mix signal.

FIG. 2 is a block diagram of an encoding apparatus according to an embodiment of the present invention. According to FIG. 2, the encoding device includes reproduction units 400 and 420 and a multi-channel encoder 410. Detailed descriptions of the same encoding processes as for the embodiment of FIG. 1 will be omitted.

According to FIG. 2, the 3D playback units 400 and 420 can be respectively located before and after the multi-channel encoder 410. Thus, the multi-channel signal can be three-dimensionally reproduced by the 3D playback unit 400 and then the three-dimensionally reproduced multi-channel signal can be encoded by the multi-channel encoder 410, thereby generating a pre-processed 3D signal down-mix encoder. Alternatively, the multi-channel signal may be down-mixed by the multi-channel encoder 410, and then the down-mix signal may be three-dimensionally reproduced by the 3D rendering unit 420, thereby generating a post-processed down-mix signal of the encoder.

Information indicating whether the multi-channel signal has been three-dimensionally reproduced before or after the down-mix can be included in the bit stream to be transmitted.

Blocks 400 and 420 may both be located in front of or behind multi-channel encoder 410.

FIG. 3 is a block diagram of a decoding apparatus according to an embodiment of the present invention. According to FIG. 3, the decoding apparatus includes 3D playback units 430 and 450 and a multi-channel decoder 440. Detailed descriptions of the same decoding processes as in the embodiment of FIG. 1 will be omitted.

According to FIG. 3, the 3D playback units 430 and 450 can be respectively located before and after the multi-channel decoder 440. The 3D playback unit 430 can remove 3D effects from the 3D encoder down-mix signal and input the down-mix signal obtained by deletion into the multi-channel decoder 430. Then, the multi-channel decoder 430 can decode the down-mix input supplied to it, thus generating a pre-processed 3D multi-channel signal. Alternatively, the multi-channel decoder 430 may reconstruct the multi-channel signal from the encoded 3D down-mix signal, and the 3D reproducing unit 450 may remove 3D effects from the reconstructed multi-channel signal, thereby forming a post-processed 3D multi-channel signal.

If the 3D encoder down-mix signal provided by the encoding device was generated by performing the 3D playback operation and then the down-mix operation, the 3D encoder down-mix signal can be decoded by performing a multi-channel decoding operation and then the 3D playback operation. On the other hand, if a 3D encoder downmix signal was generated by performing a downmix operation and then a 3D playback operation, then the 3D downmix signal can be decoded by performing a 3D playback operation and then a multi-channel decoding operation.

Information indicating whether the encoded 3D down-mix signal has been obtained by performing the 3D playback operation before or after the down-mix operation can be extracted from the bit stream transmitted by the encoding device.

Blocks 430 and 450 3D playback can both be located before or after the multi-channel decoder 440.

FIG. 4 is a block diagram of an encoding device according to another embodiment of the present invention. According to FIG. 4, an encoding device includes a multi-channel encoder 500, a 3D playback unit 510, a down-mix encoder 520, and a bit composition unit 530. Detailed descriptions of the same coding processes as in the embodiment of FIG. 1 will be omitted.

According to FIG. 4, a multi-channel encoder 500 generates a down-mix signal and spatial information based on an input multi-channel signal. The 3D reproducing unit 510 generates a 3D downmix signal by performing a 3D reproduction operation on the downmix signal.

It can be determined whether to perform the 3D playback operation on the downmix signal of the user's choice or according to the functionality of the encoding device, the characteristics of the playback environment, or the desired sound quality.

The downmix encoder 520 encodes the downmix signal generated by the multi-channel encoder 500 or the 3D downmix signal generated by the 3D reproducing unit 510.

The bit composition unit 530 generates a bit stream based on spatial information or an encoded downmix signal or a 3D encoded encoder downmix signal. The bitstream generated by the bitmap unit 530 may include downmix identification information indicating whether the encoded downmix signal included in the bitstream is a non-3D downmix signal without 3D effects or a 3D downmix signal of an encoder with 3D effects . More specifically, the downmix identification information may indicate whether the bitstream generated by the bitmap unit 530 includes a non-3D downmix signal, a 3D encoder downmix signal, or both of these signals.

FIG. 5 is a block diagram of a decoding apparatus according to another embodiment of the present invention. According to FIG. 5, the decoding apparatus includes an decompression unit 540, a downmix decoder 550, and a 3D playback unit 560. Detailed descriptions of the same decoding processes as the embodiment of FIG. 1 will be omitted.

According to FIG. 5, the bit decompression unit 540 extracts the encoded downmix signal, spatial information, and downmix identification information from the input bit stream. The downmix identification information indicates whether the encoded downmix signal is an encoded non-3D downmix signal without 3D effects or a 3D encoded downmix signal with 3D effects.

If the input bitstream includes both a non-3D downmix signal and a 3D downmix signal, only one of the non-3D downmix signal and the 3D downmix signal can be extracted from the input bitstream at the user's choice or according to functionality decoding device, the characteristics of the playback environment or the desired sound quality.

The downmix decoder 550 decodes the encoded downmix signal. If the down-mix signal obtained by decoding by the down-mix decoder 550 is a 3D encoder down-mix signal obtained by performing the 3D reproduction operation, the down-mix signal can be simply reproduced.

On the other hand, if the downmix signal obtained by decoding by the downmix decoder 550 is a downmix signal without 3D effects, the 3D reproduction unit 560 may generate a 3D downmix signal of the decoder by performing a 3D reproduction operation on the downmix signal obtained by the decoding, performed decoder 550 down-mix.

FIG. 6 is a block diagram of a decoding apparatus according to another embodiment of the present invention. According to FIG. 6, the decoding device includes an unpacking unit 600, a downmix decoder 610, a first 3D playback unit 620, a second 3D playback unit 630, and a filter information storage unit 640. Detailed descriptions of the same decoding processes as in the embodiment of FIG. 1 will be omitted.

The bit decompression unit 600 extracts the encoded three-dimensional encoder down-mix signal and spatial information from the input bit stream. The downmix decoder 610 decodes the encoded 3D encoder downmix signal.

The first 3D reproducing unit 620 removes 3D effects from the 3D downmix signal of the encoder obtained by decoding performed by the downmix decoder 610 using a filter inverse to the filter of the encoding device to perform the 3D reproduction operation. The second playback unit 630 generates a combined 3D downmix signal with 3D effects by performing a 3D playback operation on the downmix signal obtained by deletion performed by the first 3D playback unit 620 using a filter stored in the decoding device.

The second 3D reproducing unit 630 may perform the 3D reproducing operation using a filter having characteristics different from those of the encoding unit filter used to perform the 3D reproducing operation. For example, the second 3D playback unit 630 may perform a 3D playback operation using an HRTF having coefficients different from those for the HRTF used by the encoding device.

A filter information storage unit 640 stores filter information regarding a filter used to perform 3D playback, for example, HRTF coefficient information. The second 3D rendering unit 630 may generate a combined 3D downmix using the filter information stored in the filter information storage unit 640 of the filter 640.

A filter information storage unit 640 of a filter 640 may store a plurality of pieces of filter information, respectively, a plurality of filters. In this case, one of the plurality of pieces of filter information may be selected according to the choice of the user or according to the functionality of the decoding device or sound quality requirements.

People of different races may have different ear structures. Thus, HRTF coefficients optimized for different people can differ from each other. The decoding device illustrated in FIG. 6, can generate a 3D downmix signal optimized for the user. In addition, the decoding apparatus illustrated in FIG. 6, can generate a 3D down-mix signal with 3D effects corresponding to the HRTF filter desired by the user, regardless of the type of HRTF provided by the 3D down-mix signal provider.

FIG. 7 is a block diagram of a 3D reproducing apparatus according to an embodiment of the present invention. According to FIG. 7, the 3D reproducing apparatus includes first and second field converting units 700 and 720 and a 3D reproducing unit 710. In order to perform the 3D playback operation in the predetermined area, the first and second area conversion units 700 and 720 can be respectively located before and after the 3D playback unit 710.

According to FIG. 7, the downmix input is converted to a downmix signal of the frequency domain by the first region transform unit 700. More specifically, the first region converting unit 700 can convert the downmix input to a DFT region downmix signal or an FFT region downmix signal by performing an DFT or FFT.

The 3D rendering unit 710 generates a multi-channel signal by applying spatial information to the frequency domain down-mix signal provided by the first region transform unit 700. Then, the 3D rendering unit 710 710 generates a 3D down-mix signal by filtering the multi-channel signal.

The 3D down-mix signal generated by the 3D reproducing unit 710 is converted to the 3D time-domain down-mix signal by the second region converting unit 720. More specifically, the second region transform unit 720 may perform an OTFT or IFFT on the 3D downmix signal generated by the 3D rendering unit 710.

During the conversion of the frequency domain of a 3D down-mix signal to a 3D time-domain down-mix signal, data loss or data distortion, such as spectral overlapping (sampling interference), may occur.

In order to generate a multi-channel signal and a 3D down-mix signal in the frequency domain, spatial information for each parametric range can be mapped to the frequency domain, and a plurality of filter coefficients can be converted to the frequency domain.

The 3D reproducing unit 710 can generate a 3D downmix signal by multiplying the downmix signal of the frequency domain provided by the first unit 700 for transforming the area, spatial information and filter coefficients.

The time-domain signal obtained by multiplying the down-mix signal, spatial information and a plurality of filter coefficients, which are all represented in the M-point frequency domain, has M real signals. To represent the downmix signal, spatial information, and a filter in the M-point frequency domain, an M-point DFT or M-point FFT can be performed.

Valid signals are signals that do not necessarily have a value of 0. For example, a total of x valid signals can be generated by obtaining x signals from an audio signal by sampling. Of the x valid signals, y valid signals can be padded with zeros. Then the number of valid signals is reduced to (x-y). After that, the signal with a valid signals and the signal with b valid signals are convolved, thereby obtaining a total of (a + b-1) valid signals.

Multiplying the down-mix signal, spatial information and filter coefficients in the M-point frequency domain can provide the same effect as the convolution of the down-mix signal, spatial information and filter coefficients in the time domain. A signal with (3 · M-2) real signals can be generated by converting a down-mix signal, spatial information and filter coefficients in the M-point frequency domain and convolution of the conversion results.

Therefore, the number of valid signals for the signal obtained by multiplying the down-mix signal, spatial information and filter coefficients in the frequency domain and converting the multiplication result to the time domain may differ from the number of valid signals for the signal obtained by convolution of the down-mix signal, spatial information and filter coefficients in time domain. As a result, superposition of the spectra can occur during the conversion of the 3D down-mix signal in the frequency domain into a time-domain signal.

To prevent overlapping spectra, the sum of the number of valid signals for the downmix signal in the time domain, the number of valid spatial information signals mapped to the frequency domain, and the number of filter coefficients should not be more than M. The number of valid spatial information signals mapped to the frequency region, can be determined by the number of points in the frequency domain. In other words, if the spatial information provided for each parametric range is mapped to an N-point frequency domain, the number of valid spatial information signals may be N.

According to FIG. 7, the first area transform unit 700 includes a first zero padding unit 701 and a first frequency domain transform unit 702. The third reproducing unit 710 includes a display unit 711, a time domain converting unit 712, a second zero padding unit 713, a second frequency domain converting unit 714, a multi-channel signal generating unit 715, a third zero padding unit 716, a third frequency domain converting unit 717 and a unit 718 generating a 3D down-mix signal.

The first zero-fill block 701 performs the zero-fill operation on the downmix signal with X samples in the time domain so that the number of samples of the downmix signal can be increased from X to M. The first frequency-domain transform block 702 converts the zeros-downmix signal to M- point signal of the frequency domain. Zero-padded downmix has M samples. Of the M samples of a zero-padded downmix signal, only X samples are valid signals.

A display unit 711 maps spatial information for each parametric range to an N-point region. Block 712 converting the time domain converts the spatial information obtained by the display performed by the block 711 display, in the time domain. The spatial information obtained by the transform performed by the time domain transform block 712 has N samples.

The second zero padding unit 713 performs a zero padding operation on spatial information with N samples in the time domain so that the number of spatial information samples can be increased from N to M. A second frequency domain transform unit 714 converts the zero-padded spatial information into an M-point frequency signal area. Spatial information supplemented with zeros has N samples. Of the N samples of zero-padded spatial information, only N samples are valid.

The multi-channel signal generating unit 715 generates a multi-channel signal by multiplying the down-mix signal provided by the first multi-channel signal generating unit 715 and spatial information provided by the second frequency domain transform unit 714. The multi-channel signal generated by the multi-channel signal generating unit 715 has M valid signals. On the other hand, a multi-channel signal obtained by time-domain convolution of the down-mix signal provided by the first time-domain transform unit 712 and spatial information provided by the second frequency-field transform unit 714 has (X + N-1) valid signals.

The third zeros block 716 can perform the zeros operation on the Y filter coefficients that are presented in the time domain so that the number of samples can be increased to M. The third frequency domain transform unit 717 converts the zeros added filter coefficients to an M-point frequency domain. Zero-padded filter coefficients have M samples. Of the M samples, only Y samples are valid signals.

The 3D downmix signal generating unit 718 generates a 3D downmix signal by multiplying the multi-channel signal generated by the multi-channel signal generating unit 715 and the plurality of filter coefficients provided by the third frequency domain transform unit 717. The 3D downmix signal generated by the 3D downmix signal generating unit 718 has M valid signals. On the other hand, the 3D down-mix signal obtained by convolution in the time domain of the multi-channel signal generated by the multi-channel signal generating unit 715 and the filter coefficients provided by the third frequency domain converting unit 717 have (X + N + Y-2) valid signals.

It is possible to prevent aliasing by setting the M dot frequency domain used by the first, second and third frequency domain transform blocks 702, 714 and 717 to satisfy the following equation: M≥ (X + N + Y-2). In other words, it is possible to prevent overlapping spectra by allowing the first, second, and third frequency domain transform blocks 702, 714, and 717 to perform an M-point DFT or M-point FFT that satisfies the following equation: M≥ (X + N + Y-2 )

Conversion to the frequency domain can be performed using a filter bank other than the DFT filter bank, the FFT filter bank, and the QMF bank. The generation of a 3D down-mix signal can be performed using an HRTF filter.

The number of valid spatial information signals may be tuned using a method other than the aforementioned method, or may be tuned using one of the aforementioned methods, which is the most efficient and requires the least calculation.

The superposition of spectra can occur not only during the conversion of the signal, coefficient or spatial information from the frequency domain to the time domain or vice versa, but also during the conversion of the signal, coefficient or spatial information from the QMF to the hybrid region or vice versa. The aforementioned methods of preventing spectral aliasing can also be used to prevent the aliasing of spectra from occurring during the conversion of a signal, coefficient or spatial information from a QMF region to a hybrid region or vice versa.

The spatial information used to generate the multi-channel signal or 3D down-mix signal may change. As a result of changes in spatial information, signal continuity disturbances in the form of noise in the output signal may occur.

Noise in the output signal can be reduced using a smoothing method that can prevent a rapid change in spatial information.

For example, when the first spatial information applied to the first frame is different from the second spatial information applied to the second frame, when the first frame and the second frame are adjacent to each other, it is very likely that a continuity gap will occur between the first and second frames.

In this case, the second spatial information can be compensated for the use of the first spatial information, or the first spatial information can be compensated for the use of the second spatial information so that the difference between the first spatial information and the second spatial information can be reduced, and noise caused gap continuity between the first and second frames, could be reduced. More specifically, at least one of the first spatial information and the second spatial information can be replaced by an average value of the first spatial information and the second spatial information, thereby reducing noise.

Noise is also likely to be generated due to a continuity gap between a pair of adjacent parametric ranges. For example, when the third spatial information corresponding to the first parametric range is different from the fourth spatial information corresponding to the second parametric range, when the first and second parametric ranges are adjacent to each other, discontinuity is likely to occur between the first and second parametric ranges.

In this case, the third spatial information can be compensated for the use of the fourth spatial information, or the fourth spatial information can be compensated for the use of the third spatial information so that the difference between the third spatial information and the fourth spatial information can be reduced, and the noise caused by the violation continuity between the first and second parametric ranges could be reduced. More specifically, at least one of the third spatial information and the fourth spatial information can be replaced by an average value of the third spatial information and the fourth spatial information, thereby reducing noise.

Noise caused by the disruption of continuity between a pair of adjacent frames or a pair of adjacent parametric ranges can be reduced using methods other than the above methods.

More specifically, each frame can be multiplied by a window, such as a Hanning window, and the overlay and add scheme can be applied to the multiplication results so that changes between frames can be reduced. Alternatively, the output signal to which the plurality of pieces of spatial information is applied may be smoothed so that changes between the plurality of frames of the output signal can be prevented.

Decorrelation between channels in the DFT region using spatial information, for example, ICC, can be configured as follows.

The degree of decorrelation can be adjusted by multiplying the coefficient of the signal input to the one-to-two (OTT) or two-to-three (TTT) block by a predetermined value. The predefined value can be determined by the following equation: (A + (l-A · A) ^ 0.5 · i), where A indicates the ICC value applied to the predefined range of the OTT or TTT block, and i indicates the imaginary part. The imaginary part can be positive or negative.

The predetermined value may correspond to a weighting coefficient according to the characteristics of the signal, for example, the signal energy level, the energy characteristics of each signal frequency, or the type of unit to which the ICC value A is applied. By introducing the weight coefficient, the degree of decorrelation can be further adjusted, and interframe smoothing or interpolation can be applied.

As described above with reference to FIG. 7, a 3D down-mix signal can be generated in the frequency domain using HRTF or a head-related impulse response (HRIR) that is converted to the frequency domain.

Alternatively, a 3D downmix signal may be generated by convolution of the HRIR and the downmix signal in the time domain. The 3D down-mix signal generated in the frequency domain can be left in the frequency domain without undergoing inverse transform of the domain.

For the convolution of the HRIR and the downmix signal in the time domain, a finite impulse response filter (FIR) or an infinite impulse response filter (IIR) can be used.

As described above, an encoding device or a decoding device according to an embodiment of the present invention can generate a 3D down-mix signal using the first method, which involves the use of HRTF in the frequency domain, or HRIR converted to the frequency domain, the second method, which involves convolution of HRIR in time areas, or a combination of the first and second methods.

FIG. 8 to 11 illustrate bit streams according to embodiments of the present invention.

According to FIG. 8, the bitstream includes a multi-channel decoding information field that contains information necessary for generating a multi-channel signal, a 3D playback information field that contains information necessary to generate a 3D down-mix signal, and a header field that contains header information necessary for use information included in the multi-channel decoding information field, and information included in the 3D playback information field. The bitstream may include only one or two of the multi-channel decoding information fields, 3D playback information fields, and header fields.

According to FIG. 9, a bit stream that contains side information necessary for a decoding operation may include a specific configuration header field that contains header information of the entire encoded message and a plurality of frame data fields that contain side information regarding the plurality of frames. More specifically, each of the frame data fields may include a frame header field that contains header information of a corresponding frame and a frame parameter data field that contains spatial information of a corresponding frame. Alternatively, each of the frame data fields may include only a frame parameter data field.

Each of the frame parameter data fields may include multiple modules, each module including parameter data and a flag. Modules are data sets, including parameter data, such as spatial information and other data, such as downmix gain and smoothing data, which are necessary to improve the sound quality of a signal.

If the module data regarding the information defined by the frame header regions is received without any additional flag, if the information defined by the frame header fields is further classified, or if the additional flag and data are received in connection with information not defined by the frame header, then the module data may do not include any flag.

Side information regarding the 3D downmix signal, for example, HRTF coefficient information, may be included in at least one of the specific configuration header fields, frame header fields, and frame parameter data fields.

According to FIG. 10, the bitstream may include a plurality of multi-channel decoding information fields that contain information necessary for generating multi-channel signals and a plurality of 3D playback information fields that contain information necessary for generating 3D down-mix signals.

Upon receiving the bitstream, the decoding device may use the multichannel decoding information fields or 3D playback information fields to perform the decoding operation, and skip any of the multichannel decoding information fields and 3D playback information fields fields not used in the decoding operation. In this case, it can be determined which of the multi-channel decoding information fields and the 3D playback information fields should be used to perform the decoding operation according to the type of signals to be reproduced.

In other words, in order to generate multi-channel signals, the decoding device can skip the fields of 3D playback information and read the information included in the fields of multi-channel decoding information. On the other hand, in order to generate 3D down-mix signals, the decoding device may skip the fields of multi-channel decoding information and read the information included in the fields of 3D playback information.

Ways to skip some of the many fields in the bitstream are described below.

First, field length information regarding the size in bits of a field may be included in the bitstream. In this case, the field can be skipped by skipping the number of bits corresponding to the size in the bits of the field. Field length information may be located at the beginning of the field.

Secondly, the sync word can be located at the end or beginning of the field. In this case, the field may be omitted by locating the field based on the location of the sync word.

Thirdly, if the field length is predetermined and fixed, then the field can be skipped by skipping a certain amount of data corresponding to the field length. The fixed field length information relative to the field length may be included in the bitstream or may be stored in a decoding device.

Fourth, one of a plurality of fields may be omitted using a combination of two or more of the aforementioned methods of skipping a field.

Field skip information, which is information necessary for field skipping, such as field length information, sync word or fixed field length information, can be included in one of the specific configuration header fields, frame header fields, and frame parameter data fields illustrated in FIG. 9, or may be included in a field other than those illustrated in FIG. 9.

For example, to generate multi-channel signals, the decoding device can skip the fields of 3D playback information with reference to the information of the field of length, a sync word or the fixed information of the field length located at the beginning of each of the fields of 3D playback information, and read the information included in the fields of information of multi-channel decoding.

On the other hand, to generate 3D down-mix signals, the decoding device can skip the fields of multi-channel decoding information with reference to the field length information, a sync word or fixed field length information located at the beginning of each of the fields of multi-channel decoding information, and read the information included in the fields 3D playback information.

The bitstream may include information indicating whether the data included in the bitstream is necessary for generating multi-channel signals or for generating 3D down-mix signals.

However, even if the bitstream does not contain any spatial information, such as CLD, but contains only the data (for example, HRTF filter coefficients) needed to generate the 3D down-mix signal, the multi-channel signal can be reproduced through decoding using the data necessary to generate a 3D down-mix signal, without requiring spatial information.

For example, a stereo parameter, which is spatial information regarding two channels, is obtained from a downmix signal. Then, the stereo parameter is converted into spatial information regarding a plurality of channels to be reproduced, and a multi-channel signal is generated using spatial information obtained by converting to a downmix signal.

On the other hand, even if the bitstream includes only the data necessary to generate the multi-channel signal, the downmix signal can be reproduced without requiring an additional decoding operation, or the 3D downmix signal can be reproduced by performing 3D processing on the downmix signal using an additional filter HRTF.

If the bitstream includes the data necessary to generate a multi-channel signal and the data necessary to generate a 3D down-mix signal, the user is able to decide whether to generate a multi-channel signal or a 3D down-mix signal.

Data skipping methods will now be described in detail with respect to the corresponding syntaxes.

Syntax 1 indicates a method for decoding an audio signal in blocks of frames.

[Syntax 1]

In syntax 1, OttData () and TttData () are modules that represent parameters (such as spatial information including CLD, ICC and CPC) needed to recover a multi-channel signal from a downmix signal, and SmgData (), TempShapeData (), Arbitrary-DownmixData () and ResidualData () are modules that provide the information necessary to improve sound quality by correcting signal distortions that may occur during the encoding operation.

For example, if a parameter, such as CLD, ICC or CPC, and the information included in the ArbitraryDownmixData () module are used during the decoding operation, then the SmgData () and TempShapeData () modules, which are located between the TttData () and ArbitraryDownmixData () modules may be unnecessary. Thus, it is efficient to skip the SmgData () and TempShapeData () modules.

A module skipping method according to an embodiment of the present invention will hereinafter be described in detail with respect to Syntax 2 below.

[Syntax 2]

As for Syntax 2, the SkipData () module can be located in front of the module to be skipped, and the bit size of the module to be skipped is defined in the SkipData () module as bsSkipBits.

In other words, assuming that the SmgData () and TempShapeData () modules should be skipped, and that the bit size of the combined SmgData () and TempShapeData () modules is 150, the SmgData () and TempShapeData () modules can be skipped by setting bsSkipBits to 150.

A module skipping method according to another embodiment of the present invention will hereinafter be described in detail with respect to Syntax 3.

[Syntax 3]

As for Syntax 3, an unnecessary module can be skipped using bsSkipSyncflag, which is a flag indicating whether to use a sync word, and bsSkipSyncword, which is a sync word that can be located at the end of a module to be skipped.

More specifically, if the bsSkipSyncflag flag is set so that a sync word can be used, then one or more modules between the bsSkipSyncflag flag and the bsSkipSyncword sync flag, i.e., SmgData () and TempShapeData (), can be omitted.

According to FIG. 11, the bitstream may include a multi-channel header field that includes header information necessary for reproducing a multi-channel signal, a 3D playback header field that includes header information necessary for reproducing a 3D downmix signal, and a plurality of multi-channel decoding information fields that include data needed to play a multi-channel signal.

To reproduce the multi-channel signal, the decoding device may skip the 3D playback header field and read data from the multi-channel header field and the multi-channel decoding information fields.

The method for skipping a field of a 3D playback header is the same as the method for skipping fields described above with respect to FIG. 10, and therefore, a detailed description thereof will be omitted.

To reproduce the 3D down-mix signal, the decoding device can read data from the multi-channel decoding information fields and the 3D playback header field. For example, a decoding device may generate a 3D downmix signal using the downmix signal included in the multi-channel decoding information field and HRTF coefficient information included in the 3D downmix signal.

In FIG. 12 is a block diagram of an encoding / decoding apparatus for processing an arbitrary downmix signal according to an embodiment of the present invention. According to FIG. 12, an arbitrary down-mix signal is a down-mix signal other than the down-mix signal generated by the multi-channel encoder 801 included in the encoding device 800. Detailed descriptions of the processes that are the same as the processes in the embodiment of FIG. one.

According to FIG. 12, an encoding device 800 includes a multi-channel encoder 801, a spatial information synthesis unit 802, and a comparison unit 803.

Multichannel encoder 801 down-converts the input multichannel signal to stereo or down-mix mono signal and generates basic spatial information needed to reconstruct the multichannel signal from the down-mix signal.

Comparison unit 803 compares the downmix signal with an arbitrary downmix signal and generates compensation information based on the comparison result. Compensation information is necessary to perform compensation for an arbitrary downmix signal so that an arbitrary downmix signal can be converted to be close to the downmix signal. The decoding apparatus can compensate for an arbitrary downmix signal using the compensation information and reconstruct the multi-channel signal using the compensated arbitrary downmix signal. The reconstructed multi-channel signal is more similar to the original input multi-channel signal than the multi-channel signal reconstructed from an arbitrary down-mix signal generated by the multi-channel encoder 801.

The compensation information may be the difference between the downmix signal and an arbitrary downmix signal. The decoding apparatus can compensate for an arbitrary downmix signal by adding to the arbitrary downmix signal a difference between the downmix signal and the arbitrary downmix signal.

The difference between the downmix signal and an arbitrary downmix signal can be a downmix gain that indicates the difference between the energy levels of the downmix signal and an arbitrary downmix signal.

The downmix gain can be determined for each frequency range, for each time / time segment and / or for each channel. For example, one part of the down-mix gain can be determined for each frequency range, and another part of the down-mix gain can be determined for each time segment.

The down-mix gain can be determined for each parametric range or for each frequency range optimized for an arbitrary down-mix signal. Parametric ranges are frequency intervals to which spatial information is applied for the type of parameter.

The difference between the energy levels of the downmix signal and an arbitrary downmix signal can be quantized. The resolution of the quantization levels for quantizing the difference between the energy levels of the downmix signal and the arbitrary downmix signal may be the same or different from the resolution of the quantization levels for quantizing CLD between the downmix signal and the arbitrary downmix signal. In addition, quantization of the difference between the energy levels of the downmix signal and an arbitrary downmix signal may include the use of all or some of the quantization levels to quantize the CLD between the downmix signal and the arbitrary downmix signal.

Since the resolution of the difference between the energy levels of the downmix signal and an arbitrary downmix signal is generally lower than the resolution of the CLD between the downmix signal and an arbitrary downmix signal, the resolution of quantization levels to quantize the difference between the energy levels of the downmix signal and an arbitrary downmix signal mixing may be negligible compared to the resolution of the quantization levels for quantizing CLD between a downmix signal and an arbitrary downmix signal.

Compensation information for performing compensation for an arbitrary downmix signal may be extension information including residual information that determines components of the input multi-channel signal that cannot be reconstructed using an arbitrary downmix signal or downmix gain. The decoding device can reconstruct the components of the input multi-channel signal, which cannot be reconstructed using an arbitrary down-mix signal or amplify the down-mix using extension information, thereby restoring a signal that is practically indistinguishable from the original input multi-channel signal.

Methods for generating extension information are as follows.

Multichannel encoder 801 may generate information regarding components of the input multichannel signal that the downmix signal lacks as first spreading information. A decoding apparatus can reconstruct a signal that is almost indistinguishable from the original input multi-channel signal by applying the first extension information to generate a multi-channel signal using a down-mix signal and basic spatial information.

Alternatively, multi-channel encoder 801 can reconstruct a multi-channel signal using a down-mix signal and basic spatial information, and generate a difference between the reconstructed multi-channel signal and the original input multi-channel signal as first spreading information.

The comparator 803 may generate, as the second extension information, information regarding components of the downmix signal that lack an arbitrary downmix signal, i.e., components of the downmix signal that cannot be compensated using the downmix gain. The decoding apparatus can recover a signal almost indistinguishable from a downmix signal using an arbitrary downmix signal and second extension information.

Extension information may be generated using residue encoding methods other than the above method.

The downmix gain and extension information can be used as compensation information. More specifically, the downmix gain and extension information can be obtained for the entire frequency range of the downmix signal and can be used together as compensation information. Alternatively, the downmix gain can be used as compensation information for one part of the frequency range of the downmix signal, and the extension information can be used as compensation information for another part of the frequency range of the downmix signal. For example, extension information can be used as compensation information for the low-frequency range of the down-mix signal, and amplification of down-mix can be used as compensation information for the high-frequency range of the down-mix signal.

Extension information regarding portions of the down-mix signal other than the low-frequency range of the down-mix signal, such as peaks or dips, which can significantly affect the sound quality, can also be used as compensation information.

The spatial information synthesis unit 802 synthesizes basic spatial information (e.g., CLD, CPC, ICC, and CTD) and compensation information, thereby generating spatial information. In other words, the spatial information that is transmitted to the decoding device may include basic spatial information, down-mix gain, and also the first and second extension information.

Spatial information may be included in the bitstream along with an arbitrary downmix signal, and the bitstream may be transmitted to the decoding device.

Extension information and an arbitrary downmix signal may be encoded using an audio coding method such as an AAC method, an MP3 method, or a BSAC method. Extension information and an arbitrary down-mix signal may be encoded using the same audio coding method or different audio coding methods.

If the extension information and an arbitrary down-mix signal are encoded using the same audio coding method, the decoding device can decode both the extension information and an arbitrary down-mix signal using a single audio decoding method. In this case, since an arbitrary down-mix signal can always be decoded, the extension information can also always be decoded. However, since an arbitrary downmix signal is generally input to the decoding device as a pulse code modulation (PCM) signal, the type of audio codec used to encode an arbitrary downmix signal cannot be easily identified, and thus the type of audio codec used to encode extension information also cannot be easily identified.

Therefore, the audio codec information regarding the type of audio codec used to encode an arbitrary downmix signal, and extension information can be inserted into the bitstream.

More specifically, the audio codec information may be inserted into a specific header field of the bitstream configuration. In this case, the decoding apparatus can extract audio codec information from a specific header field of the bitstream configuration and use the extracted audio codec information to decode an arbitrary downmix signal and extension information.

On the other hand, if an arbitrary down-mix signal and extension information are encoded using various audio coding methods, then the extension information may not be able to be decoded. In this case, since part of the extension information cannot be identified, no further decoding operation can be performed.

To solve this problem, the audio codec information regarding the types of audio codecs used, respectively, to encode an arbitrary downmix signal and extension information, can be inserted into a specific header field of the bitstream configuration. Then, the decoding device can read the audio codec information from a specific header field of the bitstream configuration and use the read information to decode the extension information. If the decoding device does not include any decoding unit capable of decoding the extension information, then decoding of the extension information cannot continue, and information following the extension information is read.

The audio codec information regarding the type of audio codec used to encode extension information may be represented by a syntax element included in a specific bitstream configuration header field. For example, audio codec information may be represented by a bsResidualCodecType element, which is a 4-bit syntax element, as shown in Table 1 below.

Table 1

Codec 0 AAS one MP3 2 BSAC 3 ... 15 Reserved

Extension information may include not only residual information, but also channel extension information. Channel extension information is information necessary for expanding a multi-channel signal obtained by decoding using spatial information into a multi-channel signal with a large number of channels. For example, the channel extension information may be information necessary to expand a signal with 5.1 channels or a signal with 7.1 channels into a signal with 9.1 channels.

Extension information may be included in the bitstream, and the bitstream may be transmitted to the decoding device. The decoding device may then compensate for the downmix signal or expand the multi-channel signal using the extension information. However, the decoding apparatus may skip the extension information, instead of extracting the extension information from the bitstream. For example, in the case of generating a multi-channel signal using a 3D down-mix signal included in a bit stream, or generating a 3D down-mix signal using a down-mix signal included in a bit stream, a decoding device may skip extension information.

The method for skipping extension information included in the bitstream may be the same as one of the methods for skipping fields described above with reference to FIG. 10.

For example, extension information may be skipped using at least one of the bit size information that is attached to the beginning of the bit stream including the extension information and indicates the bit size of the extension information, a sync word that is attached to the beginning or end a field including extension information and information of a fixed bit size that indicates a fixed bit size of the extension information. Bit size information, a sync word, and fixed bit size information may all be included in the bit stream. Information of a fixed bit size may also be stored in a decoding device.

According to FIG. 12, the decoding unit 810 includes a downmix compensation unit 811, a 3D playback unit 815, and a multi-channel decoder 816.

The downmix compensation unit 811 performs compensation for an arbitrary downmix signal using the compensation information included in the spatial information, for example, using the downmix gain or the extension information.

The 3D reproduction unit 815 generates a 3D downmix signal of the decoder by performing a 3D reproduction operation on the compensated downmix signal. Multi-channel decoder 816 generates a 3D multi-channel signal using a compensated down-mix signal and basic spatial information that is included in the spatial information.

The downmix compensation unit 811 may perform compensation for an arbitrary downmix signal as follows.

If the compensation information is a downmix gain, the downmix compensation unit 811 811 compensates for the energy level of the arbitrary downmix signal using the downmix gain so that the arbitrary downmix signal can be converted to a signal similar to the downmix signal.

If the compensation information is the second extension information, then the downmix compensation unit 811 can compensate for components that lack an arbitrary downmix signal using the second extension information.

Multi-channel decoder 816 can generate a multi-channel signal by sequentially applying the pre-matrix M1, the mixed matrix M2, and the post-matrix M3 to the downmix signal. In this case, the second extension information may be used to perform compensation for the downmix signal by applying the mixed matrix M2 to the downmix signal. In other words, the second extension information can be used to perform compensation for the downmix signal to which the pre-matrix M1 has already been applied.

As described above, each of the plurality of channels can be selectively compensated by applying extension information to the generation of a multi-channel signal. For example, if the extension information is applied to the center channel of the mixed matrix M2, then the left and right channel components of the downmix signal can be compensated by the extension information. If the extension information is applied to the left channel of the mixed matrix M2, the left channel component of the down-mix signal can be compensated by the extension information.

The downmix gain and extension information can be used as compensation information. For example, the low-frequency range of an arbitrary down-mix signal can be compensated for using the extension information, and the high-frequency range of an arbitrary down-mix signal can be compensated for using the down-mix gain. In addition, portions of an arbitrary down-mix signal other than the low-frequency range of an arbitrary down-mix signal, such as peaks or dips that can significantly affect sound quality, can also be compensated for using the extension information. Information regarding the part to be compensated by the extension information may be included in the bit stream. Information indicating whether the down-mix signal included in the bit stream is an arbitrary down-mix signal or not, and information indicating whether the bit stream includes compensation information can be included in the bit stream.

In order to prevent the down-mix signal generated by the coding unit 800 from being limited, the down-mix signal can be divided into a predetermined gain. The predefined gain can have a static value or a dynamic value.

The downmix compensation unit 811 can restore the original downmix signal by performing compensation for the downmix signal that is attenuated to prevent clipping using a predetermined gain.

An arbitrary downmix signal compensated by the downmix compensation unit 811 can be easily reproduced. Alternatively, an arbitrary down-mix signal to be compensated may be input to the 3D reproducing unit 815 and may be converted to a 3D decoder down-mixing signal by the 3D reproducing unit 815.

According to FIG. 12, the downmix compensation unit 811 includes a first region converter 812, a compensation processor 813, and a second region converter 814.

The first region converter 812 converts an arbitrary downmix signal to a predetermined region. Compensation processor 813 compensates for an arbitrary downmix signal in a predetermined area using compensation information, such as downmix gain or extension information.

Compensation of an arbitrary down-mix signal can be performed in the QMF / hybrid region. To this end, the first domain transformer 812 can perform QMF / hybrid analysis on an arbitrary downmix signal. The first region converter 812 may convert the region of an arbitrary downmix signal to a region other than the QMF / hybrid region, for example, a frequency region, such as a DFT or FFT region. Compensation of an arbitrary down-mix signal can also be performed in a region other than the QMF / hybrid region, for example, in the frequency domain or time domain.

The second region converter 814 converts the region of the compensated arbitrary downmix signal to the same region as the original arbitrary downmix signal. More specifically, the second region converter 814 converts the region of the compensated arbitrary downmix signal to the same region as the original arbitrary downmix signal by inversely performing the region transform operation performed by the first region converter 812.

For example, the second region converter 814 may convert the compensated arbitrary downmix signal into a time domain signal by performing QMF / hybrid synthesis on the compensated arbitrary downmix signal. In addition, the second region converter 814 may perform DFT or IFFT on a compensated arbitrary downmix signal.

The 3D reproducing unit 815, like the 3D reproducing unit 710 illustrated in FIG. 7, may perform 3D reproduction operation on a compensated arbitrary down-mix signal in the frequency domain, QMF / hybrid region, or time domain. For this, the 3D rendering unit 815 may include an area converter (not shown). The region converter converts the region of the compensated arbitrary downmix signal to the region in which the 3D reproduction operation is to be performed, or converts the region of the signal obtained by the 3D reproduction operation.

The area in which the compensation processor 813 performs compensation for an arbitrary downmix signal may be the same or different from the region in which the 3D reproducing unit 815 performs a 3D reproduction operation on the compensated arbitrary downmix signal.

In FIG. 13 is a block diagram of a downmix / 3D playback compensation unit 820 according to an embodiment of the present invention. According to FIG. 13, the downmix / 3D playback compensation unit 820 includes a first area converter 821, a second region converter 822, a compensation / 3D playback processor 823, and a third region converter 824.

The downmix / 3D reproduction compensation unit 820 can perform both the compensation operation and the 3D reproduction operation on an arbitrary downmix signal in a single area, thereby reducing the computation amount of the decoding device.

More specifically, the first region converter 821 converts the region of an arbitrary downmix signal into a first region in which a compensation operation and a 3D reproduction operation are to be performed. The second region converter 822 converts the spatial information, including the basic spatial information needed to generate the multi-channel signal, and the compensation information necessary to perform the compensation for an arbitrary downmix signal, so that the spatial information can be applied in the first region. Compensation information may include at least one of downmix gain and extension information.

For example, the second domain transformer 822 may map the compensation information corresponding to the parametric range in the QMF / hybrid region to the frequency range so that the compensation information can be easily applied in the frequency domain.

The first region may be a frequency domain, such as a DFT or FFT region, a QMF / hybrid region, or a time domain. Alternatively, the first region may be a region other than those indicated above.

There may be a time delay during the conversion of compensation information. To solve this problem, the second region converter 822 can perform a time delay compensation operation so that the time delay between the region of compensation information and the first region can be compensated.

The compensation / 3D playback processor 823 performs a compensation operation on an arbitrary down-mix signal in the first region using the transformed spatial information, and then performs a 3D playback operation on the signal obtained by the compensation operation. The compensation / 3D playback processor 23 may perform the compensation operation and the 3D playback operation in an order different from that described above.

Compensation / 3D playback processor 823 may perform a compensation operation and a 3D playback operation on an arbitrary downmix signal at the same time. For example, compensation / 3D playback processor 823 can generate a compensated 3D down-mix signal by performing a 3D rep operation on an arbitrary down-mix signal in the first region using a new filter coefficient, which is a combination of compensation information and an existing filter coefficient, typically used in the operation 3D playback.

The third region converter 824 converts the 3D down-mix signal generated by the compensation / 3D reproduction processor 823 in the frequency domain.

In FIG. 14 is a block diagram of a decoding apparatus 900 for processing a compatible downmix signal according to an embodiment of the present invention. According to FIG. 14, the decoding apparatus 900 includes a first multi-channel decoder 910, a downmix compatibility processing unit 920, a second multi-channel decoder 930, and a 3D playback unit 940. Detailed descriptions of decoding processes similar to those in the embodiment of FIG. 1 will be omitted.

A compatible downmix signal is a downmix signal that can be decoded by two or more multi-channel decoders. In other words, a compatible downmix signal is a downmix signal that is initially optimized for a predetermined multi-channel decoder, and which can then be converted to a signal optimized for a multi-channel decoder other than a predetermined multi-channel decoder, through a compatibility processing operation.

Referring to FIG. 14, suppose that the input compatible downmix signal is optimized for the first multi-channel decoder 910. In order for the second multi-channel decoder 930 to decode the input compatible downmix signal, the downmix compatibility processing unit 920 may perform a compatibility processing operation on the input compatible downmix signal so so that the input compatible down-mix signal can be converted to a signal optimized for the second m ogokanalnogo decoder 930. The first multichannel decoder 910 generates a first multichannel signal by decoding the input compatible down-mix signal. The first multi-channel decoder 910 can generate a multi-channel signal by decoding, simply using an input compatible down-mix signal without requiring spatial information.

The second multi-channel decoder 930 generates a second multi-channel signal using the down-mix signal obtained by the compatibility processing operation performed by the down-mix compatibility processing unit 920. The 3D reproduction unit 940 may generate a 3D downmix signal of the decoder by performing a 3D reproduction operation on the downmix signal obtained by the compatibility processing operation performed by the downmix compatibility processing unit 920.

A compatible downmix signal optimized for a predetermined multichannel decoder can be converted to a downmix signal optimized for a multichannel decoder other than a predetermined multichannel decoder using compatibility information such as an inversion matrix. For example, when there are first and second multi-channel encoders using different encoding methods, and the first and second multi-channel decoders using different encoding / decoding methods, the encoding device can apply a matrix to the down-mix signal generated by the first multi-channel encoder, thereby generating a compatible signal downmix, which is optimized for a second multi-channel decoder. Then, the decoding apparatus can apply the inversion matrix to the compatible downmix signal generated by the encoding device, thereby generating a compatible downmix signal that is optimized for the first multi-channel decoder.

According to FIG. 14, a downmix compatibility processing unit 920 may perform a compatibility processing operation on an input compatible downmix signal using an inversion matrix, thereby generating a downmix signal that is optimized for the second multi-channel decoder 930.

Information regarding the inversion matrix used by the downmix compatibility processing unit 920 may be stored in the decoding apparatus 900 in advance, or may be included in the input bit stream transmitted by the encoding apparatus. In addition, information indicating whether the down-mix signal included in the input bitstream is an arbitrary down-mix signal or a compatible down-mix signal can be included in the input bit stream.

According to FIG. 14, the downmix compatibility processing unit 920 includes a first region converter 921, a compatibility processor 922, and a second region converter 923.

The first region converter 921 converts the input compatible downmix signal region to a predetermined region, and the compatibility processor 922 performs a compatibility processing operation using compatibility information such as an inversion matrix so that the input compatible downmix signal in the predetermined region can be converted to a signal optimized for a second multi-channel decoder 930.

Compatibility processor 922 may perform a compatibility processing operation in the QMF / hybrid area. To this end, the first area converter 921 may perform QMF / hybrid analysis on an input compatible downmix signal. In addition, the first region converter 921 may convert the region of the input compatible downmix signal to a region other than the QMF / hybrid region, for example, a frequency region, such as a DFT region or an FFT region, and the compatibility processor 922 may perform a compatibility processing operation in a region other than than the QMF / hybrid region, for example, in the frequency domain or time domain.

The second region converter 923 converts the region of the compatible downmix signal obtained by the compatibility processing operation. More specifically, the second region converter 923 can convert the region of the compatible downmix signal obtained by the compatibility processing operation to the same region as the region of the original input compatible downmix signal by inverting the region transform operation performed by the first region converter 921.

For example, the second region converter 923 can convert the compatible downmix signal obtained by the compatibility processing operation to a time domain signal by performing QMF / hybrid synthesis on the compatible downmix signal obtained by the compatibility processing operation. Alternatively, the second region converter 923 may perform TFT or TFT on a compatible downmix signal obtained by the compatibility processing operation.

The 3D reproducing unit 940 may perform a 3D reproducing operation on a compatible downmix signal obtained by the compatibility processing operation in the frequency domain, QMF / hybrid region, or time domain. For this, the 3D rendering unit 940 may include an area converter (not shown). The region converter converts the region of the input compatible downmix signal to the region in which the 3D reproduction operation is to be performed, or converts the region of the signal obtained by the 3D reproduction operation.

The area in which the compatibility processor 922 performs the compatibility processing operation may be the same or different from the area in which the 3D playback unit 940 performs the 3D playback operation.

In FIG. 15 is a flowchart of a downmix / 3D playback compatibility processing unit 950 according to an embodiment of the present invention. According to FIG. 15, the downmix / 3D playback compatibility processing unit 950 includes a first area converter 951, a second region converter 952, a compatibility / 3D playback processor 953, and a third region converter 953.

The downmix / 3D playback compatibility processing unit 950 performs the compatibility processing operation and the 3D playback operation in a single area, thereby reducing the computation amount of the decoding device.

The first region converter 951 converts the input compatible down-mix signal into a first region in which a compatibility processing operation and a 3D playback operation are to be performed. The second region converter 952 converts spatial information and compatibility information, such as an inversion matrix, so that spatial information and compatibility information can be applied in the first region.

For example, the second domain converter 952 maps the inversion matrix corresponding to the parametric range in the QMF / hybrid region to the frequency domain so that the inversion matrix can be easily applied in the frequency domain.

The first region may be a frequency domain, such as a DFT region or an FFT region, a QMF / hybrid region, or a time domain. Alternatively, the first region may be a region other than those indicated above.

There may be a time delay during the conversion of spatial and compatibility information.

To solve this problem, the second region converter 952 may perform a time delay compensation operation so that the time delay between the spatial information and compensation information region and the first region can be compensated.

A compatibility / 3D playback processor 953 performs a compatibility processing operation on an input compatible downmix signal in a first area using the converted compatibility information, and then performs a 3D playback operation on a compatible downmix signal obtained by a compatibility processing operation. The compatibility / 3D playback processor 953 may perform the compatibility processing operation and the 3D playback operation in an order other than that described above.

Compatibility / 3D playback processor 953 may perform a compatibility processing operation and a 3D playback operation on an input compatible downmix signal at the same time. For example, compatibility / 3D playback processor 953 can generate a 3D downmix signal by performing a 3D reproduction operation on an input compatible downmix signal in the first region using a new filter coefficient, which is a combination of compatibility information and an existing filter coefficient, typically used in the operation 3D playback.

The third region converter 954 converts the 3D region of the downmix signal generated by the compatibility / 3D playback processor 953 into a frequency region.

In FIG. 16 is a block diagram of a decoding apparatus for compensating cross-coupling according to an embodiment of the present invention. According to FIG. 16, the decoding apparatus includes a bit decompression unit 960, a downmix decoder 970, a 3D playback unit 980, and a cross-coupling compensation unit 990. Detailed descriptions of decoding processes similar to those described above for the embodiment of FIG. 1 will be omitted.

The 3D downmix signal generated by the 3D reproducing unit 980 can be reproduced by the headphone. However, when the 3D down-mix signal is reproduced by speakers that are remote from the user, cross-channel cross-coupling may occur.

Therefore, the decoding device may include a cross-coupling compensation unit 990 that performs a cross-coupling compensation operation on the 3D downmix signal.

The decoding device may perform a sound field processing operation.

The sound field information used in the sound field processing operation, that is, information identifying the place where the 3D down-mix signal is to be played, may be included in the input bit stream transmitted by the encoding device, or may be selected by a decoding device.

The input bitstream may include reverberation time information. The filter used in the sound field processing operation may be controlled according to reverberation time information.

The sound field processing operation may be performed differently for the initial part and the final part of the reverb. For example, the initial part can be processed using the FIR filter, and the last part of the reverb can be processed using the IIR filter.

More specifically, a sound field processing operation can be performed on the initial part by performing a convolution operation in the time domain using the FIR filter or by performing a multiplication operation in the frequency domain and converting the result of the multiplication operation into the time domain. The sound field processing operation may be performed on the last part of the reverb in the time domain.

The present invention can be implemented as computer readable code recorded on a computer readable recording medium. The computer-readable recording medium may be any type of recording device in which data is stored in computer-readable form. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and carrier wave (e.g., data transmission over the Internet). The computer-readable recording medium may be distributed across a plurality of computer systems connected to the network so that the computer-readable code can be written to and executed from them in a decentralized manner. Functional programs, code, and code segments necessary for implementing the present invention can be readily interpreted by those skilled in the art.

As described above, according to the present invention, it is possible to efficiently encode multi-channel signals with 3D effects and adaptively restore and reproduce audio signals with optimal sound quality in accordance with the characteristics of the playback environment.

Industrial applicability

Other implementations are within the scope of the following claims. For example, grouping, data coding and entropy coding according to the present invention can be applied to various applications and various products. Data carriers storing data to which an aspect of the present invention is applied are included in the scope of the present invention.

Claims (11)

1. A method for decoding a signal, comprising
receiving three-dimensional (3D) signal down-mixing and spatial information;
decoding of a 3D down-mix signal;
extracting filter related information from the input bitstream;
removing the 3D effect from the 3D downmix signal by performing a 3D playback operation on the 3D downmix signal using a filter inverse to the filter used to generate the 3D downmix signal based on the filter related information; and
generating a multi-channel signal using spatial information and a down-mix signal from which the 3D effect is removed. 2. The method according to claim 1, wherein the inverse filter is a function of the inverse head-related transfer function (HRTF) used to generate the 3D downmix signal. 3. The method according to claim 2, in which the information related to the filter is at least one of information related to the HRTF coefficients and information related to the coefficients of the inverse HRTF function. 4. The method according to claim 2, further comprising receiving at least one of information indicating whether the input bitstream includes information related to the filter and information indicating whether the information related to the filter is information related to coefficients of the inverse HRTF function. 5. The method according to claim 1, in which the 3D playback operation is performed in one of the discrete Fourier transform (DFT) region, the fast Fourier transform region (FFT), the quadrature mirror filter (QMF) / hybrid region, and the time domain. 6. A computer-readable recording medium containing a computer program for executing a decoding method according to any one of claims 1 to 5. 7. A device for decoding a signal containing
a bit decompression unit receiving a three-dimensional (3D) down-mix signal and spatial information;
a downmix decoder decoding a 3D downmix signal;
a 3D reproducing unit removing the 3D effect from the 3D downmix signal by performing a 3D reproduction operation on the 3D downmix signal using a filter inverse to the filter used to generate the 3D downmix signal based on information related to the filter; and
a multi-channel decoder generating a multi-channel signal using spatial information and a down-mix signal from which the 3D effect is removed. 8. The device according to claim 7, in which the inverse filter is a function of the inverse associated with the head of the transfer function (HRTF) used to generate a 3D signal down-mixing. 9. The decoding device of claim 8, in which the information related to the filter is at least one of information related to the HRTF coefficients and information related to the coefficients of the inverse HRTF function. 10. The decoding apparatus of claim 8, wherein the bit decompression unit receives at least one of information indicating whether the input bit stream includes information related to the filter and information indicating whether the information related to the filter information related to the coefficients of the inverse HRTF function. 11. The decoding apparatus of claim 9, wherein the 3D playback operation is performed in one of a discrete Fourier transform (DFT) region, a fast Fourier transform (FFT) region, a quadrature mirror filter (QMF) / hybrid region, and a time domain.

Images