RU2406165C2 - Methods and devices for coding and decoding object-based audio signals - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: audio decoding method involves receiving a step-down mixing signal which is obtained by step-down mixing of multiple object signals, and additional object information containing object-based additional information and control information, extraction of metadata containing description of the object signal from the object-based additional information, pre-processing the position or level of the object signal and generation of a multichannel audio signal by using the pre-processed step-down mixing signal, spatial parametre information and control information.

EFFECT: method and device for coding and decoding audio for efficient processing of object-based audio signals.

11 cl, 33 dwg

Description

FIELD OF THE INVENTION

The present invention relates to an audio encoding method and apparatus, and an audio decoding method and apparatus in which object-based audio signals can be efficiently processed by performing encoding and decoding operations.

State of the art

In general, in the methods of encoding and decoding multi-channel audio, a certain number of channel signals of a multi-channel signal are mixed downward to a smaller number of channel signals, additional information regarding the original channel signals is transmitted, and a multi-channel signal having as many channels as the original multi-channel signal is restored.

The methods of object-based encoding and decoding of audio are basically similar to the methods of encoding and decoding multichannel audio in terms of down-mixing of several audio sources into fewer signals of audio sources and transmitting additional information relative to the original audio sources. However, in the methods of object-based encoding and decoding, the audio signals of the object, which are the basic elements (for example, the sound of a musical instrument or the voice of a person) of a channel signal are treated in the same way as channel signals in the methods of encoding and decoding multi-channel audio, and can thus be encoded.

In other words, in methods of object-based encoding and decoding of audio, it is believed that the signals of an object are entities to be encoded. In this regard, the methods of object-based encoding and decoding of audio are different from the methods of multi-channel encoding and decoding of audio, in which the operation of multi-channel encoding of audio is performed simply based on inter-channel information, regardless of the number of elements of the channel signal to be encoded.

Description of the invention

Technical challenge

The present invention provides an audio encoding method and apparatus, and an audio decoding method and apparatus in which audio signals can be encoded or decoded, so that audio signals can be applied to various environments.

Technical solution

According to an aspect of the present invention, there is provided an audio decoding method including receiving a downmix signal and object-based additional information, wherein the downmix signal is obtained by downmixing a plurality of object signals; extracting metadata from object-based additional information and displaying object-related information regarding object signals based on metadata. According to another aspect of the present invention, there is provided an audio encoding method comprising generating a downmix signal by performing downmixing a plurality of object signals; generating object-based additional information by extracting the information related to the object from the signals of the object, and inserting metadata for rendering the information related to the object into the object-based additional information.

According to another aspect of the present invention, there is provided an audio decoding apparatus including: a demultiplexer configured to extract a downmix signal and object-based additional information from an input audio signal, wherein the downmix signal is obtained by downmixing a plurality of object signals; a transcoder configured to extract metadata from object-based additional information; and a renderer that displays information related to the object regarding the signals of the object based on metadata.

According to another aspect of the present invention, there is provided a computer-readable recording medium having a computer program recorded thereon for executing an audio decoding method, the audio decoding method including: receiving a downmix signal and object-based additional information, wherein the downmix signal is obtained by performing down mixing of a plurality of object signals; extract metadata from object-based additional information; and displaying object related information regarding object signals based on metadata.

According to another aspect of the present invention, there is provided a computer-readable recording medium having a computer program recorded thereon for executing an audio encoding method, the audio encoding method including: generating a downmix signal by performing downmix of a plurality of object signals; generating object-based additional information by extracting information related to the object from the signals of the object; and embedding metadata in the object-based additional information, the metadata representing object-related information.

Brief Description of the Drawings

Figure 1 illustrates a block diagram of a typical system of object-based audio encoding / decoding;

2 illustrates a block diagram of an audio decoding apparatus according to a first embodiment of the present invention;

3 illustrates a block diagram of an audio decoding apparatus according to a second embodiment of the present invention;

4 illustrates a block diagram of an audio decoding apparatus according to a third embodiment of the present invention;

FIG. 5 illustrates a block diagram of an arbitrary down-mix gain (ADG) module that can be used in the audio decoding apparatus shown in FIG. 4;

6 illustrates a block diagram of an audio decoding apparatus according to a fourth embodiment of the present invention;

7 illustrates a block diagram of an audio decoding apparatus according to a fifth embodiment of the present invention;

Fig. 8 illustrates a block diagram of an audio decoding apparatus according to a sixth embodiment of the present invention;

9 illustrates a block diagram of an audio decoding apparatus according to a seventh embodiment of the present invention;

10 illustrates a block diagram of an audio decoding apparatus according to an eighth embodiment of the present invention;

11 and 12 illustrate diagrams for explaining the operation of a transcoder;

13-16 illustrate diagrams for explaining a configuration of object-based additional information;

17-22 illustrate diagrams for explaining combining a plurality of portions of object-based additional information into one portion of additional information;

23-27 illustrate diagrams for explaining a preprocessing operation; and

28-33 are diagrams illustrating a case of combining a plurality of decoded bitstreams with object-based signals into a single bitstream.

The best embodiment of the invention

The present invention is hereinafter described in detail with reference to the accompanying drawings, in which exemplary embodiments of the invention are shown.

The audio encoding method and apparatus and the audio decoding method and apparatus according to the present invention may be applied to object-based audio processing operations, but the present invention is not limited thereto. In other words, the audio encoding method and apparatus and the audio decoding method and apparatus can be used in various signal processing operations, except for object-based audio processing operations.

Figure 1 illustrates a block diagram of a typical system of object-based encoding / decoding audio. Typically, the audio signals input to an object-based audio encoding device do not correspond to the channels of a multi-channel signal, but are independent object signals. In this regard, the object-based audio encoding device is different from the multi-channel audio encoding device into which the channel signals of the multi-channel signal are input.

For example, channel signals, such as the front left channel signal and the front right channel signal of a 5.1 channel signal, can be input into a multi-channel audio signal, while object signals, such as a human voice or the sound of a musical instrument (e.g., the sound of a violin or piano), which are smaller entities than channel signals, can be entered into an object-based audio encoding device.

Referring to FIG. 1, an object-based audio encoding / decoding system includes an object-based audio encoding device and an object-based audio decoding device. An object-based audio encoding apparatus includes an object encoder 100, and an object-based audio decoding apparatus includes an object decoder 111 and a mixer / renderer 113.

An object encoder 100 receives N object signals and generates an object-based downmix signal with one or more channels and additional information including a number of pieces of information extracted from N object signals, such as energy difference information, phase difference information, and correlation information. The additional information and the object-based downmix signal are combined into a single bitstream, and the bitstream is transmitted to an object-based decoding device.

The additional information may include a flag indicating whether to perform channel-based audio encoding or object-based audio encoding, and thus it may be determined whether to perform channel-based audio encoding or object-based audio encoding based on the optional information. Additional information may also include energy information, grouping information, silent period information, downmix gain information and delay information regarding object signals.

The additional information and the object-based downmix signal can be combined into a single bit stream, and a single bit stream can be transmitted to an object-based audio decoding device.

An object decoder 111 receives an object-based downmix signal and additional information from an object-based audio encoding device and reconstructs object signals having properties similar to properties of N object signals based on an object-based downmix signal and additional information. The object signals generated by the object decoder 111 have not yet been allocated any position in the multi-channel space. Thus, the mixer / renderer 113 distributes each object signal generated by the object decoder 111 to a predetermined position in the multi-channel space and determines the signal levels of the object, so that the object signals can be reproduced from the corresponding positions indicated by the mixer / renderer 113, respectively , with corresponding levels defined by mixer / renderer 113. Control information regarding each object signal generated by the object decoder 111 may zmenyatsya in time, and thus, the spatial positions and the levels of the object signals generated by the object decoder 111 may vary in accordance with the control information.

FIG. 2 illustrates a block diagram of an audio decoding apparatus 120 according to a first embodiment of the present invention. As shown in FIG. 2, the audio decoding apparatus 120 may be able to perform adaptive decoding by analyzing control information.

As shown in FIG. 2, the audio decoding apparatus 120 includes an object decoder 121, a mixer / renderer 123, and a parameter converter 125. Audio decoding apparatus 120 may also include a demultiplexer (not shown) that extracts a downmix signal and additional information from a bitstream input therein, and it will be applied to all audio decoding apparatuses according to other embodiments of the present invention.

An object decoder 121 generates a number of object signals based on a downmix signal and modified additional information provided by the parameter converter 125. The mixer / renderer 123 distributes each object signal generated by the object decoder 121 to a predetermined position in the multi-channel space and determines the signal levels of the object generated by the object decoder 121 in accordance with the control information. The parameter converter 125 generates modified additional information by combining the additional information and control information. Then, the parameter converter 125 transmits the modified additional information to the object decoder 121.

An object decoder 121 may be able to perform adaptive decoding by analyzing control information in the modified supplemental information.

For example, if the control information indicates that the first object signal and the second object signal are distributed at the same position in the multi-channel space and have the same level, a typical audio decoding device can decode the first and second object signals separately and then place them in the multi-channel space when help with the blending / rendering operation.

On the other hand, the object decoder 121 of the audio decoding apparatus 120 recognizes from the control information in the modified additional information that the first and second object signals are distributed at the same position in the multi-channel space and have the same level as if they were the only sound source. Therefore, the decoder 121 objects decodes the first and second signals of the object, considering them as the only sound source without decoding them separately. As a result, decoding complexity is reduced. In addition, due to the reduction in the number of sound sources that need to be processed, the complexity of mixing / rendering is also reduced.

Audio decoding apparatus 120 can be effectively used when the number of object signals is greater than the number of output channels, since there is a high probability that a plurality of object signals will be distributed at the same spatial position.

Alternatively, the audio decoding apparatus 120 may be used when the first object signal and the second object signal are distributed at the same position in multi-channel space, but at different levels. In this case, the audio decoding apparatus 120 decodes the first and second object signals, treating the first and second object signals as a single signal, instead of decoding the first and second object signals separately and transmitting the decoded first and second object signals to the mixer / renderer 123. More specifically, the decoder 121 objects can receive information regarding the difference between the levels of the first and second signals of the object from the control information in the modified additional information, and decode rvy and second object signals based on the received information. As a result, even if the first and second signals of the object have different levels, the first and second signals of the object can be decoded as if they were the only sound source.

As an alternative, the object decoder 121 may adjust the signal levels of the object generated by the object decoder 121 in accordance with the control information. Then, object decoder 121 may decode object signals whose levels are adjustable. Therefore, the mixer / renderer 123 does not need to adjust the levels of decoded object signals provided by the object decoder 121, but it simply arranges the decoded object signals represented by the object decoder 121 in multi-channel space. Briefly, since the object decoder 121 adjusts the object signal levels generated by the object decoder 121 in accordance with the control information, the mixer / renderer 123 can easily arrange the object signals generated by the object decoder 121 in a multi-channel space without the need for additional adjustment of the object signal levels generated by the decoder 121 objects. Therefore, the complexity of blending / rendering can be reduced.

According to the embodiment of FIG. 2, an object decoder of the audio decoding apparatus 120 can adaptively perform a decoding operation by analyzing control information, thereby reducing the decoding complexity and the complexity of mixing / rendering. A combination of the above methods performed by the audio decoding apparatus 120 may be used.

FIG. 3 illustrates a block diagram of an audio decoding apparatus 130 according to a second embodiment of the present invention. As shown in FIG. 3, the audio decoding apparatus 130 includes an object decoder 131 and a mixer / renderer 133. The audio decoding device 130 is characterized by providing additional information not only to the object decoder 131, but also to the mixer / renderer 133.

An audio decoding apparatus 130 can efficiently perform a decoding operation even when there is an object signal corresponding to a period of silence. For example, the second to fourth signals of an object may correspond to a period of playing music during which a musical instrument is played, and the first signal of an object may correspond to a muting period during which only background music is played, and the first signal of an object may correspond to a period of silence during which it is played accompaniment. In this case, information indicating which of the plurality of object signals corresponds to the silence period may be included in additional information, and additional information may be provided to a mixer / renderer 133, as well as to an object decoder 131.

An object decoder 131 can minimize the decoding complexity by undecoding an object signal corresponding to a silence period. An object decoder 131 sets an object signal corresponding to a value of 0, and transmits an object signal level to a mixer / renderer 133. In general, object signals having a value of 0 are treated in the same way as object signals having a value other than 0, and thus thus undergoes blending / rendering operations.

On the other hand, the audio decoding apparatus 130 transmits additional information including information indicating which of the plurality of object signals corresponds to the silence period to the mixer / renderer 133 and can thus prevent the object signal corresponding to the silence period, undergoes the mixing / rendering operation performed by the mixer / renderer 133. Therefore, the audio decoding apparatus 130 can prevent an undesired increase in the complexity of the mixing / rendering.

FIG. 4 illustrates a block diagram of an audio decoding apparatus 140 according to a third embodiment of the present invention. As shown in FIG. 4, the audio decoding apparatus 140 uses a multi-channel decoder 141 instead of an object decoder and mixer / renderer and decodes a number of object signals after the object signals are properly arranged in the multi-channel space.

More specifically, the audio decoding apparatus 140 includes a multi-channel decoder 141 and a parameter converter 145. The multi-channel decoder 141 generates a multi-channel signal, the object signals of which have already been arranged in multi-channel space, based on the down-mix signal and spatial parameter information, which is channel-based parameter information provided by the parameter converter 145. The parameter converter 145 analyzes additional information and control information transmitted by an audio encoding device (not shown), and generates spatial parameter information based on the analysis result. More specifically, the parameter transformer 145 generates spatial parameter information by combining additional information and control information, which includes playback setting information and mixing information. Those. a parameter converter 145 converts a combination of additional information and control information into spatial data corresponding to a one-to-two (OTT) block or a two-to-three block (TTT).

The audio decoding apparatus 140 may perform a multi-channel decoding operation in which an object-based decoding operation and a mixing / rendering operation are integrated, and may thereby skip decoding each object signal. Therefore, the complexity of decoding and / or blending / rendering can be reduced.

For example, when there are 10 object signals and a multi-channel signal derived from 10 object signals must be reproduced by a 5.1-channel speaker system, a typical object-based audio decoding device generates decoded signals corresponding to 10 object signals, respectively, based on the downmix signal and additional information, and then generates a 5.1-channel signal by appropriately arranging 10 object signals in multi-channel space, so then the object signals can become suitable for 5.1-channel surround speakers. However, it is inefficient to generate 10 object signals during the generation of the 5.1-channel signal, and this problem becomes more serious when the difference between the number of object signals and the number of channels of the multi-channel signal to be generated increases.

On the other hand, in the embodiment of FIG. 4, the audio decoding apparatus 140 generates spatial parameter information suitable for a 5.1 channel signal based on additional information and control information, and provides spatial parameter information and a downmix signal to a multi-channel decoder 141 Then, the multi-channel decoder 141 generates a 5.1-channel signal based on the spatial parameter information and the downmix signal. In other words, when the number of channels to be output is 5.1 channels, the audio decoding apparatus 140 can easily generate a 5.1 channel signal based on the downmix signal without the need to generate 10 object signals, and thus is more efficient than normal audio decoding device in terms of complexity.

The audio decoding device 140 is considered effective when the number of calculations necessary to calculate the spatial parameter information corresponding to each OTT unit and the TTT block by analyzing additional information and control information transmitted by the audio encoding device is less than the number of calculations necessary to perform the mixing / rendering after decoding each signal of the object.

An audio decoding apparatus 140 can be obtained simply by adding a module for generating spatial parameter information by analyzing additional information and control information to a typical multi-channel audio decoding device, and thus can be compatible with a typical multi-channel audio decoding device. Also, audio decoding apparatus 140 can improve sound quality using existing tools of a typical multi-channel audio decoding apparatus, such as an envelope shaper, sub-band time processing (STP) tool, and decorrelator. Given all this, it is concluded that all the advantages of a typical multi-channel audio decoding method can easily be applied to an object audio decoding method.

The spatial parameter information transmitted to the multi-channel decoder 141 by the parameter converter 145 may be compressed to be suitable for transmission. Alternatively, the spatial parameter information may have the same format as the data format transmitted by a typical multi-channel encoding device. Those. the spatial parameter information may have been subjected to a Huffman decoding operation or a pilot decoding operation, and thus may be transmitted to each module as uncompressed spatial mark data. The first is suitable for transmitting spatial parameter information to a multi-channel audio decoding device in a remote location, and the second is convenient since there is no need for a multi-channel audio decoding device to convert compressed spatial label data to uncompressed spatial label data that can easily be used in the decoding operation .

Configuring spatial parameter information based on analysis of additional information and control information may cause a delay. To compensate for such a delay, an additional buffer for the downmix signal can be provided, so that the delay between the downmix signal and the bitstream can be compensated. Alternatively, an additional buffer may be provided for the spatial parameter information obtained from the control information, so that a delay between the spatial parameter information and the bitstream can be compensated. These methods, however, are inconvenient because of the need to provide an additional buffer. Alternatively, additional information may be transmitted before the downmix signal, given the possibility of a delay between the downmix signal and the spatial parameter information. In this case, there is no need to adjust the information on spatial parameters obtained by combining additional information and control information, but it can be easily used.

If the plurality of object signals of the downmix signal have different levels, an arbitrary downmix gain module (ADG), which can directly compensate for the downmix signal, can determine the relative signal levels of the object, and each of the object signals can be positioned in a predetermined position in the multi-channel space using spatial label data such as channel level difference (CLD) information, inter-channel correlation information Relation Factor (ICC) and Channel Prediction Ratio Information (CPC).

For example, if the control information indicates that the predetermined signal of the object should be distributed to a predetermined position in the multi-channel space and has a higher level than other signals of the object, a typical multi-channel decoder can calculate the difference between the channel energies of the downmix signal, and separate the downmix signal mixing on a number of output channels based on the calculation results. However, a typical multi-channel decoder cannot increase or decrease the volume of some sound in a downmix signal. In other words, a typical multi-channel decoder simply distributes the down-mix signal over a number of output channels and thus cannot increase or decrease the sound volume in the down-mix signal.

It is relatively easy to distribute each of a number of object signals of the down-mix signal generated by the object encoder to a predetermined position in the multi-channel space in accordance with the control information. However, special methods are needed to increase or decrease the amplitude of a predetermined object signal. In other words, if the downmix signal generated by the object encoder is used as is, it is difficult to reduce the amplitude of each signal object of the downmix signal.

Therefore, according to an embodiment of the present invention, the relative amplitudes of the object signals can be changed in accordance with the control information by using the ADG module 147 shown in FIG. The ADG module 147 may be installed in the multi-channel decoder 141 or may be separate from the multi-channel decoder 141.

If the relative amplitudes of the object signals of the down-mix signal are adjusted appropriately using the ADG module 147, then decoding of the object can be performed using a typical multi-channel decoder. If the down-mix signal generated by the object encoder is a monaural or stereo signal or a multi-channel signal with three or more channels, the down-mix signal may be processed by the ADG module 147. If the downmix signal generated by the object encoder has two or more channels and a predetermined object signal that requires adjustment by the ADG module 147 exists in only one of the channels of the downmix signal, the ADG module 147 can only be applied to a channel including a specific object signal, instead of applying a downmix signal to all channels. The down-mix signal processed by the ADG module 147 as described above can easily be processed using a typical multi-channel decoder without the need to modify the design of the multi-channel decoder.

Even if the final output signal is not a multi-channel signal that can be reproduced by a multi-channel speaker, but is a binaural signal, the ADG module 147 can be used to adjust the relative amplitudes of the signals of the final output signal object.

Alternative to using the ADG module 147, gain information specifying a gain value to be applied for each object signal may be included in the control information during the generation of a number of object signals. For this, the design of a typical multi-channel decoder can be modified. Even if requiring modification of the design of an existing multi-channel decoder, this method is convenient in the sense of reducing the complexity of decoding by applying a gain value to each signal of the object during the decoding operation without the need to calculate ADG and compensate for each signal of the object.

ADG module 147 can be used not only to adjust the signal levels of an object, but also to modify the spectral information of an object signal. More specifically, the ADG module 147 can be used not only to increase or decrease the level of a certain signal of an object, but also to modify the spectral information of a certain signal of an object, for example, to amplify a part with a high or low tone of a certain signal of an object. You cannot modify spectral information without using the 147 ADG module.

6 illustrates a block diagram of an audio decoding apparatus 150 according to a fourth embodiment of the present invention. As shown in FIG. 6, the audio decoding apparatus 150 includes a multi-channel binaural decoder 151, a first parameter converter 157 and a second parameter converter 159.

The second parameter converter 159 analyzes additional information and control information that is provided by the audio encoding device, and configures the spatial parameter information based on the analysis result. The first parameter converter 157 configures virtual three-dimensional (3D) parameter information that can be used by the multi-channel binaural decoder 151 by adding three-dimensional (3D) information, such as parameters of the sound perception modeling function (HRTF), to the spatial parameter information. The multi-channel binaural decoder 151 generates a binaural signal by applying the binaural parameter information to the downmix signal.

The first parameter converter 157 and the second parameter converter 159 can be replaced by one module, i.e. a parameter conversion module 155, which receives additional information, control information and three-dimensional information, and configures binaural parameter information based on the additional information, control information, and HRTF parameters.

Typically, in order to generate a binaural signal for playing a downmix signal including 10 object signals with a headphone, an object signal must generate 10 decoded signals corresponding to, respectively, 10 object signals based on the downmix signal and additional information. After that, the mixer / renderer distributes each of 10 signals of the object to a predetermined position in the multi-channel space with reference to control information in order to satisfy the environmental requirements of the 5-channel speaker. After that, the mixer / renderer generates a 5-channel signal that can be reproduced by a 5-channel speaker. After that, the mixer / renderer applies three-dimensional information to the 5-channel signal, thereby generating a 2-channel signal. Briefly, the aforementioned conventional audio decoding method includes reproducing 10 object signals, converting 10 object signals to a 5-channel signal, and generating a 2-channel signal based on the 5-channel signal, and thus is inefficient.

On the other hand, the audio decoding apparatus 150 can easily generate a binaural signal that can be reproduced using headphones, based on the signals of the object. In addition, the audio decoding apparatus 150 configures spatial parameter information by analyzing additional information and control information, and can thus generate a binaural signal using a typical multi-channel binaural decoder. In addition, the audio decoding apparatus 150 can still use a typical multi-channel binaural decoder even when it is equipped with an integrated parameter converter that receives additional information, control information and HRTF parameters and configures the binaural parameter information based on the additional information, control information and HRTF parameters.

7 illustrates a block diagram of an audio decoding apparatus 160 according to a fifth embodiment of the present invention. As shown in FIG. 7, the audio decoding apparatus 160 includes a preprocessor 161, a multi-channel decoder 163, and a parameter converter 165.

The parameter converter 165 generates spatial parameter information that can be used by the multi-channel decoder 163, and parameter information that can be used by the preprocessor 161. The preprocessor 161 performs a preprocessing operation on the downmix signal and transmits the downmix signal resulting from the preprocessing operation to multi-channel decoder 163. Multi-channel decoder 163 performs the decoding operation on the downmix signal Nia transmitted by the preprocessor 161, thereby outputting a stereo signal, a binaural stereo signal or a multichannel signal. Examples of the preprocessing operation performed by preprocessor 161 include modifying or converting a downmix signal in a time domain or a frequency domain using filtering.

If the down-mix signal inputted to the audio decoding apparatus 160 is a stereo signal, the down-mix signal may have been pre-processed by the down-mix performed by the preprocessor 161 before being input to the multi-channel decoder 163, since the multi-channel decoder 163 cannot display the object signal corresponding to the left channel of the stereo downmix signal; the right channel of the multi-channel signal by decoding. Therefore, in order to shift the object signal belonging to the left channel of the stereo down-mix signal to the right channel, it is possible that it is necessary to pre-process the stereo down-mix signal by preprocessor 161, and the down-mix signal with the pre-processing performed may be input to the multi-channel decoder 163.

The preprocessing of the stereo downmix signal may be performed based on the preprocessing information obtained from the additional information and from the control information.

FIG. 8 illustrates a block diagram of an audio decoding apparatus 170 according to a sixth embodiment of the present invention. As shown in FIG. 8, the audio decoding apparatus 170 includes a multi-channel decoder 171, a post-processor 173, and a parameter converter 175.

The parameter converter 175 generates spatial parameter information that can be used by the multi-channel decoder 163, and parameter information that can be used by the post-processor 173. The post-processor 173 performs a post-processing operation on a signal output by the multi-channel decoder 173. Examples of the signal output by the multi-channel decoder 173, include a stereo signal, a binaural stereo signal, and a multi-channel signal.

Examples of the post-processing operation performed by the post-processor 173 include modifying and converting each channel or all channels of the output signal. For example, if the additional information includes basic frequency information regarding a predetermined object signal, post processor 173 may remove harmonic components from the predetermined object signal with reference to the fundamental frequency information. The multi-channel audio decoding method may not be effective enough for use in a karaoke system. However, if the fundamental frequency information regarding the signals of the voice object is included in the additional information and the harmonic components of the signals of the voice object are deleted during the post-processing operation, a high-quality karaoke system can be realized by using the embodiment of FIG. 8. The embodiment of FIG. 8 can also be applied to object signals other than voice object signals. For example, you can remove the sound of a predefined musical instrument by using the embodiment of FIG. It is also possible to amplify the predefined harmonic components using the fundamental frequency information regarding the signals of the object by using the embodiment of FIG. 8. In short, post-processing parameters can make it possible to apply various effects, such as inserting a reverb effect, adding noise, and enhancing the low-tone part that cannot be performed by multi-channel decoder 171.

Postprocessor 173 can directly apply an additional effect to the downmix signal or add a downmix signal, to which the effect has already been applied, to the output of multi-channel decoder 171. Postprocessor 173 can modify the spectrum of the object or modify the downmix signal when necessary. If it is not suitable to directly perform an effect processing operation, such as reverb on a downmix signal, and transmit the signal obtained by the effect processing operation to a multi-channel decoder 171, the preprocessor 173 can simply add the signal obtained by the effect processing operation to the output signal of the multi-channel decoder 171, instead direct execution of the effect processing on the down-mixing signal and transmitting the result of the effect processing to the multi-channel decoder 171.

FIG. 9 illustrates a block diagram of an audio decoding apparatus 180 according to a seventh embodiment of the present invention. As shown in FIG. 9, the audio decoding apparatus 180 includes a preprocessor 181, a multi-channel decoder 183, a post-processor 185, and a parameter converter 187.

The description of preprocessor 161 is directly applicable to preprocessor 181. Postprocessor 185 can be used to add to the output of preprocessor 181 and the output of multi-channel decoder 185 and thus provide the final signal. In this case, postprocessor 185 serves merely as an adder to add signals. An effect parameter may be provided on that which of the preprocessor 181 and post-processor 185 performs the application of the effect. In addition, adding the signal obtained by applying the effect to the down-mix signal to the output of the multi-channel decoder 183 and applying the effect to the output of the multi-channel decoder 185 can be performed simultaneously.

The preprocessors 161 and 181 of FIGS. 7 and 9 may render the downmix signal in accordance with the control information provided by the user. In addition, the preprocessors 161 and 181 of FIGS. 7 and 9 can increase or decrease the signal levels of the object and change the signal spectra of the object. In this case, the preprocessors 161 and 181 of FIGS. 7 and 9 can perform the functions of an ADG module.

Rendering of the signal of the object in accordance with information about the direction of the signal of the object, adjusting the signal level of the object and changing the spectrum of the signal of the object can be performed simultaneously. In addition, some of the rendering of the signal of the object in accordance with information about the direction of the signal of the object, adjusting the signal level of the object and changing the spectrum of the signal of the object can be performed using preprocessors 161 or 181, and that of rendering the signal of the object in accordance with information about the signal direction object, adjusting the signal level of the object and changing the spectrum of the signal of the object is not performed by the preprocessor 161 or 181, can be performed by using the ADG module. For example, it is inefficient to change the spectrum of an object signal by using an ADG module that uses a quantization level interval and a parameter band interval. In this case, the preprocessor 161 or 181 can be used to accurately change the spectrum of the object signal on a frequency basis, and the ADG module can be used to adjust the signal level of the object.

10 illustrates a block diagram of an audio decoding apparatus according to an eighth embodiment of the present invention. As shown in FIG. 10, the audio decoding apparatus 200 includes a rendering matrix generator 201, a transcoder 203, a multi-channel decoder 205, a preprocessor 207, an effects processor 208, and an adder 209.

The rendering matrix generator 201 generates a rendering matrix that represents object position information regarding the signal positions of the object and playback configuration information regarding the signal levels of the object and provides a rendering matrix to transcoder 203. The rendering matrix generator 201 generates three-dimensional information, such as a coefficient HRTF based on position information. HRTF is a transfer function that describes the transmission of sound waves between a sound source in an arbitrary position and the eardrum, and returns a value that changes in accordance with the direction and height of the sound source. If a signal with no directivity is filtered using HRTF, the signal can be heard as if it were being played from some direction.

Information about the position of the object and information about the configuration of the playback, which is received by the generator 201 of the rendering matrix, can vary in time and can be provided by the end user.

Transcoder 203 generates channel-based additional information based on object-based additional information, a rendering matrix, and three-dimensional information, and provides multi-channel decoder 209 with channel-based additional information and three-dimensional information necessary for multi-channel decoder 209. That is, transcoder 203 transmits channel-based additional information regarding M channels, which is obtained from information about object-based parameters regarding N object signals and three-dimensional information of each of N object signals to multi-channel decoder 205.

The multi-channel decoder 205 generates a multi-channel audio signal based on the down-mix signal and channel-based additional information provided by the transcoder 203, and performs three-dimensional rendering of the multi-channel audio signal in accordance with the three-dimensional information, thereby generating a three-dimensional multi-channel signal. The rendering matrix generator 201 may include a database of three-dimensional information (not shown).

If there is a need for preprocessing the downmix signal before inputting the downmix signal to multi-channel decoder 205, transcoder 203 transmits information regarding the preprocessing to preprocessor 207. Object-based additional information includes information regarding all the signals of the object and a rendering matrix includes information about the position of the object and information about configuring playback. Transcoder 203 may generate channel-based additional information based on object-based additional information and a rendering matrix, and then generates channel-based additional information necessary for mixing and reproducing object signals in accordance with the channel information. After that, the transcoder 203 transmits the channel-based additional information to the multi-channel decoder 205.

The channel-based supplemental information and three-dimensional information represented by transcoder 205 may include frame indices. Thus, the multi-channel decoder 205 can synchronize the channel-based additional information and three-dimensional information by using frame indices and, thus, may be able to apply three-dimensional information only to certain frames of the bitstream. In addition, even if the three-dimensional information is updated, it is possible to easily synchronize the channel-based additional information and the updated three-dimensional information by using frame indices. Those. frame indices may be included in the channel-based additional information and three-dimensional information, respectively, so that the multi-channel decoder 205 synchronizes the channel-based additional information and three-dimensional information.

The preprocessor 207 may pre-process the input downmix signal, if necessary, before the input downmix signal is input to the multi-channel decoder 205. As described above, if the input downmix signal is a stereo signal and there is a need to play the signal of the object belonging to the left channel from the right channel, the downmix signal can be pre-processed by preprocessor 207 per d input to the multichannel decoder 205 because the multichannel decoder 205 can not move an object signal from one channel to another. The information necessary for preprocessing the input down-mix signal may be provided to preprocessor 207 by transcoder 205. The down-mix signal obtained by preprocessing performed by preprocessor 207 may be transmitted to multi-channel decoder 205.

The effects processor 208 and the adder 209 can directly apply an additional effect to the downmix signal or add a downmix signal in which the effect has already been applied to the output of the multi-channel decoder 205. The effects processor 208 can modify the spectrum of the object or modify the downmix signal when necessary . If it is not suitable to directly perform an effect processing operation, such as reverb, on the downmix signal and transmit the signal obtained by the effect processing operation to the multi-channel decoder 205, the effect processor 208 can simply add the signal obtained by the effect processing operation to the output signal of the multi-channel decoder 205 instead of directly performing the effect processing on the downmix signal and transmitting the result of the effect processing to the multi-channel decoder 205.

Below, this document describes in detail the rendering matrix generated by the rendering matrix generator 201.

The rendering matrix is a matrix that represents the position and configuration of the playback signals of the object. Those. if there are N object signals and M channels, then the rendering matrix can indicate how N object signals are displayed on M channels in different ways.

More specifically, when N object signals are mapped onto M channels, an N × M size rendering matrix can be set. In this case, the rendering matrix includes N rows that represent respectively N object signals and M columns that represent M channels, respectively. Each of the M coefficients in each of the N lines may be a real number or an integer indicating the ratio of the part of the signal of the object allocated to the corresponding channel to the entire signal of the object.

More specifically, the M coefficients in each of the N rows of an N × M size rendering matrix may be real numbers. Then, if the sum of the M coefficients in the row of the N × M rendering matrix is equal to a predetermined reference value, for example 1, it can be determined that the signal level of the object has not changed. If the sum of the M coefficients is less than 1, it is determined that the signal level of the object has been reduced. If the sum of the M coefficients is greater than 1, it is determined that the signal level of the object has increased. The predetermined reference value may be a numerical value other than 1. The amount by which the signal level of an object changes may be limited to a range of 12 dB. For example, if the predefined reference value is 1 and the sum of the M coefficients is 1.5, then it can be determined that the signal level of the object has increased by 12 dB. If the predefined reference value is 1 and the sum of the M coefficients is 0.5, then it is determined that the signal level of the object has decreased by 12 dB. If the predetermined reference value is 1 and the sum of the M coefficients is 0.5-1.5, it is determined that the object signal has changed by a predetermined value from -12 dB to +12 dB, and the predetermined value can be linearly determined in accordance with the sum M coefficients.

The M coefficients in each of the N rows of the N × M size rendering matrix may be integers. Then, if the sum of the M coefficients in the row of the N × M rendering matrix is equal to a predetermined reference value, for example 10, 20, 30 or 100, it can be determined that the signal level of the object has not been changed. If the sum of the M coefficients is less than a predetermined reference value, it can be determined that the signal level of the object has not been reduced. If the sum of the M coefficients is greater than the predetermined reference value, it can be determined that the signal level of the object has not been increased. The amount by which the signal level of the object changes may be limited to a range of, for example, 12 dB. The value by which the sum of the M coefficients differs from the predetermined reference value may represent the value (unit: dB) by which the signal level of the object has changed. For example, if the sum of the M coefficients is a sum that is larger than a predetermined reference value, then it can be determined that the signal level of the object has been increased by 2 dB. Therefore, if the predetermined reference value is 20 and the sum of the M coefficients is 23, then it can be determined that the signal level of the object has been increased by 6 dB. If the predetermined reference value is 20 and the sum of the M coefficients is 15, then it can be determined that the signal level of the object has been reduced by 10 dB.

For example, if there are six object signals and five channels (i.e., front left (FL), front right (FR), center C, rear left (RL) and rear right (RR) channels), a size rendering matrix can be set 6 × 5, having six rows corresponding respectively to six signals of the object, and five columns corresponding respectively to five channels. The coefficients of a 6 × 5 rendering matrix can be integers indicating the ratio with which each of the six object signals is distributed across five channels. A 6 × 5 rendering matrix may have a reference value of 10. Thus, if the sum of the five coefficients in any one of the six rows of a 6 × 5 rendering matrix is 10, then it can be determined that the level of the corresponding object signal has not changed. The value by which the sum of the five coefficients in any one of the six rows of the 6 × 5 rendering matrix differs from the reference value, represents the amount by which the level of the corresponding signal of the object has changed. For example, if the sum of the five coefficients in any one of the six rows of a 6 × 5 rendering matrix differs from the reference value by 1, then it can be determined that the level of the corresponding signal of the object has changed by 2 dB. A 6 × 5 rendering matrix can be represented by equation (1):

[Equation 1]

Referring to the 6 × 5 rendering matrix of equation (1), the first row corresponds to the first signal of the object and represents the ratio with which the first signal of the object is distributed over the FL-, FR-, C-, RL- and RR-channels. Since the first coefficient of the first line has the largest integer value of 3 and the sum of the coefficients of the first line is 10, it is determined that the first signal of the object is distributed mainly to the FL channel and that the level of the first signal of the object has not changed. Since the second coefficient of the second line, which corresponds to the second signal of the object, has the largest integer value of 4 and the sum of the coefficients of the second line is 12, it is determined that the second signal of the object is distributed mainly on the FR channel and that the level of the second signal of the object has been increased 4 dB. Since the third coefficient of the third line, which corresponds to the third signal of the object, has the largest integer value of 12 and the sum of the coefficients of the third line is 12, it is determined that the third signal of the object is distributed only on the C-channel and that the level of the third signal of the object was increased by 4 dB . Since all the coefficients of the fifth row, which corresponds to the fifth signal, have the same integer value 2, and the sum of the coefficients of the fifth row is 10, it is determined that the fifth signal of the object is evenly distributed over FL-, FR-, C-, RL- and RR- channels and that the level of the fifth signal of the object has not changed.

Alternatively, when N object signals are mapped onto M channels, an N × (M + 1) size rendering matrix can be set. The N × (M + 1) size rendering matrix is very similar to the N × M size rendering matrix. More specifically, in an N × (M + 1) size rendering matrix, like the N × M size rendering matrix, the first through Mth coefficients in each of the N rows represent the ratio with which the corresponding object signal is distributed across FL-, FR- , C, RL and RR channels. However, an N × (M + 1) size rendering matrix, unlike an N × M size rendering matrix, has an additional column (i.e., the (M + 1) th column) to represent the signal levels of the object.

The N × (M + 1) size rendering matrix, unlike the N × M rendering matrix, indicates how the object signal is distributed across the M channels and whether the object signal level has changed individually. Thus, by using an N × (M + 1) size rendering matrix, it is easy to obtain information regarding the change, if any, of the signal level of an object without the need for additional calculations. Since the N × (M + 1) size rendering matrix is almost the same as the N × M rendering matrix, the N × (M + 1) size rendering matrix can easily be converted to an N × M rendering matrix, or vice versa, without the need for additional information.

Alternatively, when N signals of an object are mapped onto M channels, an N × 2 size rendering matrix can be set. An N × 2 rendering matrix has a first column indicating the angular position of the object's signals, and a second column indicating the change, if any, in the level of each object's signal. An N × 2 rendering matrix can represent the angular position of the object’s signals at regular intervals of 1 or 3 degrees in the range of 0-360 degrees. An object signal that is evenly distributed in all directions can be represented by a predetermined value, rather than an angle.

An N × 2 rendering matrix can be converted to an N × 3 rendering matrix, which can indicate not only two-dimensional directions of an object’s signals, but also three-dimensional directions of an object’s signals. More specifically, the second column of an N × 3 size rendering matrix can be used to indicate the three-dimensional directions of the object signals. The third column of the N × 3 size rendering matrix indicates the change, if any, of the level of each object signal, using the same method as that used by the N × M size rendering matrix. If the final playback mode of the object decoder is binaural stereo, the rendering matrix generator 201 may transmit three-dimensional information indicating the position of each object signal or an index corresponding to the three-dimensional information. In the latter case, the transcoder 203 may need to have three-dimensional information corresponding to the index transmitted by the rendering matrix generator 201. In addition, if three-dimensional information indicating the position of each object signal is received from the rendering matrix generator 201, the transcoder 203 may be able to calculate three-dimensional information that can be used by the multi-channel decoder 205 based on the received three-dimensional information, the rendering matrix, and object-based additional information.

The rendering matrix and three-dimensional information can adaptively change in real time in accordance with the modification made for information about the position of the object and information about the configuration of playback by the end user. Therefore, information regarding whether the rendering matrix and three-dimensional information is updated, and updates, if any, of the rendering matrix and three-dimensional information can be transmitted to transcoder 203 at regular time intervals, for example, at 0.5 s intervals. Then, if updates to the rendering matrix and three-dimensional information are detected, transcoder 203 may linearly convert the received updates and the existing rendering matrix and existing three-dimensional information, assuming that the rendering matrix and three-dimensional information linearly change in time.

If the position information and the playback configuration information have not been modified by the end user after transmitting the rendering matrix and three-dimensional information to transcoder 203, information indicating that the rendering matrix and three-dimensional information have not been changed can be transmitted to transcoder 203. On the other hand, if the information about the position of the object and information about the configuration of the playback were modified by the end user after transmitting the rendering matrix and three-dimensional information on the trans a encoder 203, information indicating that the rendering matrix and three-dimensional information have been changed, and updates to the rendering matrix and three-dimensional information can be transmitted to transcoder 203. More specifically, updates to the rendering matrix and updates to three-dimensional information can be transmitted separately to transcoder 203. Alternatively, matrix updates the rendering and / or updating of three-dimensional information may jointly be represented by a predetermined representative value. Then, the predetermined representative value may be transmitted to the transcoder 203 together with information indicating that the predetermined representative value corresponds to updates to the rendering matrix or updates to three-dimensional information. In this way, transcoder 203 can be easily notified if the rendering matrix and 3D information have been updated or not.

An N × M size rendering matrix, like the matrix indicated by equation (1), may also include an additional column for representing information about the three-dimensional direction of the object signals. In this case, an additional column can represent information about the three-dimensional direction of the object's signals as angles in the range from -90 to +90 degrees. An additional column can be provided not only for a matrix of size N + M, but also for a matrix of rendering size N × (M + 1) and a matrix of size N × 2. Information about the three-dimensional direction of the object signals may not be necessary for use in the normal decoding mode of a multi-channel decoder. Instead, information about the three-dimensional direction of the object's signals may be necessary for use in a binaural mode of a multi-channel decoder. Information about the three-dimensional direction of the signals of the object can be transmitted together with the rendering matrix. Alternatively, information about the three-dimensional direction of the signals of the object can be transmitted together with three-dimensional information. Information about the three-dimensional direction of the object signals does not affect the channel-based additional information, but does affect the three-dimensional information during the decoding operation of the binaural mode.

Information regarding the spatial positions and signal levels of an object may be provided as a rendering matrix. Alternatively, information regarding spatial positions and signal levels of an object may be represented as modifications to the spectrum of the signal of the object, such as amplification of low-tone parts or high-tone parts of object signals. In this case, information regarding modifications to the spectrum of the object's signals can be transmitted as level changes in each parameter band that is used in a multi-channel codec. If the end user controls the modifications of the spectrum of the signals of the object, information regarding the modifications of the spectrum of signals of the object can be transmitted as a spectrum matrix separately from the rendering matrix. The spectrum matrix can have as many rows as there are object signals, and have as many columns as there are parameters. Each spectrum matrix coefficient indicates information regarding the level adjustment of each parameter band.

The principle of the transcoder 203 is described in detail later in this document. Transcoder 203 generates channel-based additional information for multi-channel decoder 205 based on object-based additional information, rendering matrix information and three-dimensional information, and transmits channel-based additional information to multi-channel decoder 205. In addition, transcoder 203 generates three-dimensional information for multi-channel decoder 205 and transmits three-dimensional information to a multi-channel deco ep 205. If an input downmix signal preprocessing necessary before entering the multi-channel decoder 205, the transcoder 203 may transmit information regarding the input downmix signal.

Transcoder 203 may receive object-based additional information indicating how a plurality of object signals are included in the downmix input. Object-based supplemental information may indicate how many object signals are included in the downmix input by using the OTT block and TTT block and using CLD, ICC, and CPC information. Object-based additional information may provide descriptions of various methods that may be performed by an object encoder to indicate information regarding each of the plurality of object signals, and may thus be able to indicate how object signals are included in the additional information.

In the case of the TTT block of a multi-channel codec, downmixing or upmixing of L-, C- and R-signals into L- and R-signals can be performed. In this case, the C-signal can share a bit of both L- and R-signals. However, this rarely happens in the case of downmixing or upmixing of object signals. Therefore, the OTT block is widely used to perform upmixing or downmixing to encode an object. Even if the C signal includes an independent component of the signal, and not parts of the L and R signals, the TTT block can be used to perform upmixing or downmixing to encode the object.

For example, if there are six object signals, six object signals can be converted into a downmix signal by the OTT block, and information regarding each object signal can be obtained by using the OTT block, as shown in FIG. 11.

As shown in FIG. 11, six object signals can be represented by a single downmix signal and information (such as CLD and ICC information) summarized by five OTT blocks 211, 213, 215, 217 and 219. The design depicted in FIG. 11 can be modified in numerous ways. That is, as shown in FIG. 11, the first OTT unit 211 can receive two of the six object signals. In addition, the way in which the OTT blocks 211, 213, 215, 217, and 219 are hierarchically connected can be freely changed. Therefore, additional information may include hierarchical structure information indicating how OTT blocks 211, 213, 215, 217 and 219 are hierarchically connected, and input position information indicating to which OTT block each signal of an object is input. If the OTT blocks 211, 213, 215, 217 and 219 form an arbitrary tree structure, the method used in the multi-channel codec to represent an arbitrary tree structure can be used to indicate such information about the hierarchical structure. In addition, such input position information may be indicated in numerous ways.

Additional information may also include information regarding the muting period of each object signal. In this case, the tree-like structure of OTT blocks 211, 213, 215, 217 and 219 can adaptively change in time. For example, as shown in FIG. 11, when the first object signal OBJECT1 is muted, information regarding the first OTT block 211 is not necessary, and only the second object signal OBJECT2 can be input to the fourth OTT block 217. Then the tree structure of the blocks 211, 213, 215, 217 and 219 OTT can be changed accordingly. Thus, information regarding the change, if any, in the tree structure of the OTT blocks 211, 213, 215, 217 and 219 can be included in the additional information.

If a predefined object signal is muted, information indicating that the OTT block corresponding to the predefined object signal is not in use and information indicating that labels from the OTT block are unavailable can be provided. Thus, it is possible to reduce the size of the additional information by not including information regarding OTT blocks or TTT blocks that are not in use in the additional information. Even if the tree structure of the multiple OTT or TTT blocks is modified, it can be easily determined which of the OTT or TTT blocks are turned on or off based on information indicating which signals of the object are muted. Therefore, it is not necessary to often transmit information regarding modifications, if any, of the tree structure of OTT or TTT blocks. Instead, information may be transmitted indicating which signal of the object is muted. The decoder can then easily determine which part of the tree structure of the OTT or TTT blocks needs to be modified. Therefore, it is possible to minimize the size of the information that is necessary for transmission to the decoder. In addition, labels relating to object signals can be easily transmitted to a decoder.

12 illustrates a diagram for explaining how a plurality of object signals are included in a downmix signal. In the embodiment of FIG. 11, the structure of the multi-channel encoding OTT blocks is adopted as it is. However, in the embodiment of FIG. 12, a change in the structure of the OTT blocks of multi-channel coding is used. That is, referring to FIG. 12, a plurality of object signals are input into each block, and at the end only one downmix signal is generated. As shown in FIG. 12, information regarding each of the plurality of object signals may be represented by the ratio of the energy level of each object signal to the total energy level of the object signals. However, when the number of object signals increases, the ratio of the energy level of each object signal to the total energy level of the object signals decreases. To solve this problem, a search is made for one of the many signals of the object (hereinafter referred to as the signal of the object with the highest energy) that has the highest energy level in a predetermined band of parameters, and the ratio of the energy levels of other signals of the object (hereinafter referred to as signals of the object with the least energy) to the energy level of the signal of the object with the highest energy can be provided as information regarding each of the signals of the object. In this case, if information is given indicating the signal of the object with the highest energy and the absolute value of the energy level of the signal of the object with the highest energy, then the energy levels of other signals of the object with the lowest energy can be easily determined.

The energy level of the signal of the object with the highest energy is needed to combine multiple bit streams into a single bit stream, as is done in a multi-communication control device (MCU). However, in most cases, the energy level of the signal of the object with the highest energy is not necessary, since the absolute value of the energy level of the signal of the object with the highest energy can easily be obtained from the ratios of the energy levels of other signals of the object with the lowest energy to the energy level of the signal of the object with the highest energy.

For example, suppose that there are four object signals A, B, C, and D that belong to a predefined parameter strip, and that the object signal A is the signal of the object with the highest energy. Then the energy E _{P of} a predetermined band of parameters and the absolute value E _{A of} the energy level of the signal A of the object satisfy equation (2):

[Equation 2]

where a, b and c respectively indicate the ratio of the energy level of the signals B, C and D of the object to the energy level of the signal of the object. Referring to equation (2), it is possible to calculate the absolute value E _{A of} the energy level of the signal A of the object based on the ratios a, b and c and energy E _{P of} a predetermined parameter band. Therefore, if it is not necessary to combine multiple bit streams into a single bit stream using the MCU, then it may not be necessary to include in the bit stream the absolute value E _{A of} the energy level of signal A of the object. Information indicating whether the absolute value E _{A of} the energy level of signal A of the object is included in the bitstream can be included in the header of the bitstream, thereby reducing the size of the bitstream.

On the other hand, if there is a need to combine multiple bit streams into a single bit stream using the MCU, the energy level of the object signal with the highest energy is needed. In this case, the sum of the energy levels calculated on the basis of the ratio of the energy levels of the signals of the object with the least energy to the energy level of the signal of the object with the highest energy may not be the same as the energy level of the downmix signal obtained by downmixing all the signals of the object. For example, when the energy level of the down-mix signal is 100, the sum of the calculated energy levels may be 98 or 103, for example, due to errors caused during the quantization and dequantization operations. To solve this issue, the difference between the energy level of the downmix signal and the sum of the calculated energy levels can be appropriately compensated by multiplying each calculated energy level by a predetermined coefficient. If the energy level of the downmix signal is X and the sum of the calculated energy levels is Y, each calculated energy level can be multiplied by X / Y. If the difference between the energy level of the down-mix signal and the sum of the calculated energy levels is not compensated, such quantization errors can be included in the parameter bands and frames, thereby causing signal distortion.

Therefore, information is needed indicating which of the plurality of object signals has the largest absolute energy value in a predetermined parameter band. Such information may be represented by a number of bits. The number of bits required to indicate which of the plurality of object signals has the largest absolute energy value in a predetermined parameter band varies according to the number of object signals. When the number of object signals increases, the number of bits needed to indicate which of the plurality of object signals has the largest absolute energy value in a predetermined parameter band increases. On the other hand, when the number of object signals decreases, the number of bits necessary to indicate which of the plurality of object signals has the largest absolute energy value in a predetermined parameter band decreases. A predetermined number of bits can be distributed in advance to indicate which of the plurality of object signals has the largest absolute energy value in a predetermined parameter band. Alternatively, the number of bits for indicating which of the plurality of object signals has the largest absolute energy value in a predetermined parameter band may be determined based on certain information.

The size of information indicating which of the plurality of object signals has the largest absolute energy value in each parameter band can be reduced by using the same method used to reduce the size of CLD, ICC and CPC information for use in the OTT and / or TTT blocks of a multi-channel codec for example, by using a time-differential method, a frequency-differential method, or a pilot coding method.

To indicate which of the plurality of object signals has the largest absolute energy value in each parameter band, an optimized Huffman table can be used. In this case, information may be required indicating in which order the energy levels of the signals of the object are compared with the energy level of that signal of the object that has the highest absolute energy. For example, if there are five object signals (i.e., the first to fifth object signals) and the third object signal is the object signal with the highest energy, information regarding the third object signal can be provided. Then the ratio of the energy levels of the first, second, fourth and fifth signals of the object to the energy level of the third signal of the object can be provided in various ways, and this is described in more detail later in this document.

The ratio of the energy levels of the first, second, fourth and fifth signals of the object to the energy level of the third signal of the object can be provided sequentially. Alternatively, the ratio of the energy levels of the fourth, fifth, first and second signals of the object to the energy level of the third signal of the object can be sequentially provided in a cyclical manner. Then, information indicating the order in which the ratio of the energy levels of the first, second, fourth and fifth signals of the object to the energy level of the third signal of the object can be provided can be included in the file header or can be transmitted at intervals of a number of frames. A multi-channel codec can determine CLD and ICC information based on the sequence numbers of OTT blocks. Similarly, information is needed indicating how each signal of an object is mapped onto a bitstream.

In the case of a multi-channel codec, information regarding the signals corresponding to each channel can be identified by the sequence numbers of the OTT or TTT blocks. According to an object-based audio encoding method, if there are N object signals, then N signaling of the object signals may be required accordingly. However, it is sometimes necessary that the end user control N signals of an object using an object decoder. In this case, the end user may need not only serial numbers of N signals of the object, but also descriptions of N signals of the object, such as descriptions indicating that the first signal of the object corresponds to the voice of a woman and that the second signal of the object corresponds to the sound of a piano. Descriptions of the N signals of the object may be included in the header of the bitstream as metadata and then transmitted along with the bitstream. More specifically, descriptions of N signals of an object may be provided as text, or may be provided by using a code table or code words.

Sometimes correlation information regarding correlation between object signals is necessary. For this, the correlation between the signal of the object with the highest energy and other signals of the object with the lowest energy can be calculated. In this case, a single correlation value can be assigned for all signals of the object, which is comparable to using a single ICC value in all OTT blocks.

If the object signals are stereo signals, the ratio of the energy of the left channel to the energy of the right channel of the object signals and ICC information are necessary. The ratios of the energy of the left channel to the energy of the right channel of the object signals can be calculated using the same method as used to calculate the energy levels of the set of object signals based on the absolute value of the energy level of that signal of the object, which is the signal of the object with the highest energy, and level ratios energy of other signals of the object with the least energy to the energy level of the signal of the object with the highest energy. For example, if the absolute values of the energy levels of the left and right channels of the signal of the object with the highest energy are A and B, respectively, the ratio of the energy level of the left channel of the signal of the object with the lowest energy to A and the ratio of the energy level of the right channel of the signal of the object with the lowest energy to B are x and y, respectively, the energy levels of the left and right channels of the signal of the object with the least energy can be calculated as A * x and B * y. Thus, the ratio of the energy of the left channel to the energy of the right channel of the stereo signal of the object can be calculated.

The absolute value of the energy level of the signal of the object with the highest energy and the ratio of the energy levels of other signals of the object with the lowest energy to the energy level of the signal of the object with the highest energy can also be used when the signals of the object are monophonic signals, the down-mix signal obtained by the monophonic signals of the object represents is a stereo signal, and the monophonic signals of the subject are included in both channels of the stereo downmix signal . In this case, the ratio of the energy of the part of each monophonic signal of the object included in the left channel of the stereo downmix signal, and the energy of part of the corresponding monophonic signal of the object included in the right channel of the stereo downmix signal, and correlation information, and it is directly applied to stereo signals object. If a monophonic signal of an object is included in both the L- and R-channels of the stereo downmix signal, the components of the L- and R-channels of the monophonic signal of the object can have only a difference of levels, and the monophonic signal of the object can have a correlation value of 1 over all parameter bands. In this case, in order to reduce the amount of data, information may additionally be provided indicating that the monophonic signal of the object has a correlation value of 1 over all parameter bands. Then there is no need to indicate a correlation value of 1 for each band of parameters. Instead, a correlation value of 1 may be indicated for all parameter bands.

During generation of the downmix signal by summing a plurality of object signals, a restriction may occur. To solve this problem, the downmix signal can be multiplied by a predetermined gain, so that the maximum level of the downmix signal can exceed the limit threshold. A predetermined gain may vary over time. Therefore, information regarding a predetermined gain is necessary. If the downmix signal is a stereo signal, different gain values may be provided for the L- and R-channels of the downmix signal to prevent clipping. To reduce the number of data transmissions, different gain values may not be transmitted separately. Instead, a sum of different gain values and a ratio of different gain values can be transmitted. Then you can reduce the dynamic range and reduce the number of data transfers compared with the case of transmitting different values of the gain separately.

In order to further reduce the number of data transmissions, a bit may be provided indicating whether there has been a restriction during the generation of the downmix signal by summing a plurality of object signals. Then, if it is only determined that there has been a limitation, gain values may be transmitted. Such clipping information may be necessary to prevent clipping during the summation of a plurality of downmix signals to combine a plurality of bit streams. To prevent clipping, the sum of the plurality of downmix signals can be multiplied by the inverse of a predetermined gain value to prevent clipping.

13-16 illustrate diagrams for explaining various methods for configuring object-based supplemental information. The embodiments of FIGS. 13-16 can be applied not only to monophonic or stereo signals of an object, but also to multi-channel signals of an object.

As shown in FIG. 13, a multi-channel signal (OBJECT A (CH1) to OBJECT A (CHn)) of an object is input to the encoder 221 of the object. Then, the encoder 221 of the object generates a down-mix signal and additional information based on the multi-channel signal (OBJECT A (CH1) - OBJECT A (CHn)) of the object. The object encoder 223 receives a plurality of object signals OBJECT1-OBJECTn and a downmix signal generated by the object encoder 221 and generates another downmix signal and another additional formation based on the object OBJ.1-OBJ.N signals and the received downmix signal. A multiplexer 225 combines the additional information generated by the object encoder 221 and the additional information generated by the object encoder 223.

As shown in FIG. 14, an object encoder 233 generates a first bitstream based on a multi-channel signal (OBJECT A (CH1) - OBJECT A (CHn)) of the object. Then, the object encoder 231 generates a second bitstream based on a plurality of multi-channel object signals OBJECT1-OBJECTn. Then, the object encoder 235 combines the first and second bitstreams into a single bitstream by using almost the same method as used to combine multiple bitstreams into a single bitstream using the MCU.

As shown in FIG. 15, multi-channel encoder 241 generates a down-mix signal and channel-based additional information based on a multi-channel signal (OBJECT A (CH1) to OBJECT A (CHn)) of an object. The object encoder 243 receives the down-mix signal generated by the multi-channel encoder 241 and a plurality of multi-channel object signals OBJECT1-OBJECTn and generates an object bit stream and additional information based on the received down-mix signal and the object signals OBJECT1-OBJECTn. The multiplexer 245 combines the channel-based additional information generated by the multi-channel encoder 241 and the additional information generated by the encoder 243 of the objects, and displays the result of the combination.

As shown in FIG. 16, multi-channel encoder 253 generates a down-mix signal and channel-based supplemental information based on a multi-channel signal (OBJECT A (CH1) to OBJECT A (CHn)) of an object. An object encoder 251 generates a downmix signal and additional information based on a plurality of multi-channel object signals OBJECT1-OBJECTn. The object encoder 255 receives the down-mix signal generated by the multi-channel encoder 253 and the down-mix signal generated by the object encoder 251 and combines the received down-mix signals. The multiplexer 257 combines the additional information generated by the encoder 251 objects, and channel-based additional information generated by the multi-channel encoder 253, and displays the result of the combination.

When using object-based audio encoding in a teleconference, it is sometimes necessary to combine multiple object bitstreams into a single bitstream. The following describes in detail the combination of multiple object bitstreams into a single object bitstream.

17 illustrates a diagram for explaining the combination of two object bitstreams. As shown in FIG. 17, when two object bitstreams are combined into a single object bitstream, additional information, such as CLD and ICC information present in the two object bitstreams, should accordingly be modified. Two object bitstreams can be combined into a single object bitstream simply by using an additional OTT block, i.e. of the eleventh OTT block, and the use of additional information such as information about the CLD and ICC provided by the eleventh OTT block.

The tree configuration information of each of the two object bitstreams must be combined into the integrated tree configuration information in order to combine the two bitstreams into a single object bitstream. For this, additional configuration information can be modified, if there is any generated by combining two bit streams of the object, the indices of a number of OTT blocks used to generate two bit streams of the object can be modified, and only a few additional processes can be performed, such as the calculation process performed by the eleventh block of the OTT, and down-mixing of two signals down-mixing of two bit streams of the object. Thus, two object bit streams can easily be combined into a single object bit stream without the need to modify information regarding each of the plurality of object signals from which two object signals originate.

As shown in FIG. 17, an eleventh OTT block may be optional. In this case, two down-mix signals of two bit streams of an object can be used, since they are a two-channel down-mix signal. Thus, two object bitstreams can be combined into a single object bitstream without the need for additional calculations.

FIG. 18 illustrates a diagram for explaining combining two or more independent object bitstreams into a single object bitstream having a stereo downmix signal. As shown in FIG. 18, if two or more independent object bitstreams have a different number of parameter bands, the mapping of parameter bands can be performed on the object bitstreams, so that the number of parameter bands of one of the object bitstreams having a smaller number of parameter bands can be increased so that it is the same as the number of parameter bands of another object bitstream.

More specifically, the mapping of parameter bands may be performed using a predetermined mapping table. In this case, the display of the parameter bands can be performed using a simple linear formula.

If there are overlapping parameter bands, the parameter values can be mixed accordingly, taking into account the amount by which the overlapping parameter bands overlap. In those situations where preference is given to low complexity, the mapping of parameter bands can be performed on two object bit streams, so that the number of parameter bands of one of the two bit streams of an object having more parameter bands can be reduced so that it is the same as the number of parameter bands of another object bitstream.

In the embodiments of FIGS. 17 and 18, two or more independent object bitstreams can be combined into an integrated object bitstream without having to calculate existing parameters of the object's independent bit streams. However, in the case of combining a plurality of downmix signals, the parameters regarding the downmix signals may need to be calculated again using a quadrature mirror filter (QMF) / hybrid analysis. However, this calculation requires a large number of calculations, thus compromising the advantages of the embodiments of FIGS. 17 and 18. Therefore, it is necessary to propose methods for extracting parameters without the need for QMF / hybrid analysis or synthesis, even when downmixing downmix signals is performed. For this, energy information regarding the energy of each parameter band of each downmix signal may be included in the object bitstream. Then, when down-mixing of down-mixing signals is performed, information such as CLD information can easily be calculated based on such energy information without the need for QMF / hybrid analysis or synthesis. Such energy information may represent the highest energy level for each parameter band or the absolute value of the energy level of the object signal with the highest energy for each parameter band. The number of calculations can be further reduced by using ICC values obtained from the time domain for the entire parameter band.

During downmixing of a plurality of downmix signals, a restriction may occur. To solve this issue, down-mix signal levels can be reduced. If the levels of the downmix signals are reduced, it may be necessary to include level information regarding the reduced levels of the downmix signals in the object bitstream. Level information to prevent restriction can be applied to each frame of an object's bitstream or can only be applied to some frames in which restriction occurs. The levels of the original down-mix signals can be calculated by re-applying the level information to prevent restriction during the decoding operation. Level information to prevent restriction can be calculated in the time domain, and thus, it does not need to be subjected to QMF / hybrid synthesis or analysis. Combining the plurality of object signals into a single object bitstream can be performed using the structure depicted in FIG. 12, and this is described in detail herein with reference to FIG. 19.

FIG. 19 illustrates a diagram for explaining combining two independent object bitstreams into a single object bitstream. As shown in FIG. 19, the first block 261 generates a first object bitstream and the second block 263 generates a second object bitstream. Then, the third block 265 generates a third object bitstream by combining the first and second bit streams. In this case, if the first and second bit streams of the object include information on the absolute value of the energy level of the signal of the object with the highest energy for each band of parameters and the ratio of the energy levels of other signals of the object with the lowest energy to the energy level of the signal of the object with the highest energy and information about gain factor relating to gain values that are multiplied by down-mix signals by the first and second blocks 261 and 263, the third block 265 may generate a third bit otok of the object simply by combining the first and second bit streams without the need for additional calculation or extraction of parameters.

The third block 265 receives a plurality of downmix signals DOWNMIX_A and DOWNMIX_B. The third block 265 converts the downmix signals DOWNMIX_A and DOWNMIX_B into pulse code modulation (PCM) signals and adds PCM signals, thereby generating a single downmix signal. During this process, however, a restriction may occur. To solve this problem, the downmix signals DOWNMIX_A and DOWNMIX_B can be multiplied by a predetermined gain value. Information regarding a predetermined gain value may be included in the third object bitstream and transmitted together with the third object bitstream.

Combining multiple object bitstreams into a single object bitstream is described in more detail later in this document. As shown in FIG. 19, paramA may include information regarding which of the plurality of object signals OBJECT1-OBJECTn is the signal of the object with the highest energy, and the ratio of the energy levels of other signals of the object with the lowest energy to the energy level of the signal of the object with the highest energy. Similarly, SIDE INFO may include information regarding which of the plurality of object signals OBJECT1-OBJECTn is the signal of the object with the highest energy and the ratio of the energy levels of other signals of the object with the lowest energy to the energy level of the signal of the object with the highest energy.

SIDE_INFO_A and SIDE_INFO_B can be included in parallel in one bit stream, as shown in Fig.20. In this case, a bit may further be provided indicating whether more than one bit stream exists in parallel.

As shown in FIG. 20, to indicate whether or not a predetermined bit stream is an integrated bit stream including more than one bit stream, information indicating whether a predetermined bit stream is an integrated bit stream, information regarding the number of bit streams if there are any included in the predefined bitstream, and information regarding the initial position of the bit streams, if there are any included in the predefined lenny bitstream may be provided at the head of the predetermined bitstream and followed by more than one bitstream, if there is any, in the predetermined bitstream. In this case, the decoder can determine whether the predefined bitstream is an integrated bitstream including more than one bitstream by analyzing the information at the head of the predefined bitstream. This type of method for combining bit streams does not require additional processes, with the exception of adding multiple identifiers to the bit stream. However, such identifiers should be provided at intervals of a certain number of frames. In addition, this type of method for combining bit streams requires a decoder to determine whether or not each bit stream that the decoder receives is an integrated bit stream.

As an alternative to the aforementioned method of combining bit streams, a plurality of bit streams may be combined into a single bit stream so that the decoder cannot recognize that the single bit stream is or not an integrated bit stream. Hereinafter, this is described in detail with reference to FIG.

As shown in FIG. 21, the energy level of the signal of the object with the highest energy represented by SIDE_INFO_A is compared with the energy level of the signal of the object with the highest energy represented by SIDE_INFO_B. Then it is determined which one of the two signals of the object has a higher energy level, which is the signal of the object with the highest energy integrated bit stream. For example, if the energy level of the signal of the object with the highest energy represented by SIDE_INFO_A is greater than the energy level of the signal of the object with the highest energy represented by SIDE_INFO_B, the signal of the object with the highest energy represented by SIDE_INFO_A may become the signal of the object with the highest energy of the integrated bit stream. Then the energy ratio information SIDE_INFO_A can be used in the integrated bitstream as it is, while the energy ratio information SIDE_INFO_B can be multiplied by the ratio of energy levels

Then, the information on the energy ratio of which of SIDE_INFO_A and SIDE_INFO_B includes information regarding the signal of the object with the highest energy of the integrated bit stream can be used in the integrated bit stream, and information on the ratio of the energies of the signal of the object with the highest energy represented by param A, and the signal of the object with the highest energy represented by SIDE_INFO_B. This method includes re-computing the energy ratio information SIDE_INFO_B. However, the recalculation of energy ratio information SIDE_INFO_B is relatively simple. In this method, the decoder may not be able to determine whether or not the bitstream that it receives is an integrated bitstream including more than one bitstream, and thus a typical decoding method can be used.

Two object bitstreams, including stereo downmix signals, can easily be combined into a single object bitstream without having to recalculate information regarding the object's signals, using almost the same method as used to combine bitstreams including monophonic down-mix signals. In the object bitstream, information regarding the tree structure that performs down-mixing of the object signals is followed by information about the object signal received from each branch (i.e., each block) of the tree structure.

The bit streams of the object have been described above, assuming that some object is distributed only to the left channel or the right channel of the stereo downmix signal. However, object signals are typically distributed between both channels of the stereo downmix signal. Therefore, later in this document, it is described in detail how to generate an object bitstream based on object bitstreams that are distributed between two channels of a stereo downmix signal.

FIG. 22 illustrates a diagram for explaining a method of generating a stereo downmix signal by mixing a plurality of object signals, and more specifically, a method of performing downmixing of four object signals OBJECT1 to OBJECT4 into L and R stereo signals. As shown in FIG. 22, some of the four object signals OBJECT1-OBJECT4 belong to both L- and R-channels of the downmix signal. For example, the first object signal OBJECT1 is distributed between the L- and R-channels with respect to a: b, as indicated by equation (3):

[Equation 3]

If the object signal is distributed between the L- and R-channels of the stereo downmix signal, additional information on the channel allocation relation may be required regarding the ratio (a: b) at which the object signal is distributed between the L- and R-channels. Then, information regarding object signals, such as CLD and ICC information, can be calculated by performing downmixing using OTT blocks for the L- and R-channels of the stereo downmix signal, and this will be described in more detail below with reference to FIG. .23.

As shown in FIG. 23, if CLD and ICC information provided from a plurality of OTT blocks during a downmix operation and information on a channel allocation ratio of each of a plurality of object signals are provided, then a multi-channel bitstream that varies adaptively to any modifications made to information about the position of the object and information about configuring playback by the end user. In addition, if the stereo downmix signal is to be processed by the downmix preprocessing, then information regarding how the stereo downmix signal is processed by the downmix preprocessing can be obtained and the information received is transmitted to the preprocessor. That is, if there is no information about the channel distribution ratio of each of the multiple signals of the object, there is no way to calculate the multi-channel bitstream and obtain the information necessary for the preprocessor to work. Information on the distribution ratio of the channel signal of an object can be represented as the ratio of two integers or scalar quantities (unit: dB).

As described above, if an object signal is distributed between two channels of a stereo downmix signal, information about the channel allocation ratio of the object signal may be required. The channel allocation relationship information may have a fixed value indicating the relationship with which the object signal is distributed between the two channels of the stereo downmix signal. Alternatively, the channel allocation information of the object signal may vary from one frequency band to another frequency band of the object signal, especially when the channel allocation ratio information is used as ICC information. If the stereo down-mix signal is obtained through a complex down-mix operation, i.e. if the object signal belongs to two channels of the stereo downmix signal and downmix is performed by changing the ICC information from one frequency band to another frequency band of the object signal, a detailed description of the downmix of the object signal may additionally be required to decode the object signal that is ultimately rendered account. This embodiment can be applied to all possible object structures that have already been described.

Hereinafter, pre-processing is described in detail below with reference to FIGS. 24-27. If the downmix signal input to the object decoder is a stereo signal, the downmix signal input may need to be pre-processed before being input to the multichannel decoder of the object decoder, since the multichannel decoder cannot display the signal belonging to the left channel of the input downmix signal, to the right channel. Therefore, in order for the end user to move the position of the signal of the object belonging to the left channel of the input downmix signal to the right channel, the input downmix signal may require pre-processing, and the preprocessed downmix signal can be input to the multi-channel decoder.

The preprocessing of the stereo downmix signal can be performed by obtaining the preliminary processing information from the object bitstream and from the rendering matrix and the corresponding processing of the stereo downmix signal in accordance with the preprocessing information, and this will be described in detail later in this document.

24 illustrates a diagram for explaining how to configure a stereo down-mix signal based on four object signals OBJECT1-OBJECT4. As shown in Fig. 24, the first object signal OBJECT1 is distributed between the L- and R-channels with the ratio a: b, the second object signal OBJECT2 is distributed between the L- and R-channels with the ratio c: d, the third object signal OBJECT3 is distributed only on L-channel, and the fourth signal OBJECT4 object is distributed only on the R-channel. Information such as CLD and ICC can be generated by passing each of the first to fourth object signals OBJECT1-OBJECT4 through a number of OTTs, and a downmix signal can be generated based on the generated information.

Suppose that the end user receives a rendering matrix by appropriately setting the positions and levels of the first to fourth object signals OBJECT1-OBJECT4 and that there are five channels. The rendering matrix can be represented by equation (4):

[Equation 4]

As shown in equation (4), when the sum of the five coefficients in each of the four rows is equal to a predetermined reference value, i.e. 100, it is determined that the level of the corresponding signal of the object has not changed. The value by which the sum of the five coefficients in each of the four lines differs from the predefined reference value may be the amount (unit: dB) by which the level of the corresponding signal of the object has changed. The first, second, third, fourth and fifth columns of the rendering matrix of equation (4) represent the FL, FR, C, RL and RR channels, respectively.

The first row of the rendering matrix of equation (4) corresponds to the first signal OBJECT1 of the object and has a total of five coefficients, i.e. 30, 10, 20, 30, and 10. Since the sum of the five coefficients of the first row is 100, it is determined that the level of the first signal OBJECT1 of the object has not changed and that only the spatial position of the first signal OBJECT1 of the object has changed. Even if the five coefficients of the first row represent different channel directions, they can mainly be classified by two channels: L and R channels. Then the ratio with which the first signal OBJECT1 of the object is distributed between the L- and R-channels can be calculated as 70% (= (30 + 30 + 20) * 0.5): 30% (= 10 + 10 + 20) * 0, 5). Therefore, the rendering matrix of equation (4) indicates that the level of the first signal OBJECT1 of the object has not changed and that the first signal OBJECT1 of the object is distributed between the L- and R-channels with a ratio of 70%: 30%. If the sum of the five coefficients of any one of the rows of the rendering matrix of equation (4) is less than or greater than 100, it can be determined that the level of the corresponding signal of the object has changed, and then the corresponding signal of the object can be processed by pre-processing or can be converted and transmitted as ADG.

In order to pre-process the downmix signals, the ratio with which the downmix signals are distributed between the parameter bands from which the parameters are extracted from the QMF / hybrid transform of the downmix signals and the downmix signals can be redistributed between the parameter bands in according to the setting of the rendering matrix. Below in this document, various methods for redistributing down-mix signals between parameter bands are described in detail.

In the first redistribution method, the down-mix signals of the L- and R-channels are decoded separately using their respective additional information (such as CLD and ICC information) and using almost the same method as that used by the multi-channel codec. Then, the object signals distributed between the down-mix signals of the L- and R-channels are restored. To reduce the amount of computation, the downmix signals of the L and R channels can be decoded using only CLD information. The ratio with which each of the reconstructed object signals is distributed between the down-mix signals of the L and R channels can be determined based on additional information.

Each of the reconstructed object signals can be redistributed between the down-mix signals of the L- and R-channels in accordance with the rendering matrix. Then, down-mixing of the redistributed object signals on a per-channel basis is performed by means of OTT blocks, thereby completing the preliminary processing. In short, the first redistribution method adopts the same method as that used by the multi-channel codec. However, the first redistribution method requires as many decoding processes as there are object signals for each channel, and requires a redistribution process and a channel-based downmix process.

In the second method of redistribution, in contrast to the first method of redistribution, the object signals are not restored from the L- and R-signals of down-mixing. Instead, each of the L- and R-downmix signals is divided into two parts: one part L_L or R_R, which should be left in the corresponding channel, and the other part L_R or R_L, which should be redistributed, as shown in Fig. 25. As shown in FIG. 25, L_L indicates the portion of the L-channel downmix signal that should be left in the L-channel, and L_R indicates the portion of the L-channel downmix signal that should be added to the R-channel. Similarly, R_R indicates the portion of the R-channel down-mix signal that should be left in the R-channel, and R_L indicates the portion of the R-channel down-mix signal that should be added to the L-channel. Each of the down-mix signals of the L- and R-channels can be divided into two parts (L_L and L_R or R_R and R_L) in accordance with the ratio with which each object signal is distributed between the L- and R-down-mix signals, as defined by the equation (2), and the ratio with which each object signal should be distributed between the pre-processed L- and R-channels L and R, as defined by equation (3). Therefore, it can be determined how the downmix signals of the L- and R-channels should be redistributed between the pre-processed L- and R-channels L and R by comparing the relationship with which each object signal is distributed between the L- and R-downmix signals, and the relationship with which each object signal must be distributed between the pre-processed L- and R-channels L and R.

The division of the L-channel signal into L_L and L_R signals in accordance with a predetermined energy ratio has been described above. If the L-channel signal is divided into L_L and L_R signals, ICC determination between the L_L and L_R signals may be required. The ICC between the L_L and L_R signals can be easily determined based on ICC information regarding the signals of the object. Those. The ICC between the L_L and L_R signals can be determined based on the relationship with which each object signal is distributed between the L_L and L_R signals.

Below in this document, a second method for redistributing down-mix is described in more detail. Assume that the L and R down-mix signals L and R are obtained by the method shown in FIG. 24, and that the first, second, third and fourth object signals OBJECT1, OBJECT2, OBJECT3 and OBJECT4 are distributed between the down-mix signals L and R L- and R-channels with ratios 1: 2, 2: 3, 1: 0 and 0: 1, respectively. Downmixing of a plurality of object signals can be performed by means of a number of OTT blocks, and information, such as CLD and ICC information, can be obtained from downmixing of the object signals.

An example of a rendering matrix set for the first to fourth object signals OBJECT1-OBJECT4 is one that is represented by equation (4). The rendering matrix includes information on the position of the first to fourth signals of the object OBJECT1-OBJECT4. Thus, pre-processed L and R down-mix signals of L and R channels can be obtained by performing pre-processing using a rendering matrix. How to establish and interpret the rendering matrix has already been described above with reference to equation (3).

The ratio with which each of the first to fourth object signals OBJECT1-OBJECT4 is distributed between the pre-processed L and R channel down-mix signals L and R can be calculated as indicated by equation (5):

[Equation 5]

The ratio with which each of the first to fourth object signals OBJECT1 to OBJECT4 is distributed between the L and R down-mix signals L and R can be calculated as indicated by equation (6):

[Equation 6]

As shown in equation (5), the sum of the part of the third object signal OBJECT3 allocated to the pre-processed L-channel down-mix signal L and the part of the third object signal OBJECT3 distributed to the R-channel down-mix signal R is 110, and thus it is determined that the level of the third signal OBJECT3 of the object has increased by 10. On the other hand, the sum of the part of the fourth signal OBJECT4 of the object distributed to the pre-processed signal L of the down-mix L-channel and the part of the fourth signal OBJECT4 of the object, p the R-channel downmix signal R is equal to 95, and thus it is determined that the level of the fourth object signal OBJECT4 has decreased by 5. If the rendering matrix for the first-fourth object signals OBJECT1-OBJECT4 has a reference value of 100, and the value, by which the sum of the coefficients in each row of the rendering matrix differs from the reference value 100, represents the value (unit: dB) by which the level of the corresponding signal of the object has changed, it can be determined that the level of the third signal OBJECT3 ARP increased by 10 dB and that of the fourth object signal OBJECT4 has decreased by 5 dB.

Equations (5) and (6) can be converted into equation (7):

[Equation 7]

Equation (7) compares the ratio with which each of the first to fourth object signals OBJECT1-OBJECT4 is distributed between the down-mix signals of the L and R channels before preprocessing, and the ratio with which each of the first to fourth object signals OBJECT1-OBJECT4 is distributed between the down-mix signals of the L- and R-channels after pre-processing. Therefore, by using equation (7), it is easy to determine which part of each of the first to fourth object signals OBJECT1-OBJECT4 should be redistributed by pre-processing. For example, referring to equation (7), the ratio with which the second object signal OBJECT2 is distributed between the down-mix signals of the L- and R-channels changes from 40:60 to 30:70, and thus, it can be determined that one the fourth (25%) of the part of the second object signal OBJECT2 previously distributed to the L-channel down-mix signal should be moved to the R-channel down-mix signal. This may become more apparent when referring to equation (8):

[Equation 8]

OBJECT1: 55% of the portion of OBJECT1 previously allocated to R must be moved to L

OBJECT2: 25% of the portion of OBJECT1 previously allocated to L must be moved to R

OBJECT3: 50% of the portion of OBJECT1 previously allocated to L must be moved to R

OBJECT4: 50% of the portion of OBJECT1 previously allocated to R must be moved to L

By using equation (8), the signals L_L, L_R, R_L and R_R of FIG. 25 can be represented as indicated by equation (9):

[Equation 9]

The value of each object signal in equation (9) can be represented as the ratio with which the corresponding object signal is distributed between the L and R channels by using the information about the dequantized CLD provided by the OTT block, as indicated by equation (10):

[Equation 10]

The CLD information used in each parsing unit in FIG. 25 may be determined as indicated by equation (11):

[Equation 11]

ε: where 1 is a constant to avoid dividing by zero, for example, 96 dB below the maximum input signal

In this way, CLD and ICC information used in the parser to generate L_L and L_R signals based on the L-channel downmix signal can be determined, and CLD and ICC information used in the parser for generating R_L signals can also be determined and R_R based on the downmix signal of the R channel. If L_L, L_R, R_L, and R_R signals are received, as shown in FIG. 25, L_R and R_R signals can be added, thereby obtaining a pre-processed stereo downmix signal. If the final channel is a stereo channel, L-and R-channel down-mix signals obtained by pre-processing can be output. In this case, the change, if any, of the level of each signal of the object must still be regulated. For this, a predefined module that acts as an ADG module may be further provided. Information for adjusting the level of each signal of an object can be calculated using the same method as that used to calculate information about ADG, and this is described in more detail below. Alternatively, the level of each object signal may be adjusted during the pre-processing operation. In this case, adjusting the level of each object signal can be performed using the same method as that used for ADG processing. Alternative to the embodiment of FIG. 25, the decorrelation operation may be performed by the decorrelator and mixer rather than the parsing units PARSING1 and PARSING2, as shown in FIG. 26, to adjust the correlation between the L and R signals obtained by mixing. As shown in FIG. 26, Pre_L and Pre_R denote L- and R-channel signals obtained by level adjustment. One of the signals Pre_L and Pre_R can be input to the decorrelator and then subjected to the mixing operation performed by the mixer, thereby obtaining a signal with adjusted correlation.

The pre-processed stereo downmix signal may be input to a multi-channel decoder. In order to provide a multi-channel output signal compatible with object position information and playback configuration information set by the end user, not only a pre-processed downmix signal is required, but also channel-based additional information to perform multi-channel decoding. The rest of this document describes in detail how to obtain channel-based supplementary information, again considering the above example. The preprocessed downmix signals L and R, which are input to the multi-channel decoder, can be determined based on equation (5), as indicated by equation (12):

[Equation 12]

The ratio with which each of the first to fourth object signals OBJECT1-OBJECT4 is distributed on the FL-, RL-, C-, FR- and RR-channels can be determined as indicated by equation (13):

[Equation 13]

The preprocessed downmix signals L and R can be expanded to 5.1 channels via MPS (MPEG Surround), as shown in FIG. As shown in FIG. 27, the parameters of the TTT0 TTT block and the OTTA, OTTV and OTTS OTT blocks may require calculation in units of parameter bands, even if the parameter bands are not shown for convenience.

The TTT0 TTT unit can be used in two different modes: an energy-based mode and a prediction mode. When used in an energy-based mode, the TTT0 TTT block requires two pieces of CLD information. When used in prediction mode, the TTT0 TTT block requires two pieces of CPC information and a piece of ICC information.

In order to calculate CLD information in an energy-based mode, the energy ratio of the L, R, and C signals in FIG. 27 can be calculated using equations (6), (10) and (13). The signal energy level L "can be calculated as indicated by equation (14):

[Equation 14]

Equation (14) can also be used to calculate the energy level R "or C. After that, the CLD information used in the TTT0 TTT block can be calculated based on the energy levels of the signals L", R "and C, as indicated by equation (15):

[Equation 15]

Equation (14) can be established based on equation (10). Even if equation (10) determines only how to calculate the energy values for the L channel, the energy values for the R channel can be calculated using equation (10). Thus, CLD and ICC values of the third and fourth OTT blocks can be calculated based on the CLD and ICC values of the first and second OTT blocks. This, however, may not necessarily apply to all tree structures, but only to some tree structures for decoding object signals. Information included in the object bitstream can be transmitted to each OTT block. Alternatively, information included in the object bitstream can only be transmitted to some OTT blocks, and information indicating OTT blocks that have not received the information can be obtained by calculation.

Parameters such as CLD and ICC information can be calculated for the OTTA, OTTV, and OTTC OTT blocks by using the above method. Such multichannel parameters can be input to a multichannel decoder and then subjected to multichannel decoding, thereby obtaining a multichannel signal that is rendered appropriately in accordance with the position information of the object and the playback configuration information required by the end user.

Multichannel parameters may include an ADG parameter if the signal level of the object has not yet been adjusted by preprocessing. The calculation of the ADG parameter later in this document is described in detail by considering again the above example.

When the rendering matrix is set so that the level of the third signal of the object can increase by 10 dB, the level of the fourth signal of the object can decrease by 5 dB, the level of the component of the third signal of the object in L can increase by 10 dB and the level of the component of the fourth signal of the object in L can decrease by 5 dB, the ratio of RatioADG, L energy levels before and after adjusting the levels of the third and fourth signals of the object can be calculated using equation (16):

[Equation 16]

Ratio _{ADG, L} can be determined by substituting equation (10) into equation (16). The Ratio _{ADG, R} ratio for the R channel can also be calculated using equation (16). Each of the ratios Ratio _{ADG, L} and Ratio _{ADG, R} represents a change in the corresponding parameter bands of energy due to adjustment levels of the object signals. Thus, the values ADG (L) and ADG (R) ADG may be calculated using relationships Ratio _{ADG, L} and Ratio _{ADG, R,} as indicated by equation (17):

[Equation 17]

If the parameters ADG (L) and ADG (R) ADG are determined, the parameters ADG (L) and ADG (R) ADG are quantized using the quantization table ADG, and the quantized ADG values are transmitted. If there is a need for further fine-tuning of the ADG (L) and ADG (R) ADG values, the ADG (L) and ADG (R) ADG values can be adjusted by the preprocessor, not the MPS decoder.

The number and interval of parameter bands for representing the object signals in the object bitstream may be different from the number and interval of parameter bands used in a multi-channel decoder. In this case, the parameter bands of the object bitstream can be mapped linearly onto the parameter bands of the multi-channel decoder. More specifically, if a certain parameter band of an object bitstream is distributed across two parameter bands of a multichannel decoder, a linear mapping can be performed so that a certain parameter band of an object bitstream can be divided in accordance with the ratio with which a corresponding parameter band is distributed between two parameter bands of a multichannel decoder. decoder. On the other hand, if more than one parameter band of the object bitstream is included in some parameter band of the multi-channel decoder, the parameter values of the object bitstream can be averaged. Alternatively, parameter band mapping may be performed using an existing multi-channel standard parameter strip mapping table.

When encoding an object is used for teleconferencing, the voices of various people correspond to the signals of the object. The object decoder outputs voices correspondingly corresponding to the object signals to certain speakers. However, when more than one person is speaking at the same time, it is difficult for the object decoder to appropriately distribute people's voices to different speakers using decoding, and rendering people's voices can cause sound distortion and lower sound quality. To take action on this subject, information indicating whether more than one person is speaking at the same time may be included in the bitstream. Then, if it is determined, based on the information that more than one person is talking at the same time, the channel-based bitstream can be modified, so that hard-to-decode signals, almost similar to down-mix signals, can be output to each speaker.

For example, suppose there are three people a, b, and c, and the voices of three people a, b, and c need to be decoded, and thus output to speakers A, B, and C, respectively. When three people a, b and c are simultaneously talking, the voices of three people a, b and c can all be included in the downmix signal, which is obtained by downmixing the object signals representing the voices of three people a, b and c respectively. In this case, information regarding portions of the downmix signal corresponding respectively to the voices of three people a, b and c can be configured as a multi-channel bitstream. Then the downmix signal can be decoded using the usual method of decoding objects, so that the voices of three people a, b and c can be output to the speakers A, B and C, respectively. The outputs of each speaker A, B, and C, however, may be distorted and may thus have a lower percentage of correct recognition than the original downmix signal. In addition, the voices of three people a, b and c cannot be properly isolated from each other. To solve this issue, information indicating that the simultaneous utterances of three people a, b and c are spoken, can be included in the bitstream. Then the transcoder can generate a multi-channel bitstream, so that the down-mix signal obtained by down-mixing the object signals corresponding to the voices of three people a, b and c, respectively, can be output to each loudspeaker A, B and C in the form in which it is . In this way, signal distortion can be prevented.

In fact, when more than one person is talking at the same time, it is difficult to isolate each person’s voice. Therefore, the sound quality may be higher when the downmix signal is output as it is than when the downmix signal is rendered so that voices of different people can be isolated from each other and can be output to different speakers. To this end, the transcoder can generate a multi-channel bitstream, so that the down-mix signal obtained from the simultaneous pronunciations of more than one person can be output to all speakers or that the down-mix signal can be amplified and then output to the speakers.

To indicate whether an object bit-down mixing signal occurs from the simultaneous utterances of one or more people, the object encoder can modify the object bit stream accordingly instead of providing additional information as described above. In this case, the object decoder can perform the usual operation of decoding the object bitstream, so that the downmix signal can be output to the speakers as it is, or that the downmix signal can be amplified, but not to the extent that distortion occurs signal, and then output to the speakers.

Below in this document, three-dimensional information, such as HRTF, that is provided to a multi-channel decoder is described in detail.

When the object decoder operates in binaural mode, the multi-channel decoder in the object decoder also operates in binaural mode. The end user can transmit three-dimensional information, such as HRTF, which is optimized based on the spatial positions of the object signals, to a multi-channel decoder.

More specifically, when there are two object signals, i.e. OBJECT1 and OBJECT2, and two signals of OBJECT1 and OBJECT2 of the object are located in positions 1 and 2, respectively, the generator of the rendering matrix or transcoder can have three-dimensional information indicating the position of signals of OBJECT1 and OBJECT2 of the object. If the generator of the rendering matrix has three-dimensional information indicating the positions of the signals OBJECT1 and OBJECT2 of the object, the generator of the rendering matrix can transmit three-dimensional information indicating the positions of the signals OBJECT1 and OBJECT2 of the object to the transcoder. On the other hand, if the transcoder has three-dimensional information indicating the position of the object signals OBJECT1 and OBJECT2, the rendering matrix generator can only transmit index information corresponding to the three-dimensional information to the transcoder.

In this case, the binaural signal can be generated based on three-dimensional information specifying the positions 1 and 2, as indicated by equation (18):

[Equation 18]

A multi-channel binaural decoder receives binaural sound by performing decoding on the assumption that a 5.1-channel speaker system will be used to reproduce sound, and binaural sound can be represented by equation (19):

[Equation 19]

The L-channel component of the OBJECT1 signal of the object can be represented by equation (20):

[Equation 20]

The component of the R-channel of the signal OBJECT1 of the object and the components of the L- and R-channels of the signal OBJECT2 of the object can all be determined using equation (20).

For example, if the ratio of the energy levels of the OBJECT1 and OBJECT2 signals of the object to the total energy level is a and b, respectively, the ratio of the part of the signal OBJECT1 of the object distributed to the FL channel to the entire signal OBJECT1 of the object is equal to c and the ratio of the part of the signal OBJECT2 of the object distributed to the FL channel , to the whole signal OBJECT2 of the object is d, the ratio with which the signals OBJECT1 and OBJECT2 of the object are distributed to the FL channel is ac: bd. In this case, the HRTF of the FL channel can be determined as indicated by equation (21):

[Equation 21]

In this way, three-dimensional information for use in a multi-channel binaural decoder can be obtained. Since three-dimensional information for use in a multi-channel binaural decoder better represents the actual position of the object signals, it is possible to reproduce binaural signals more vividly through binaural decoding using three-dimensional information for use in a multi-channel binaural decoder than when performing multi-channel decoding using three-dimensional information corresponding to the positions five loudspeakers.

As described above, three-dimensional information for use in a multi-channel binaural decoder can be calculated based on three-dimensional information representing the spatial position of the object signals and information about the ratio of energies. Alternatively, three-dimensional information for use in a multi-channel binaural decoder may be generated by appropriately performing decorrelation by adding three-dimensional information representing the spatial positions of the object signals based on the ICC information of the object signals.

Effect processing can be performed as part of pre-processing. Alternatively, the effect processing result may simply be added to the output of the multi-channel decoder. In the first case, in order to perform effects processing on the object signal, it may be necessary to extract the object signal in addition to dividing the L-channel signal into L_L and L_R and dividing the R-channel signal into R_R and R_L.

More specifically, an object signal may first be extracted from the L- and R-channel signals. Then, the L-channel signal may be divided into L_L and L_R, and the R-channel signal may be divided into R_R and R_L. Processing effects can be performed on the signal of the object. Then the signal of the object with the processed effects can be divided into the components of the L- and R-channels in accordance with the rendering matrix. After that, the L-channel component of the object signal with the processed effects can be added to L_L and R_L, and the R-channel component of the object signal with the processed effects can be added to R_R and L_R.

Alternatively, pre-processed L and R L- and R-channel signals may be generated first. After that, the object signal can be extracted from the pre-processed L and R signals of the L and R channels. After that, the processing of effects on the object signal can be performed, and the result of the processing of effects can be added back to the pre-processed signals of the L- and R-channels.

The signal spectrum of an object can be modified using effects processing. For example, the level of a high-tone part or a low-tone part of an object signal may selectively increase. For this, only a part of the spectrum corresponding to a part with a high tone or a part with a low tone of an object signal can be modified. In this case, information related to the object included in the bitstream of the object may require modification accordingly. For example, if the level of the low-tone part of some object signal increases, the energy of the low-tone part of some object signal can also increase. Thus, energy information included in an object bitstream no longer properly represents the energy of an object signal. To solve this issue, the energy information included in the bitstream of the object can be directly modified in accordance with the change in energy of some signal of the object. Alternatively, the spectrum change information provided by the transcoder can be applied to the formation of a multi-channel bit stream, so that the energy change of some object signal can be reflected in the multi-channel bit stream.

28-33 illustrate diagrams for explaining including a plurality of portions of object-based additional information and a plurality of downmix signals in a portion of additional information and a downmix signal. In the case of a teleconference, it is sometimes necessary to combine a plurality of portions of object-based additional information and a plurality of downmix signals into additional information and a downmix signal. In this case, it is necessary to consider a number of factors.

FIG. 28 illustrates an object encoded bitstream diagram. As shown in FIG. 28, the object-encoded bitstream includes a downmix signal and additional information. The downmix signal is synchronized with additional information. Therefore, an object-encoded bitstream can easily be decoded without taking into account additional factors. However, in the case of combining multiple bit streams into a single bit stream, you must make sure that the down-mix signal of a single bit stream is synchronized with additional information of a single bit stream.

29 is a diagram for explaining a combination of a plurality of object-encoded bitstreams BS1 and BS2. As shown in FIG. 29, positions 1, 2, and 3 indicate frame numbers. To combine multiple downmix signals into a single downmix signal, the downmix signals can be converted to pulse code modulation (PCM) signals, downmix of the PCM signals in the time domain can be performed, and the downmix PCM signal can be converted to the codec format compression. During these processes, a delay d may be generated, as shown in FIG. 29 (b). Therefore, when a bitstream to be decoded is obtained by combining a plurality of bit streams, it must be ensured that the down-mix signal of the bit stream to be decoded is properly synchronized with the additional information of the bit stream to be decoded.

If a delay is set between the down-mix signal and the additional bitstream information, the bitstream can be compensated for by a predetermined value corresponding to the delay. The delay between the downmix signal and the additional bitstream information may vary in accordance with the type of compression codec used to generate the downmix signal. Therefore, a bit indicating the delay, if any, between the down-mix signal and the additional information of the bit stream may be included in the additional information.

30 illustrates combining two bitstreams BS1 and BS2 into a single bitstream when down-mixing signals of bitstreams BS1 and BS2 are generated by different types of codecs or the configuration of the additional information of the bitstream BS1 is different from the configuration of the additional information of the bitstream BS2. As shown in FIG. 30, when the down-mix signals of the bit streams BS1 and BS2 are generated by different types of codecs or the configuration of the additional information of the bitstream BS1 is different from the configuration of the additional information of the bitstream BS2, it can be determined that the bit streams BS1 and BS2 have different delays d1 and d2 signals resulting from converting the downmix signals to time-domain signals and converting the time-domain signals using a single compression codec. In this case, if the bitstreams BS1 and BS2 are simply summed without regard to various signal delays, the downmix signal of the bitstream BS1 may not be aligned correctly with the downmix signal BS2 and the additional information of the bitstream BS1 may not be aligned correctly with the additional information of the bitstream BS2. To solve this issue, the down-mix signal of the bitstream BS1, which is delayed by d1, can be further delayed so that it is synchronized with the down-mix signal of the bitstream BS2, which is delayed by d2. Then, the bitstreams BS1 and BS2 can be combined using the same method of the embodiment of FIG. 30. If there is more than one bitstream to combine, the bitstream that has the greatest delay can be used as the reference bitstream, and then other bitstreams can be further delayed so that they synchronize with the reference bitstream. A bit indicating the delay between the downmix signal and the additional information may be included in the object bitstream.

A bit may be provided indicating whether there is a signal delay in the bitstream. If only the bit information indicates that there is a signal delay in the bit stream, then information determining the signal delay can be further provided. In this way, the amount of information needed to indicate the signal delay, if any, in the bitstream can be minimized.

32 illustrates a diagram for explaining how to compensate for one or two bitstreams BS1 and BS2 having different signal delays by a difference between different signal delays, and in particular, how to compensate for a bitstream BS2 that has a longer signal delay than bitstream BS1. As shown in FIG. 32, the first to third frames of additional information of the bitstream BS1 can all be used as they are. On the other hand, the first to third frames of the additional information of the bitstream BS2 cannot be used as they are, since the first to third frames of the additional information of the bitstream BS2 are not synchronized accordingly with the first to third frames of the additional information of the bitstream BS1 . For example, a second frame of additional information of bitstream BS1 corresponds not only to a part of the first frame of additional information of bitstream BS2, but also to part of a second frame of additional information of bitstream BS2. The ratio of the portion of the second frame of additional information of the bitstream BS2 corresponding to the second frame of additional information of the bitstream BS1 can be calculated with the entire second frame of additional information of the bitstream BS2, and the ratio of the portion of the first frame of the additional information of bitstream BS2 corresponding to the second frame of the additional information of bitstream BS1, with all the first frame of additional information of the bitstream BS2, and the first and second frames of additional information of the bitstream BS2 could m averaged or interpolated based on the calculation results. Thus, the first to third frames of the additional information of the bitstream BS2 can be synchronized accordingly with the first to third frames of the additional information of the bitstream BS1, as shown in FIG. 32 (b). Then, the additional information of the bitstream BS1 and the additional information of the bitstream BS2 can be combined using the method of the embodiment of FIG. 29. The down-mix signals of the bit streams BS1 and BS2 can be combined into a single down-mix signal without the need for delay compensation. In this case, the delay information corresponding to the delay d1 of the signal may be stored in the combined bitstream obtained by combining the bit streams BS1 and BS2.

33 illustrates a diagram for explaining how to compensate for that bitstream of two bitstreams having different signal delays that has a shorter signal delay. As shown in FIG. 33, the first to third frames of the additional information of the BS2 bitstream can all be used as they are. On the other hand, the first to third frames of the additional information of the bitstream BS1 cannot be used as they are, since the first to third frames of the additional information of the bitstream BS1 are not synchronized accordingly with the first to third frames of the additional information of the bitstream BS2 . For example, the first frame of the additional information of the bitstream BS2 corresponds not only to the portion of the first frame of the additional information of the bitstream BS1, but also to the part of the second frame of the additional information of the bitstream BS1. The ratio of the portion of the first frame of the additional information of the bitstream BS1 corresponding to the first frame of the additional information of the bitstream BS2 can be calculated with the entire first frame of the additional information of the bitstream BS1, and the ratio of the portion of the second frame of the additional information of the bitstream BS1 corresponding to the first frame of the additional information of the bitstream BS2 , with the entire second frame of additional information of the bitstream BS1, and the first and second frames of additional information of the bitstream BS1 can middle or interpolated on the basis of calculation results. Thus, the first to third frames of the additional information of the bitstream BS1 can be synchronized accordingly with the first to third frames of the additional information of the bitstream BS2, as shown in FIG. 33 (b). Then, the additional information of the bitstream BS1 and the additional information of the bitstream BS2 can be combined using the method of the embodiment of FIG. 29. The down-mix signals of the bit streams BS1 and BS2 can be combined into a single down-mix signal without the need for delay compensation, even if the down-mix signals have different signal delays. In this case, the delay information corresponding to the delay d2 of the signal may be stored in the combined bitstream obtained by combining the bit streams BS1 and BS2.

If a plurality of object-encoded bit streams are combined into a single bit stream, down-mix signals of object-encoded bit streams may require combining into a single down-mix signal. In order to combine a plurality of downmix signals corresponding to different compression codecs into single downmix signals, the downmix signals can be converted to PCM or frequency domain signals and PCM or frequency domain signals can be summed in the corresponding region. After that, the summation result can be converted using a predefined compression codec. Different signal delays may occur according to whether downmix signals are summed during the PCM operation or are summed in the frequency domain and according to the type of compression codec. Since the decoder cannot easily recognize various signal delays from the bit stream to be decoded, it may be necessary to include delay information in the bit stream defining the various signal delays. Such delay information may represent the number of delay samples in the PCM signal or the number of delay samples in the frequency domain.

The present invention can be implemented as a computer-readable code recorded in a computer-readable recording medium. The computer-readable recording medium may be any type of recording device in which data is stored in a computer-readable manner. Examples of a computer-readable recording medium include read-only memory (ROM), random access memory (RAM), a compact disk, a magnetic tape, a floppy disk, an optical data storage device, and a carrier wave (e.g., data transmission over the Internet). The computer-readable recording medium can be distributed across a plurality of computer systems connected to the network, so that the computer-readable code is written to it and executed from it in a decentralized manner. Functional programs, code, and code segments needed to implement the present invention can be easily interpreted by a person skilled in the art.

As described above, according to the present invention, audio images are localized for each signal of the object by taking advantage of the advantages of object-based encoding and decoding of audio. Thus, it is possible to offer more realistic sounds while playing back object signals. In addition, the present invention can be applied to interactive games, and thus can provide the user with a more realistic virtual reality experience.

Although the present invention has been specifically shown and described with reference to its exemplary embodiments, it is understood by a person skilled in the art that various changes in form and detail can be made therein without departing from the spirit and scope of the present invention as defined by the following claims.

Claims (11)

1. An audio decoding method comprising the following steps:
receiving a downmix signal, object-based additional information and control information, wherein a downmix signal is obtained by downmixing a plurality of object signals and control information that controls the position or level of a specified object signal included in the downmix signal:
extracting metadata, including a description of the object signal, from object-based additional information;
generate parameter information and spatial parameter information by using control information and object-based additional information, the parameter information is used to pre-process the downmix signal, and the spatial parameter information is used to generate a multi-channel audio signal:
pre-processing the position or signal level of the object included in the downmix signal by superimposing the parameter information to the downmix signal: and
generating a multi-channel audio signal by using the pre-processed downmix signal, spatial parameter information and control information,
while the metadata uses a text format. 2. The method of decoding audio according to claim 1, in which the metadata contains at least one of: a number corresponding to the signal of the object, and a description of the signal of the object. 3. The audio decoding method of claim 1, wherein the metadata is included in the header of the object-based additional information. 4. An audio encoding method comprising the following steps:
generating a downmix signal by downmixing a plurality of object signals;
generating object-based additional information from a plurality of object signals;
inserting metadata representing an object signal description into object-based additional information, and
generating a bitstream by combining a downmix signal and object-based additional information into which metadata is inserted. 5. An audio decoding apparatus comprising:
a demultiplexer configured to extract the downmix signal, object-based additional information and control information from the inputted audio signal, wherein the downmix signal is obtained by downmixing a plurality of object signals and control information controlling the position or level of the signal of the object included in the downmix signal:
a parameter converter configured to extract metadata representing the description of the object signal from object-based additional information, and to generate parameter information and spatial parameter information by using control information and object-based additional information, wherein the parameter information is used to preprocess the downmix signal mixing, and spatial parameter information is used to generate multi-channel sound signal:
a preprocessor configured to pre-process the position or signal level of the object included in the downmix signal by superimposing the parameter information to the downmix signal: and
a multi-channel decoder configured to upmix a pre-processed downmix signal by using spatial parameter information,
while the metadata uses a text format. 6. The device according to claim 5, in which the metadata contains at least one of: a number corresponding to the signal of the object, and a description of the signal of the object. 7. The device according to claim 5, in which metadata is included in the header of the object-based additional information. 8. The processor-readable recording medium on which the program is recorded to execute the method according to claim 1 in the processor. 9. A computer-readable recording medium having a computer program recorded thereon for performing an audio decoding method comprising:
receiving a downmix signal, object-based additional information, and control information, wherein the downmix signal is obtained by downmixing a plurality of object signals and control information controlling a position or level of said object signal included in the downmix signal;
extracting metadata, including an object signal description, from object-based additional information;
generating parameter information and spatial parameter information by using control information and object-based additional information, wherein the parameter information is used for preprocessing the downmix signal, and the spatial parameter information is used to generate a multi-channel audio signal;
pre-processing the position or signal level of the object included in the downmix signal by superimposing the parameter information to the downmix signal; and
upmixing the pre-processed downmix signal by using spatial parameter information;
while the metadata uses a text format. 10. The computer-readable recording medium of claim 9, wherein the metadata is included in the header of the object-based additional information. 11. A computer-readable recording medium having a computer program recorded thereon for performing an audio encoding method comprising:
generating a downmix signal by downmixing a plurality of object signals:
generating object-based additional information from a plurality of object signals; and
inserting metadata, including an object signal description, into object-based additional information;
generating a bitstream by combining a downmix signal and object-based additional information that includes metadata.

Images