RU2417549C2 - Audio signal processing method and device - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: disclosed is a method of processing an audio signal, involving reception of a step-down mixed signal, information on the object and information on mixing; generation of information on step-down mixing processing using information on the object and information on mixing; processing the step-down mixed signal using information on mixing; and generation of multichannel information using information on the object and information on mixing, where the number of channels for the step-down mixed signal is equal to the number of channels for the processed step-down mixed signal.

EFFECT: providing unlimited gain control when processing an audio signal and object panning based on the user selection.

19 cl, 21 dwg, 4 tbl

Description

Technical field

The present invention relates to a method and apparatus for processing an audio signal, and more particularly, to a method and apparatus for decoding an audio signal received on a digital medium, for example, a broadcast signal and so on.

State of the art

When mixing down a few audio objects into a monophonic or stereo signal, parameters can be extracted from individual object signals. These parameters can be used in an audio decoder, and the positioning / panning of individual sources can be controlled by the user.

Disclosure of invention

Technical problem

However, in order to control the individual signals of the objects, changing the position / panning of the individual sources included in the downmix signal must be performed accordingly.

However, for backward compatibility with respect to the channel-oriented decoding method (for example, MPEG Surround), the object parameter must be flexibly transformed into the multi-channel parameter required during the upmixing process.

Technical solution

Accordingly, the present invention is directed to a method and apparatus for processing an audio signal that substantially eliminates one or more problems due to limitations and disadvantages of the prior art.

The purpose of the present invention is to provide a method and apparatus for processing an audio signal in order to infinitely control the gain and panning of an object.

Another objective of the present invention is to provide a method and apparatus for processing an audio signal in order to control gain and panning of an object based on a user's selection.

Additional advantages, objects, and features of the invention will be set forth in part in the description that follows, and in part will become apparent to those of ordinary skill in the art after examination of the following description, or may be studied by putting the invention into practice. The objectives and other advantages of the invention can be realized and achieved through the design shown in detail in the description and claims, as well as the accompanying drawings.

Useful Results

The present invention provides the following results or advantages.

First, the present invention can provide a method and apparatus for processing an audio signal to infinitely control the gain and panning of an object.

Secondly, the present invention can provide a method and apparatus for processing an audio signal to control amplification and panning of an object based on a user's selection.

Description of drawings

The accompanying drawings, which are included to provide an additional understanding of the invention and form part of this application, illustrate embodiments of the invention and together with the description serve to explain the principle of the invention. In the drawings:

1 is a typical block diagram for explaining a basic idea of reproducing a downmix signal based on a reproduction configuration and user control.

2 is an exemplary block diagram of an apparatus for processing an audio signal according to one embodiment of the present invention, corresponding to a first circuit.

FIG. 3 is an exemplary block diagram of an apparatus for processing an audio signal according to another embodiment of the present invention corresponding to a first circuit.

4 is a typical block diagram of an apparatus for processing an audio signal according to one embodiment of the present invention, corresponding to a second circuit.

5 is a typical block diagram of an apparatus for processing an audio signal according to another embodiment of the present invention, corresponding to the second scheme.

6 is a typical block diagram of an apparatus for processing an audio signal according to another embodiment of the present invention, corresponding to the second scheme.

7 is a typical block diagram of an apparatus for processing an audio signal according to one embodiment of the present invention, corresponding to a third circuit.

FIG. 8 is an exemplary block diagram of an apparatus for processing an audio signal according to another embodiment of the present invention, corresponding to a third circuit.

Fig.9 is a typical block diagram for explaining the main idea of the playback module.

10A-10C are exemplary block diagrams of a first embodiment of a downmix processing module illustrated in FIG. 7.

11 is a typical block diagram of a second embodiment of a downmix processing module illustrated in FIG. 7.

FIG. 12 is a typical block diagram of a third embodiment of a downmix processing module illustrated in FIG. 7.

FIG. 13 is a typical block diagram of a fourth embodiment of a downmix processing module illustrated in FIG. 7.

14 is a typical block diagram of a binary audio stream structure of a compressed audio signal according to a second embodiment of the present invention.

15 is a typical block diagram of an apparatus for processing an audio signal according to a second embodiment of the present invention.

FIG. 16 is an exemplary block diagram of a binary audio stream structure of a compressed audio signal according to a third embodiment of the present invention. FIG.

17 is an exemplary block diagram of an apparatus for processing an audio signal according to a fourth embodiment of the present invention.

Fig. 18 is a typical block diagram for explaining a transmission scheme for a variable object type.

19 is an exemplary block diagram of an apparatus for processing an audio signal according to a fifth embodiment of the present invention.

The best embodiment of the invention

To achieve these goals and other advantages and in accordance with the purpose of the invention, which is implemented and broadly described in this document, a method for processing an audio signal is provided, comprising: receiving a down-mixed signal, object information and mixing information; generating information about downmix processing using object information and mixing information; processing the downmixed signal using downmix processing information and generating multi-channel information using object information and mixing information in which the number of channels of the downmixed signal is equal to the number of channels of the processed downmix .

According to the present invention, object information includes at least one of object level information and object correlation information.

According to the present invention, the downmix processing information corresponds to information for controlling panning of an object if the number of channels of the downmix signal corresponds to at least two.

According to the present invention, downmix processing information corresponds to information for controlling gain of an object.

According to the present invention, the processing of the downmix signal is performed using a 2 × 2 module if the downmix signal corresponds to a stereo signal.

According to the present invention, one channel of the processed downmix signal corresponds to a combination of one channel of the downmix signal multiplied by the first gain and another channel of the downmix signal multiplied by the second gain, if the signal is mixed down corresponds to a stereo signal.

According to the present invention, further comprising generating an output signal in a time domain using a processed downmix signal.

According to the present invention, the downmix signal corresponds to a subband region signal generated by a set of subband analysis filters.

According to the present invention, multi-channel information includes at least one of channel level information and channel correlation information.

According to the present invention, further comprising generating a multi-channel signal using a downmix processed signal and multi-channel information.

According to the present invention, mixing information is generated using at least one of the position information of the object and the playback configuration information.

According to the present invention, a downmix signal is received as a broadcast signal.

According to the present invention, a downmix signal is received on a digital medium.

In another aspect of the present invention, there is provided a method for processing an audio signal, comprising: receiving a downmix signal, object information and mixing information; decomposing the downmix signal into a subband signal; generating information about downmix processing using object information and mixing information and processing a subband signal using downmix processing information; generating an output signal using the processed subband signal, where the number of channels of the downmix signal is equal to the number of output signals, and the output signal corresponds to a time-domain signal.

In another aspect of the present invention, there is provided a computer-readable medium with instructions stored thereon which, when executed by a processor, cause the processor to perform operations comprising: receiving a downmix signal, object information and mixing information; generating information about downmix processing using object information and mixing information; processing the downmixed signal using downmix processing information and generating multi-channel information using object information and mixing information in which the number of channels of the downmixed signal is equal to the number of channels of the processed downmix .

In another aspect of the present invention, there is provided a computer-readable medium with instructions stored thereon which, when executed by a processor, cause the processor to perform operations comprising: receiving a downmix signal, object information and mixing information; decomposing the downmix signal into a subband signal; generating information about downmix processing using object information and mixing information and processing a subband signal using downmix processing information; generating an output signal using the processed subband signal, where the number of channels of the downmix signal is equal to the number of output signals, and the output signal corresponds to a time-domain signal.

In another aspect of the present invention, there is provided an apparatus for processing an audio signal comprising: an information generating module receiving information about an object and mixing information and generating information about downmix processing using object information and mixing information and generating multi-channel information using information about the facility and mixing information; and a downmix processing processing unit receiving the downmix signal and downmix processing information and a downmix processing signal using the downmix processing information; where the number of channels of the downmix signal is equal to the number of channels of the processed downmix signal.

In another aspect of the present invention, there is provided an apparatus for processing an audio signal, comprising: an information generating module receiving a downmix signal, object information and mixing information, the information generating module generating down mixing information using the object information and information about mixing; and a downmix processing module, decomposing the downmix signal into a subband signal, processing the subband signal using downmix processing information, and generating an output signal using the processed subband signal, where the number of channels of the downmix signal is the number of output signals, and the output signal corresponds to a time-domain signal.

In another aspect of the present invention, there is provided a method for processing an audio signal, the method comprising: receiving a downmix signal using a signal of a plurality of objects; generating information about an object representing the relationship between signals of a plurality of objects using signals from a plurality of objects and a downmix signal, and transmitting a downmix signal and information about an object where a downmix signal is allowed to be processed by a signal mixed with by decreasing, so that the number of channels of the signal mixed with decreasing is equal to the number of processed signals mixed with decreasing.

It should be understood that both the foregoing general description and the following detailed description of the present invention are illustrative and explanatory and are intended to provide further explanation of the claimed invention.

Embodiments for the invention

Detailed reference will now be made to preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Where, possibly, the same reference numbers in the drawings will be used to refer to the same or similar parts.

Before describing the present invention, it should be noted that most of the terms disclosed in the present invention correspond to general terms well known in the art, but some terms are selected by the applicant as necessary and will be disclosed below in the following description of the present invention. Therefore, it is preferred that the terms defined by the applicant be understood based on their meanings in the present invention.

In particular, “parameter” in the following description means information including values, parameters in the narrow sense, coefficients, elements, and so on. Hereinafter, the term “parameter” will be used instead of the term “information” as an object parameter, a mixing parameter, a downmix processing parameter, and so on, which does not impose a limitation on the present invention.

In mixing the signals of several channels or signals of an object, an object parameter and a spatial parameter can be extracted. A decoder can generate an output signal using a downmix signal and an object parameter (or spatial parameter). The output signal may be reproduced by the decoder based on the playback configuration and user control. The reproduction process is explained below in detail with reference to FIG.

1 is a typical diagram for explaining a basic idea of reproducing a downmix signal based on a reproduction configuration and user control. 1, decoder 100 may include a reproduction information generating unit 110 and a reproduction unit 120, and may also include a renderer 110a and synthesis 120a instead of a reproduction information generating unit 110 and reproduction unit 120.

The playback information generating unit 110 may be configured to receive additional information from the encoder including an object parameter or spatial parameter, as well as to obtain a playback configuration or user control from a device setting or user interface. An object parameter may correspond to a parameter extracted in mixing with decreasing at least one signal of the object, and a spatial parameter may correspond to a parameter extracted in mixing with decreasing at least one channel signal. In addition, type information and characteristic information for each object may be included in additional information. Type information and characteristic information may describe the name of the instrument, the name of the artist, and so on. The playback configuration may include speaker position and surrounding information (virtual speaker position), and user control may correspond to control information entered by the user to control the positions of objects and object amplifications, and may also correspond to control information to control the playback configuration. Meanwhile, the playback configuration and user control can be represented as mixing information, which does not impose a limitation on the present invention.

The reproduction information generating unit 110 may be configured to generate reproduction information using mixing information (reproduction configuration and user control) and received additional information. The playback module 120 may be configured to generate a multi-channel parameter using the playback information if audio down-mixing is not transmitted (abbreviated as “down-mixed signal”), and generating multi-channel signals using the playback information and the down-mixed signal if mixing is transmitted with decreasing audio signal.

The renderer 110a may be configured to generate multi-channel signals using mixing information (playback configuration and user control) and received additional information. Synthesis 120a may be configured to synthesize multi-channel signals using multi-channel signals generated by the renderer 110a.

As previously stated, a decoder can reproduce a downmix signal based on a reproduction configuration and user control. Meanwhile, in order to control individual object signals, the decoder can obtain the object parameter as additional information and control the object’s panning and object gain based on the transmitted object parameter.

1. Control gain and panning of object signals

Variable methods for controlling individual object signals may be provided. First, if the decoder receives an object parameter and generates individual object signals using the object parameter, then individual object signals can be controlled based on mixing information (playback configuration, object level, etc.).

Secondly, if the decoder generates a multi-channel parameter for input into the multi-channel decoder, then the multi-channel decoder can decompose the down-mixed signal received from the encoder using the multi-channel parameter. The above second method can be classified into three types of circuits. In particular, 1) the use of a traditional multi-channel decoder, 2) a change to a multi-channel decoder, 3) mixing processing with lowering audio signals before being introduced into a multi-channel decoder can be provided. A conventional multi-channel decoder may correspond to channel-oriented spatial audio coding (eg, MPEG Surround decoder), which does not impose a limitation on the present invention. Details of the three types of schemes will be explained below.

1.1 Using a multi-channel decoder

The first scheme can use the traditional multi-channel decoder as is, without changing the multi-channel decoder. First, the case of using ADG (arbitrary amplification of a signal mixed down) to control the gain of objects and the case of using the configuration 5-2-5 to control panning of an object is explained below with reference to FIG. Next, the case of communication with the scene re-mixing module is explained below with reference to FIG. 3.

2 is an exemplary block diagram of an apparatus for processing an audio signal according to one embodiment of the present invention, corresponding to a first circuit. 2, an apparatus 200 for processing an audio signal (hereinafter simply referred to as a “decoder 200”) may include an information generation module 210 and a multi-channel decoder 230. The information generation module 210 may receive additional information including an object parameter from an encoder and mixing information from the user interface and can form a multichannel parameter, including arbitrary amplification of the signal mixed with decreasing, or the gain of the gain change (hereinafter simply “ADG”). The ADG may describe the ratio of the first gain estimated based on the mixing information and the object information to the second gain estimated based on the object information. In particular, the information generating module 210 can generate an ADG only if the downmix signal corresponds to a monophonic signal. The multi-channel decoder 230 may receive down-mixing of the audio signal from the encoder and the multi-channel parameter from the information generating unit 210 and may generate the multi-channel output signal using the down-mixed signal and the multi-channel parameter.

The multi-channel parameter may include the difference in channel levels (hereinafter abbreviated as “CLD”), inter-channel correlation (hereinafter abbreviated as “ICC”), and the channel prediction coefficient (hereinafter abbreviated as “CPC”).

Since the CLD, ICC, and CPC describe the intensity difference or correlation between the two channels, they are designed to control the pan and correlation of an object. You can control the position of the object and the diffuseness (sonicity) of the object using CLD, ICC, etc. Meanwhile, the CLD describes the relative level difference instead of the absolute level, and the energy of the two separated channels is conserved. Therefore, you cannot control the amplification of objects by manipulating the CLD, etc. In other words, a particular object cannot be muted or increased in volume using CLD, etc.

In addition, the ADG describes a time and frequency dependent gain for controlling a correction factor by a user. If this correction factor is applied, it is possible to control the variation of the downmix signal before multi-channel decomposition. Therefore, if the ADG parameter is received from the information generation module 210, the multi-channel decoder 230 can be controlled by object amplifications with a specific time and frequency using the ADG parameter.

The case when the received stereo signal, mixed down, is output as a stereo channel, can be specified by the following formula 1.

[Formula 1]

where x [] are the input channels, y [] are the output channels, g _x are the amplifications, and w _xx is the weight.

It is necessary to control the cross-link between the left channel and the right channel to pan the object. In particular, part of the left channel in the signal mixed with decreasing can be output as the right channel of the output signal, part of the right channel in the signal mixed with decreasing can be output as the left channel of the output signal. In the formula 1, w ₁₂ and w ₂₁ may be cross components.

The above case corresponds to a 2-2-2 configuration, which means 2-channel input, 2-channel transmission and 2-channel output. To perform a 2-2-2 configuration, a 5-2-5 configuration (2-channel input, 5-channel transmission and 2-channel output) in conventional channel-oriented spatial sound coding (e.g., MPEG Surround) can be used. First, in order to output 2 channels for a 2-2-2 configuration, a channel of 5 output channels in a 5-2-5 configuration can be set to a blocked channel (false channel). In order to obtain cross-coupling between 2 transmitted channels and 2 output channels, the above-mentioned CLDs and CPCs can be adjusted. The gain g _x in formula (1) is obtained using the above-mentioned ADG, and the weight coefficient w ₁₁ ~ w ₂₂ in formula (1) is obtained using CLD and CPC.

In implementing a 2-2-2 configuration using a 5-2-5 configuration, in order to reduce complexity, a default mode of conventional spatial audio coding can be applied. Since the default CLD characteristic is assumed to output 2 channels, it is possible to reduce the amount of computation if the default CLD is used. In particular, since it is not necessary to synthesize a false channel, the amount of computation can be significantly reduced. Therefore, applying the default mode is correct. In particular, only the default CLD of 3 CLDs (corresponding to 0, 1, and 2 in the MPEG Surround standard) is used for decoding. On the other hand, 4 CLDs from the left channel, right channel, and center channel (corresponding to 3, 4, 5, and 6 in the MPEG Surround standard) and 2 ADGs (corresponding to 7 and 8 in the MPEG Surround standard) are formed to control the object. In this case, the CLDs corresponding to 3 and 5, which describe the difference in channel levels between the left channel plus the right channel and the center channel ((l + r) / c), should be correctly set to 150 dB (almost infinite) to mute the center channel. And in order to realize cross-coupling, energy-based decomposition or prediction-based decomposition can be performed, which is caused if the TTT mode ("bsTttModeLow" in the MPEG Surround standard) corresponds to the energy-based mode (with subtraction, matrix compatibility is enabled) (3rd mode) or prediction mode (1st mode or 2nd mode).

FIG. 3 is an exemplary block diagram of an apparatus for processing an audio signal according to another embodiment of the present invention corresponding to a first circuit. 3, an apparatus 300 for processing an audio signal according to another embodiment of the present invention (hereinafter simply a decoder 300) may include an information generating module 310, a scene reproducing module 320, a multi-channel decoder 330, and a scene remixing module 350.

The information generating module 310 may be configured to obtain additional information including an object parameter from the encoder if the downmix signal corresponds to a mono channel signal (that is, the number of downmix channels is “1”) can receive mixing information from the user interface and can form a multi-channel parameter using additional information and mixing information. The number of downmix channels can be estimated based on the flag information included in the additional information, as well as the downmix signal itself and the user's choice. The information generating module 310 may have the same configuration as the above information generating module 210. The multi-channel parameter is input to the multi-channel decoder 330, the multi-channel decoder 330 may have the same configuration as the above-mentioned multi-channel decoder 230.

The scene reproducing module 320 may be configured to obtain additional information including an object parameter from the encoder if the downmix signal corresponds to a non-mono channel signal (that is, the number of downmix channels is greater than "2") can receive mixing information from the interface user and can configure the parameter re-mixing, using additional information and information about mixing. The re-mix parameter corresponds to the parameter to re-mix the stereo channel and produce more than 2-channel results. The remix parameter is input to the scene remix module 350. The scene remixing unit 350 may be configured to remix the downmix signal using the remix parameter if the downmix signal is more than 2-channel signal.

In short, two paths could be considered as separate implementations for individual applications in the decoder 300.

1.2 Changing the multi-channel decoder

The second circuit may modify a conventional multi-channel decoder. First, the case of using the virtual output to control the amplification of objects and the case of changing the settings of the device for controlling the panning of the object with reference to FIG. Next, a case of performing TBT functionality (2 × 2) in a multi-channel decoder will be explained with reference to FIG. 5.

4 is a typical block diagram of an apparatus for processing an audio signal according to one embodiment of the present invention, corresponding to a second circuit. 4, an apparatus 400 for processing an audio signal according to one embodiment of the present invention, corresponding to a second circuit (hereinafter simply “decoder 400”), may include an information generating unit 410, an internal multi-channel synthesis 420, and an output conversion unit 430. The internal multi-channel synthesis 420 and the output conversion module 430 may be included in the synthesis module.

The information generating module 410 may be configured to obtain additional information, including an object parameter from the encoder and a downmix processing parameter from the user interface. And the information generating module 410 may be configured to generate a multi-channel parameter and device setting information using additional information and mixing information. The multi-channel parameter may have the same configuration as the multi-channel parameter mentioned above. Therefore, the details of the multi-channel parameter in the following description will be omitted. The device configuration information may correspond to the parameterized HRTF for binaural processing, which will be explained in the description "1.2.2 Using the device configuration information".

The internal multi-channel synthesis 420 may be configured to receive a multi-channel parameter and device setting information from the parameter generating unit 410 and the downmix signal from the encoder. The internal multi-channel synthesis 420 may be configured to generate a temporary multi-channel output including a virtual output, which will be explained in the description of "1.2.1 Using the virtual output".

1.2.1 Using virtual output

Since a multi-channel parameter (e.g., CLD) can control the panning of an object, it is difficult to control the gain of the object, as well as the panning of the object using a traditional multi-channel decoder.

Meanwhile, in order to control the gain of the object, the decoder 400 (especially the internal multi-channel synthesis 420) can convert the relative energy of the object into a virtual channel (for example, a central channel). The relative energy of the object corresponds to the energy that needs to be reduced. For example, to mute an object, decoder 400 may convert more than 99.9% of the object’s energy into a virtual channel. Then, the decoder 400 (in particular, the output conversion module 430) does not output a virtual channel into which the remaining energy of the object is converted. In conclusion, if more than 99.9% of the object is converted to a virtual channel that is not output, the desired object can be almost muted.

1.2.2 Using device setup information

Decoder 400 may adapt device setup information to control object panning and object gain. For example, a decoder may be configured to generate a parameterized HRTF for binaural processing in the MPEG Surround standard. Parameterized HRTF can be variable according to the device setting. It can be assumed that the signals of objects can be controlled in accordance with the following formula 2.

[formula 2]

where obj _k are the signals of the objects, L _new and R _new are the desired stereo signal and a _k and b _k are the coefficients for controlling the object.

The object information about the signals obj _{k of the} objects can be estimated from the parameter of the object included in the transmitted additional information. The coefficients a _k , b _k, which are set in accordance with the gain of the object and the panning of the object, can be estimated from the mixing information. The desired object gain and object panning can be adjusted using the coefficients a _k , b _k .

The coefficients a _k , b _k can be set to match the HRTF parameter for binaural processing, which will be explained in detail below.

In the MPEG Surround standard (configuration 5-1-5 ₁ ) (from document ISO / IEC FDIS 23003-1: 2006 (E), Information Technology - MPEG Audio Technologies - Part1: MPEG Surround) binaural processing is as follows.

[formula 3]

where y _B is the output, the matrix H is the transformation matrix for binaural processing.

[formula 4]

Elements of the matrix H are defined as follows:

[formula 5]

[formula 6]

[formula 7]

1.2.3 Execution of TBT (2 × 2) functionality in a multi-channel decoder

5 is a typical block diagram of an apparatus for processing an audio signal according to another embodiment of the present invention, corresponding to the second scheme. 5 is a typical block diagram of the functionality of TBT in a multi-channel decoder. 5, a TBT module 510 may be configured to receive input signals and TBT control information and to generate output signals. TBT module 510 may be included in decoder 200 of FIG. 2 (or, in particular, in multi-channel decoder 230). Multi-channel decoder 230 can be implemented in accordance with the MPEG Surround standard, which does not impose a limitation on the present invention.

[formula 9]

where x are input channels, y are output channels, and w is weight.

Output y ₁ may correspond to a combination of input x ₁ from the downmix signal multiplied by the first gain factor w ₁₁ and input x ₂ multiplied by the second gain factor w ₁₂ .

The TBT control information inputted to the TBT module 510 includes elements that can make up the weight w (w ₁₁ , w ₁₂ , w ₂₁ , w ₂₂ ).

In the MPEG Surround standard, the OTT (One-in-Two) module and the TTT (Two-in-Three) module are not suitable for remixing an input signal, although the OTT module and TTT module can decompose the input signal.

To remix the input signal, a TBT module 510 (2 × 2) (hereinafter abbreviated as “TBT module 510”) may be provided. The TBT module 510 may be configured to receive a stereo signal and output a remixed stereo signal. Weight w can be formed using CLD and ICC.

If the weight element w ₁₁ ~ w ₂₂ is transmitted as TBT control information, then the decoder can control the gain of the object as well as the pan of the object using the received weight element. In the transfer of the weight element w, a variable circuit may be provided. First, the TBT control information includes a cross term like w ₁₂ and w ₂₁ . Secondly, TBT control information does not include a cross term like w ₁₂ and w ₂₁ . Third, the number of members in TBT control information is adaptively changing.

First you need to take a cross term like w ₁₂ and w ₂₁ to control the panning of an object when the left signal of the input channel goes to the right signal of the output channel. In the case of N input channels and M output channels, members whose number is N × M can be transmitted as TBT control information. Members can be quantized based on the CLD parameter quantization table provided in MPEG Surround, which does not limit the present invention.

Secondly, while the left object is shifted to the right position (i.e., when the left object moves to the more left position or the left position near the center position, or when only the level of the object is adjusted), it is not necessary to use the cross term. In this case, the transfer is carried out in the correct way, without a cross member. In the case of N input channels and M output channels, N terms may be transmitted.

Third, the amount of TBT control information adaptively varies according to the need for a cross term to reduce the bit rate of TBT control information. The cross_flag flag information indicating whether there is a cross member is specified for transmission as TBT control information. The value of the cross_flag flag information is shown in the following table 1.

Table 1
Cross_flag value cross_flag Value 0 No cross member (includes only non-cross member)
(only w ₁₁ and w ₂₂ are present) one Includes Cross Member
(w ₁₁ , w ₁₂ , w ₂₁ and w ₂₂ are present)

If "cross_flag" is 0, then the TBT control information does not include a cross term, only a non-cross term like w ₁₁ and w _{22 is} present. Otherwise ("cross_flag" is 1), the TBT control information includes a cross term.

In addition, the flag information "reverse_flag" indicating whether there is a cross member or non-cross member is set for transmission as TBT control information. The meaning of the reverse_flag flag information is shown in the following table 2.

table 2
Reverse_flag value reverse_flag Value 0 No cross member (includes only non-cross member)
(only w ₁₁ and w ₂₂ are present) one Cross member only
(only w ₁₂ and w ₂₁ are present)

If "reverse_flag" is 0, then the TBT control information does not include a cross term, only a non-cross term like w ₁₁ and w _{22 is} present. Otherwise ("reverse_flag" is 1), the TBT control information includes only the cross term.

Moreover, the side_flag flag information indicating whether there is a cross member and a non-cross member is set for transmission as TBT control information. The value of the side_flag flag information is shown in the following table 3.

Table 3
Side_flag value side_flag Value 0 No cross member (includes only non-cross member)
(only w ₁₁ and w ₂₂ are present) one Includes Cross Member
(w ₁₁ , w ₁₂ , w ₂₁ and w ₂₂ are present) 2 The opposite
(only w ₁₂ and w ₂₁ are present)

Since table 3 corresponds to the combination of table 1 and table 2, details of table 3 will be omitted.

1.2.4 Performing TBT (2 × 2) functionality in a multi-channel decoder by changing the stereo decoder

The case "1.2.2 Using the device setup information" can be performed without changing the stereo decoder. Below with reference to Fig.6 shows the implementation of the functionality of TBT by changing the stereo decoder used in the MPEG Surround decoder.

6 is a typical block diagram of an apparatus for processing an audio signal according to another embodiment of the present invention, corresponding to the second scheme. In particular, the audio signal processing apparatus 630 shown in FIG. 6 may correspond to a binaural decoder included in the multi-channel decoder 230 of FIG. 2 or the synthesis module of FIG. 4, which does not limit the present invention.

An apparatus 630 for processing an audio signal (hereinafter referred to as “binaural decoder 630”) may include a 632 QMF analysis, a parameter transform 634, a spatial synthesis 636, and a 638 QMF synthesis. Elements of the binaural decoder 630 may have the same configuration as the MPEG Surround binaural decoder MPEG Surround. For example, spatial synthesis 636 can be configured consisting of 1 2 × 2 matrix (filter), in accordance with formula 10:

[formula 10]

where y ₀ are the input channels of the QMF domain, and y _B are binaural output channels, k represents the channel index of the hybrid QMF, and i is the tap index in the HRTF filter, and n is the QMF interval index. Binaural decoder 630 may be configured to perform the aforementioned functionality described in "1.2.2 Using Device Setting Information". However, the elements h _ij can be formed using a multi-channel parameter and mixing information instead of a multi-channel parameter and HRTF parameter. In this case, the binaural decoder 600 may perform the functionality of the TBT module 510 in FIG. 5. Details of binaural decoder 630 elements will be omitted.

The binaural decoder 630 may operate in accordance with the binaural_flag flag information. In particular, binaural decoder 630 may be dispensed with if the binaural_flag flag information is “0”, otherwise (binaural_flag is “1”), binaural decoder 630 may operate as follows.

Table 4
Binaural_flag value binaural_flag Value 0 Non binaural mode (binaural decoder disabled) one binaural mode (binaural decoder is on)

1.3 Processing downmix audio mixing before entering a multi-channel decoder

The first scheme for using a traditional multi-channel decoder is explained in subclause "1.1", the second scheme for changing a multi-channel decoder is explained in subclause "1.2". A third circuit for processing down-mix mixing before being input to a multi-channel decoder will be explained below.

7 is a typical block diagram of an apparatus for processing an audio signal according to one embodiment of the present invention, corresponding to a third circuit. FIG. 8 is an exemplary block diagram of an apparatus for processing an audio signal according to another embodiment of the present invention, corresponding to a third circuit. 7, an apparatus 700 for processing an audio signal (hereinafter simply “decoder 700”) may include an information generating unit 710, a downmix processing unit 720, and a multi-channel decoder 730. Referring to FIG. 8, an apparatus 800 for processing an audio signal (hereinafter simply, “decoder 800”) may include an information generating module 810 and a multi-channel synthesis module 840 having a multi-channel decoder 830. Decoder 800 may be another feature of the decoder 700. In other words, the information generating module 810 has The same configuration as the information generation module 710, the multi-channel decoder 830 has the same configuration as the multi-channel decoder 730, and the multi-channel synthesis module 840 may have the same configuration as the downmix processing module 720 and the multi-channel module 730. Therefore elements of decoder 700 will be explained in detail, and details of elements of decoder 800 will be omitted.

The information generating module 710 may be configured to obtain additional information, including an object parameter from the encoder and mixing information from the user interface, and to generate a multi-channel parameter for output to the multi-channel decoder 730. From this point of view, the information generating module 710 has the same configuration, as the above-mentioned information generation module 210 of FIG. 2. The downmix processing processing parameter may correspond to a parameter for controlling object gain and object panning. For example, you can change either the position of the object or the gain of the object if the signal of the object is in the left channel and the right channel. You can also reproduce the signal of the object, which should be in the opposite position, if the signal of the object is only in one of the left channel and the right channel. In order for these cases to be fulfilled, the downmix processing module 720 may be a TBT module (2 × 2 matrix operation). If the information generating module 710 can be configured to generate the ADG described with reference to FIG. 2 to control the gain of the object, then the downmix processing processing parameter may include a parameter for controlling the panning of the object rather than the gain of the object.

In addition, the information generation module 710 can be configured to receive HRTF information from the HRTF database and to generate an additional multi-channel parameter including the HRTF parameter for input to the multi-channel decoder 730. In this case, the information generation module 710 can generate a multi-channel parameter and an additional multi-channel parameter in the same region of the subband and transmit them synchronously with each other to multi-channel decoder 730. An additional multi-channel parameter, including a pair HRTF, will be explained in detail in subparagraph "3. Processing in binaural mode."

Downmix processing module 720 may be configured to receive downmix audio from an encoder and downmix processing parameter from information generation module 710 and to decompose a subband signal using a set of subband analysis filters. The downmix processing module 720 may be configured to generate the processed downmix signal using the downmix signal and the downmix processing parameter. In this processing, a downmix signal can be pre-processed to control object panning and object gain. The processed downmix signal may be input to a multi-channel decoder 730 for decomposition.

In addition, the processed signal, mixed down, with the same success can be output and play through the speaker. In order to directly output the processed signal through the speakers, the downmix processing module 720 may perform a set of synthesis filters using the proposed subband domain signal and output the time domain PCM signal. By selecting a user, one can choose whether to output directly as a PCM signal or input to a multi-channel decoder.

Multichannel decoder 730 may be configured to generate a multichannel output signal using a downmix processed signal and a multichannel parameter. The multi-channel decoder 730 may introduce a delay when the processed downmix signal and the multi-channel parameter are input to the multi-channel decoder 730. The processed downmix signal can be synthesized in the frequency domain (eg, QMF region, hybrid QMF region, etc. .), and a multichannel parameter can be synthesized in the time domain. MPEG Surround introduces delay and timing for communication with HE-AAC. Therefore, the multi-channel decoder 730 may introduce a delay in accordance with the MPEG Surround standard.

The configuration of the downmix processing processing unit 720 will be explained in detail with reference to FIGS. 9 to 13.

1.3.1 General case and special cases of the downmix processing module

Fig.9 is a typical block diagram for explaining the main idea of the playback module. 9, a reproducing module 900 may be configured to generate M output signals using N input signals, a reproduction configuration, and user control. N input signals can correspond to either object signals or channel signals. In addition, N input signals can correspond to either an object parameter or a multi-channel parameter. The configuration of the playback module 900 may be implemented in one of the downmix processing module 720 of FIG. 7, the aforementioned reproduction module 120 of FIG. 1, and the aforementioned renderer 110a of FIG. 1, which does not limit the present invention.

If the playback module 900 can be configured to directly generate M channel signals using N object signals without summing the individual object signals corresponding to a specific channel, then the configuration of the playback module 900 can be represented by the following formula 11.

[formula 11]

C i is the i-th channel signal, O _j is the j-th input signal, and R _ji is the matrix transforming the j-th input signal into the i-th channel.

If the matrix R is divided into a component E of energy and a component of decorrelation, then formula 11 can be represented as follows.

[formula 12]

You can control the position of the object using the energy component E, and you can control the diffuseness of the object using the decorrelation component D.

Assuming that only the i-th input signal is input, which should be output through the j-th channel and the k-th channel, formula 12 can be represented as follows.

[formula 13]

α _{j_i} is the gain part converted to the j-th channel, β _{k_i} is the gain part converted to the k-th channel, θ is the diffuseness level, and D (o _i ) is the decorrelated result.

Assuming that decorrelation is skipped, formula (13) can be simplified as follows.

[formula 14]

If the weight values for all inputs converted to a specific channel are estimated in accordance with the above method, weights can be obtained for each channel using the following method.

1) Summation of weight values for all inputs converted to a specific channel. For example, if input 1 O ₁ and input 2 O _{2 are} input, and the output channel corresponds to the left channel L, the central channel C and the right channel R, then the resulting weight values α _{L (tot)} , α _{C (tot)} , α _{R (tot )} can be obtained as follows:

[formula 15]

where α _L1 is the weight value for input 1 converted to the left channel L, α _C1 is the weight value for input 1 converted to the central channel C, α _C2 is the weight value for input 2 converted to the central channel C, and α _R2 is weight value for input 2 converted to the right channel R.

In this case, only input 1 is converted to the left channel, only input 2 is converted to the right channel, input 1 and input 2 are converted together to the central channel.

2) Summing the weight values for all inputs converted to a specific channel, then dividing the sum by the most prevailing pair of channels and converting the decorrelated signal to another channel for the effect of the surround sound. In this case, the predominant pair of channels can correspond to the left channel and the central channel, if a certain input is located at a point between the left and central channel.

3) Evaluation of the weight value of the most prevailing channel, giving the attenuated convoluted signal to another channel, whose value is the relative value of the estimated weight value.

4) Using weight values for each channel pair, properly combining the decorrelated signal, then setting additional information for each channel.

1.3.2 The case where the downmix processing module includes a mixing section corresponding to a 2 × 4 matrix

10A-10C are exemplary block diagrams of a first embodiment of a downmix processing module illustrated in FIG. 7. As mentioned previously, the first embodiment of the mixing module 720a (hereinafter simply “downmix processing module 720a”) may be an implementation of the reproducing module 900.

, формула 12 упрощается следующим образом.First, assuming that

, formula 12 is simplified as follows.

[formula 15]

The downmix processing module in accordance with Formula 15 is illustrated in FIG. 10A. Referring to FIG. 10A, the downmix processing module 720a may be configured to bypass the input signal in the case of a monaural input signal (m) and to process the input signal in the case of a stereo input signal (L, R). The downmix processing processing unit 720a may include a decorrelation section 722a and a mixing section 724a. The decorrelation section 722a has a decorrelator aD and a decorrelator bD, which can be configured to decorrelate the input signal. The decorrelation portion 722a may correspond to a 2 × 2 matrix. The mixing section 724a may be configured to convert the input signal and the decorrelated signal to each channel. The mixing section 724a may correspond to a 2 × 4 matrix.

и

формула 12 упрощается следующим образом.Secondly, assuming that

and

formula 12 is simplified as follows.

[formula 15-2]

The downmix processing module in accordance with Formula 15 is illustrated in FIG. 10B. Referring to FIG. 10B, a decorrelation section 722 ′ including two decorrelators D ₁ , D ₂ can be configured to generate decorrelated signals D ₁ (a * O ₁ + b * O ₂ ), D ₂ (c * O ₁ + d * O ₂ ).

, формула 12 упрощается следующим образом.Third, assuming that

, formula 12 is simplified as follows.

[formula 15-3]

The downmix processing module in accordance with formula (15) is illustrated in FIG. 10C. 10C, the decorrelation section 722 ″ including two decorrelators D ₁ , D ₂ can be configured to generate decorrelated signals D ₁ (O ₁ ), D ₂ (O ₂ ).

1.3.2 The case where the downmix processing module includes a mixing section corresponding to a 2 × 3 matrix

Formula (15) can be represented as follows:

[formula 16]

The matrix R is a 2 × 3 matrix, the O matrix is a 3 × 1 matrix, and C is a 2 × 1 matrix.

11 is a typical block diagram of a second embodiment of a downmix processing module illustrated in FIG. 7. As mentioned earlier, the second embodiment of the downmix processing processing unit 720b (hereinafter simply referred to as the “downmix processing processing unit 720b”) may be an implementation of the reproducing module 900, as well as the downmix processing processing unit 720a. Referring to FIG. 11A, the downmix processing module 720b may be configured to skip an input signal in the case of a monaural input signal (m) and to process the input signal in the case of a stereo input signal (L, R). The downmix processing module 720b may include a decorrelation section 722b and a mixing section 724b. The decorrelation section 722b has a decorrelator D, which can be configured to decorrelate the input signal O ₁ , O ₂ and output the decorrelated signal D (O ₁ + O ₂ ). The decorrelation section 722b may correspond to a 1 × 2 matrix. The mixing section 724b may be configured to convert the input signal and the decorrelated signal to each channel. The mixing section 724b may correspond to a 2 × 3 matrix, which may be shown as the R matrix in formula (16).

In addition, the decorrelation section 722b may be configured to decorrelate the difference signal O ₁ -O ₂ as a common signal from the two input signals O ₁ , O ₂ . The mixing section 724b may be configured to convert the input signal and the decorrelated common signal to each channel.

1.3.3 The case where the downmix processing module includes a multi-matrix mixing section

A certain object signal may be audible as a similar sensation without being positioned in a predetermined position, which may be called a “spatial sound signal”. For example, the applause or noise of a concert hall may be an example of a spatial audio signal. The spatial sound signal must be reproduced through all speakers. If the spatial sound signal is reproduced as the same signal through all the speakers, it is difficult to sense the “spatiality” of the signal due to the high internal correlation (IC) of the signal. Therefore, there is a need to add a correlated signal to the signal of each channel signal.

FIG. 12 is a typical block diagram of a third embodiment of a downmix processing module illustrated in FIG. 7. 12, a third embodiment of a downmix processing processing unit 720c (hereinafter simply “downmix processing processing unit 720c”) can be configured to generate a spatial audio signal using an input signal O _i , which may include a decorrelation section 722c with N decorrelators mixing section 724c. The decorrelation section 722c may have N decorrelators D ₁ , D ₂ , ..., D _N , which may be configured to decorrelate the input signal O _i . Mixing section 724c may have N matrices R _j , R _k , ..., R _l that can be configured to generate output signals C _j , C _k , ..., C _l using the input signal O _i and the decorrelated signal D _X (O _i ) The matrix R _j can be represented as the following formula.

[formula 17]

O _i is the i-th input signal, R _j is the matrix transforming the i-th input signal O _i into the j-th channel, and C _{j_i} is the j-th output signal. The value θ _{j_i} is the decorrelation coefficient.

Θ _{j_i} can be estimated based on the ICC included in the multi-channel parameter. Furthermore, 724c mixing section may generate output signals based on the spatial information, component θ _{j_i} decorrelation coefficient received from the user interface module 710 by generating information that does not impose a limitation on the present invention.

The number (N) of decorrelators may be equal to the number of output channels. Alternatively, the decorrelated signal may be added to output channels selected by the user. For example, you can place some spatial audio signal on the left, right, and center and output as a spatial audio signal through the speaker of the left channel.

1.3.4 The case where the downmix processing module includes an additional mixing section

FIG. 13 is a typical block diagram of a fourth embodiment of a downmix processing module illustrated in FIG. 7. A fourth embodiment of the downmix processing module 720d (hereinafter simply referred to as the “downmix processing module 720d") may be configured to bypass if the input signal corresponds to a monaural signal (m). The downmix processing processing unit 720d includes an additional mixing section 722d, which may be configured to mix the stereo signal into a monaural signal if the input signal matches the stereo signal. An additional mono down-mix channel (m) is used as input to the multi-channel decoder 730. The multi-channel decoder 730 can control object panning (especially pick-up) using the monaural input signal. In this case, the information generating module 710 can generate a multi-channel parameter based on the configuration 5-1-5 ₁ according to the MPEG Surround standard.

In addition, if amplification is applied to a monaural downmix signal, for example, the aforementioned artistic amplification of a downmix signal (ADG) of FIG. 2, then it is easier to control the panning of the object and the gain of the object. An ADG may be generated using the information generation module 710 based on the mixing information.

2. Mixing with increasing channel signals and control signals of objects

14 is a typical block diagram of a binary audio stream structure of a compressed audio signal according to a second embodiment of the present invention. 15 is a typical block diagram of an apparatus for processing an audio signal according to a second embodiment of the present invention. Turning to part (a) of FIG. 14, a downmix signal, α, a multi-channel parameter β, and an object parameter γ are included in the structure of the binary signal stream. The multi-channel parameter β is a parameter for decomposing a downmix signal. On the other hand, the parameter γ of the object is a parameter for controlling the panning of the object and the gain of the object. According to part (b) of FIG. 14, a downmix signal, α, default parameter β 'and object parameter γ are included in the binary signal stream structure. The default parameter β 'may include preset information for controlling the gain of the subject and panning of the subject. The preset information may correspond to the example proposed by the encoder manufacturer. For example, preset information may describe that the guitar signal is located at a point between the left and center, and the guitar level is set to a certain volume, and the number of output channels at that time is set to a specific channel. In the binary signal stream, a default parameter may be present either for each frame or for a given frame. Binary information may include flag information indicating whether the default setting for this frame is different from the default setting for the previous frame. By including the default parameter in the binary signal stream, you can apply a lower bit rate than the additional information with the object parameter, which is included in the binary signal stream. In addition, in FIG. 14, header information of a binary stream is skipped. The binary stream sequence can be rearranged.

15, an audio signal processing apparatus 1000 in accordance with a second embodiment of the present invention (hereinafter simply “decoder 1000”) may include a binary stream demultiplexer 1005, an information generating unit 1010, a downmix processing processing unit 1020, and multi-channel decoder 1030. The demultiplexer 1005 may be configured to split the multiplexed audio signal into a downmix signal, α, a first multi-channel parameter β and an object parameter γ. The information generating module 1010 may be configured to generate a second multi-channel parameter using the object parameter γ and the downmix processing parameter. The downmix processing parameter contains mode information indicating whether the first multi-channel information β is applied to the downmix processed signal. Information about the mode may correspond to information for selection by the user. In accordance with the mode information, the information generating unit 1010 decides whether to transmit the first multi-channel parameter β or the second multi-channel parameter.

The downmix processing processing unit 1020 may be configured to determine the processing scheme in accordance with the mode information included in the mixing information. In addition, the downmix processing processing unit 1020 may be configured to process the downmix signal, α, in accordance with a specific processing scheme. Then, the downmix processing processing unit 1020 transmits the downmix processed signal to the multi-channel decoder 1030.

Multi-channel decoder 1030 may be configured to obtain either a first multi-channel parameter β or a second multi-channel parameter. If the parameter β 'is included in the binary stream by default, then the multi-channel decoder 1030 can use the default parameter β' instead of the multi-channel parameter β.

Then, the multi-channel decoder 1030 may be configured to generate a multi-channel output signal using the processed downmix signal and the received multi-channel parameter. The multi-channel decoder 1030 may have the same configuration as the above-mentioned multi-channel decoder 730, which does not impose a limitation on the present invention.

3. Binaural processing

A multi-channel decoder can operate in binaural mode. This enables multi-channel sensation through the headphones through filtering with the function of modeling sound perception (HRTF). For the binaural decoding side, the downmix signal and multi-channel parameters are used in conjunction with HRTF filters added to the decoder.

Fig. 16 is an exemplary block diagram of an apparatus for processing an audio signal according to a third embodiment of the present invention. According to FIG. 16, an audio signal processing apparatus according to a third embodiment (hereinafter simply “decoder 1100”) may comprise an information generating module 1110, a downmix processing processing module 1120, and a multi-channel decoder 1130 with a synchronization matching section 1130a.

The information generating module 1110 may have the same configuration as the information generating module 710 of FIG. 7, with the formation of a dynamic HRTF. The downmix processing unit 1120 may have the same configuration as the downmix processing unit 720 of FIG. 7. Like the preceding elements, the multi-channel decoder 1130, with the exception of the synchronization matching portion 1130a, is the same case of the elements mentioned above. Therefore, the details of the information generating unit 1110, downmix processing unit 1120, and multi-channel decoder 1130 will be omitted.

Dynamic HRTF describes the relationship between object signals and virtual speaker signals corresponding to the azimuthal angle and elevation angle of HRTF, which is time-dependent information in accordance with real-time user control.

Dynamic HRTF may correspond to one of the coefficients of the HRTF filter itself, parameterized coefficient information, and index information if the multi-channel decoder contains the entire set of HRTF filters.

There is a need to reconcile dynamic HRTF information with a frame of a downmix signal, regardless of the type of dynamic HRTF. To match the HRTF information with a downmix signal, three types of circuits can be provided, namely:

1) Insertion of marking information into each HRTF information and downstream mixed signal, then matching HRTF with downstream mixed signal based on the inserted marking information. In this scheme, it is typical that the marking information can be included in the auxiliary field in the MPEG Surround standard. The marking information may be presented in the form of time information, counter information, index information, etc.

2) Insert HRTF information in a frame of a stream of binary signals. In this diagram, mode information can be set indicating whether the current frame corresponds to the default mode. If a default mode is applied that describes the HRTF information of the current frame identical to the HRTF information of the previous frame, then the bit rates of the HRTF information can be reduced.

2-1) In addition, it is possible to determine transmission information indicating whether the HRTF information of the current frame has already been transmitted. If transmission information is used that describes the HRTF information of the current frame identical to the transmitted HRTF information of the frame, it is also possible to reduce the bit rates of the HRTF information.

3) Transmitting multiple HRTF information in advance, then transmitting identifying information indicating which HRTF among the transmitted HRTF information for each frame.

In addition, if the HRTF coefficient changes unexpectedly, distortion may be generated. To reduce this distortion, you need to smooth the coefficient or the reproduced signal.

4. Playback

17 is an exemplary block diagram of an apparatus for processing an audio signal according to a fourth embodiment of the present invention. An audio signal processing apparatus 1200 according to a fourth embodiment of the present invention (hereinafter simply referred to as “processor 1200”) may comprise an encoder 1210 on the encoding side 1200A and a reproducing module 1220 and synthesis module 1230 on the decoding side 1200B. Encoder 1210 may be configured to receive a multi-channel signal of the object and generate mixing with lowering the audio signal and additional information. The playback module 1220 may be configured to obtain additional information from the encoder 1210, the playback configuration, and user control from the device or user interface settings and generate playback information using additional information, the playback configuration, and user control. The synthesis module 1230 may be configured to synthesize a multi-channel output signal using the playback information and the received downmix signal from the encoder 1210.

4.1 Applying Effect Mode

An effect mode is a mode for remixing or restoring a signal. For example, there may be a "live" mode, a "club" mode, a karaoke mode, etc. The effect mode information may correspond to a set of mixing parameters generated by the manufacturer, another user, etc. If information about the effect mode is applied, the end user does not need to fully control the panning of the object and the gain of the object, since the user can select one of the predefined information about the effect mode.

Two methods of generating information about the effect mode can be distinguished. Firstly, it is possible that the effect mode information is generated by the 1200A encoder and transmitted to the 1200B decoder. Secondly, information about the effects mode can be generated automatically on the decoding side. Details of the two methods will be described below.

4.1.1 Transmission of effect mode information to the decoding side

Effect mode information can be generated by the manufacturer on the 1200A encoder. In accordance with this method, the decoder 1200B may be configured to obtain additional information including information about the effects mode, and output a user interface with which the user can select one of the information about the effects mode. Decoder 1200B may be configured to form an output channel based on selected effect mode information.

In addition, it is unacceptable for the listener to hear the downmix signal as it is if the 1200A encoder downmixes the signal to improve the signal quality of the objects. However, if the effect mode information is applied to the 1200B decoder, the downmix signal can be reproduced with maximum quality.

4.1.2 Generating information about the effect mode on the decoding side

Effect mode information may be generated at the 1200B decoder. Decoder 1200B may be configured to search for suitable effect mode information for a downmix signal. Then the decoder 1200B can be configured to independently select one of the found effect modes (automatic adjustment mode) or provide the user with the opportunity to select one of them (user select mode). Further, the decoder 1200B can be configured to obtain information about the object (the number of objects, the names of instruments, etc.) included in the additional information, and control the object based on the selected information about the effects mode and information about the object.

In addition, you can control similar objects in the mass. For example, rhythm-related instruments may be similar objects in the case of a “rhythm feel mode”. Control in the total mass means controlling each object at the same time, and not controlling objects using the same parameter.

In addition, you can control the object based on the settings of the decoder and the environment of the device (including whether headphones or speakers are used). For example, if the volume setting of the device is low, the object corresponding to the main melody can be emphasized, if the volume setting of the device is high, the object corresponding to the main melody can be suppressed.

4.2 Type of input signal object on the coding side

The input signal input to the encoder 1200A can be classified into three types as follows.

1) Monophonic object (monophonic channel object)

A monophonic object is the most common type of object. You can synthesize an internal downmix signal by simply adding objects. You can also synthesize an internal signal down-mixed using object gain and object panning, which can be one of the user controls and the information provided. In generating the downmix internal signal, it is also possible to generate playback information using at least one of an object’s characteristics, user input, and information provided with the object.

If there is an external signal down-mixed, then information indicating the relationship between the external down-mixed signal and the object can be extracted and transmitted.

2) Stereo object (stereo channel object)

You can synthesize an internal signal down-mixed by simply adding up the objects, as in the case of the monophonic object mentioned above. You can also synthesize an internal signal down-mixed using object gain and object panning, which can be one of the user controls and the information provided. If the downmix signal corresponds to a monaural signal, then it is possible for the 1200A encoder to use an object converted to a monaural signal to generate a downmix signal. In this case, it is possible to extract and transmit information associated with the subject (for example, pan information in each time-frequency domain) when converted to a monaural signal. As with the previous monophonic object, in the formation of the internal signal mixed downwardly, it is also possible to generate playback information using at least one of the characteristics of the object, user input and information provided with the object. As with the previous monophonic object, if there is an external signal mixed with decreasing, then information indicating the relationship between the external signal mixed with decreasing and the object can be extracted and transmitted.

3) Multichannel object

In the case of a multi-channel object, it is possible to perform the above method described with a monaural object and a stereo object. In addition, you can enter a multi-channel object as an MPEG Surround view. In this case, it is possible to generate object-based downmixing (e.g., downmixing SAOC) using the downmix channel and use multi-channel information (e.g. spatial information in MPEG Surround) to generate multi-channel information and playback information. Therefore, it is possible to reduce the amount of computation, since a multi-channel object present in the form of MPEG Surround does not need to be decoded and encoded using an object-oriented encoder (for example, an SAOC encoder). If the downmix of the object corresponds to stereo and the downmix (for example, SAOC downmix) in this case is monophonic, then the aforementioned method described with the stereo object can be applied.

4) Transfer scheme for a variable object type

As noted above, a variable type of object (monaural object, stereo object and multi-channel object) can be transmitted from the encoder 1200A to the decoder 1200B. A transmission scheme for a variable object type may be provided as follows.

According to FIG. 18, when a downmix signal includes a plural object, additional information includes information for each object. For example, when a multiple object consists of the Nth monophonic object (A), the left channel of the (N + 1) th object (B), and the right channel of the (N + 1) th object (C), additional information includes information for 3 objects (A, B, C).

The additional information may include correlation flag information indicating whether the object is part of a stereo or multi-channel object, for example a monophonic object, one channel (L or R) of a stereo object, and so on. For example, the correlation flag information is “0” if a monophonic object is present, the correlation flag information is “1” if there is one channel of the stereo object. When one part of the stereo object and another part of the stereo object are transmitted sequentially, the correlation flag information for the other part of the stereo object can be any value (for example, “0”, “1”, or something else). In addition, correlation flag information for another part of the stereo object may not be transmitted.

In addition, in the case of a multi-channel object, the correlation flag information for one part of the multi-channel object may be a value describing the number of multi-channel objects. For example, in the case of a 5.1-channel object, the correlation flag information for the left channel from 5.1 channel can be “5”, the correlation flag information for the other channel (R, Lr, Rr, C, LFE) from 5.1 channel can be either “0 "or not be transmitted.

4.3 Object Attribute

An object can have three types of attributes, namely:

a) The only object

A single object can be configured as a source. You can apply one parameter to a single object to control the panning of the object and the gain of the object during the formation of the signal mixed with decreasing, and playback. “One parameter” can mean not only one parameter for the entire time / frequency domain, but also one parameter for each time / frequency interval.

b) Group entity

A single object can be configured as more than two sources. You can apply one parameter to a group object to control the panning of the object and the gain of the object, although the group object is entered at least as two sources. Details of the group entity are explained with reference to FIG. 19 as follows. According to FIG. 19, the encoder 1300 includes a grouping module 1310 and a downmix module 1320. Grouping module 1310 may be configured to group at least two objects among inputted data of several objects based on grouping information. Grouping information may be generated by the coding side manufacturer. The downmix module 1320 can be configured to generate a downmix signal using a grouped object generated by the grouping module 1310. The mixing module 1320 can be configured to generate additional information for the grouped object.

c) Combined object

A combined object is an object combined with at least one source. You can control the pan and gain of the object in the total mass, but keep the connection between the combined objects unchanged. For example, in the case of a drum, you can control the drum, but keep the connection between the big drum, tom-tom and cymbals unchanged. For example, when the snare drum is at the center point and the cymbals are at the left point, you can position the snare drum at the right point and position the zymbals at the point between the center and right if the drum is moving in the right direction.

Communication information between the combined entities may be transmitted to the decoder. On the other hand, the decoder can retrieve communication information using a combined entity.

4.4 Hierarchical management of objects

You can manage objects hierarchically. For example, after controlling the drum, each sub-element of the drum can be controlled. To hierarchically manage objects, three schemes are provided below:

a) UI (user interface)

Only the representing element can be displayed without displaying all objects. If the representing element is selected by the user, all objects are displayed.

b) Grouping objects

After grouping the objects to represent the representing element, you can control the representing element to manage all objects grouped as a representing element. Information extracted during grouping can be transmitted to the decoder. Also, grouping information may be generated in the decoder. The application of control information in the total mass can be performed based on predefined control information for each element.

c) Object configuration

You can use the above combined object. Information about the elements of the combined object can be generated either in the encoder or in the decoder. Information about the elements from the encoder can be transmitted as another form, different from information about the combined object.

It will be apparent to those skilled in the art that various modifications and changes can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention cover the modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents.

Industrial applicability

Accordingly, the present invention is applicable to encoding and decoding an audio signal.

Claims (19)

1. A method for processing an audio signal, comprising the steps of:
receiving a downmix signal, object information and mixing information;
generating downmix processing information using object information and mixing information;
processing the downmix signal using downmix processing information; and
form multi-channel information using information about the object and information about mixing, while the number of channels of the signal mixed with decreasing is equal to the number of channels of the processed signal mixed with decreasing. 2. The method according to claim 1, in which information about the object includes at least one of information about the level of the object and information about the correlation of the object. 3. The method of claim 1, wherein the downmix processing processing information corresponds to information for controlling panning of an object if the number of channels of the downmix signal corresponds to at least two. 4. The method of claim 1, wherein the downmix processing information corresponds to information for controlling gain of an object. 5. The method of claim 1, wherein the step of processing the downmix signal is performed using the 2 × 2 module if the downmix signal corresponds to a stereo signal. 6. The method according to claim 1, wherein one channel of the processed downmix signal corresponds to a combination of one channel of the downmix signal multiplied by the first gain and another channel of the downmix signal multiplied by the second gain, if the downmix signal matches the stereo signal. 7. The method according to claim 1, additionally containing a stage in which:
form the output signal in the time domain using the processed signal mixed down. 8. The method of claim 7, wherein the downmix signal corresponds to a subband region signal generated by a set of subband analysis filters. 9. The method of claim 1, wherein the multi-channel information includes at least one of channel level information and channel correlation information. 10. The method according to claim 1, additionally containing phase, in which:
form a multi-channel signal using the processed signal, mixed down, and multi-channel information. 11. The method according to claim 1, in which the mixing information is generated using at least one of the information about the position of the object and information about the configuration of the playback. 12. The method of claim 1, wherein the downmix signal is received as a broadcast signal. 13. The method of claim 1, wherein the downmix signal is received on a digital medium. 14. A method for processing an audio signal, comprising the steps of:
receiving a downmix signal, object information and mixing information;
decomposing the downmix signal into a subband signal;
generating downmix processing information using object information and mixing information; and
processing a subband signal using downmix processing information;
generating an output signal using the processed subband signal,
where the number of channels of the downmix signal is equal to the number of output signals, and the output signal corresponds to a time-domain signal. 15. A machine-readable medium with instructions stored on it that, when executed by a processor, cause the processor to perform operations comprising:
receiving a downmixed signal, object information and mixing information;
generating information about downmix processing using object information and mixing information;
processing the downmix signal using the downmix processing information; and
formation of multichannel information using information about the object and information about mixing,
moreover, the number of channels of the signal mixed with decreasing is equal to the number of channels of the processed signal mixed with decreasing. 16. A machine-readable medium with instructions stored on it, which, when executed by a processor, causes the processor to perform operations comprising:
receiving a downmixed signal, object information and mixing information;
decomposing the downmix signal into a subband signal;
generating information about downmix processing using object information and mixing information; and
subband signal processing using downmix processing information;
generating an output signal using the processed subband signal,
moreover, the number of channels of the downmix signal is equal to the number of output signals, and the output signal corresponds to a time-domain signal. 17. An apparatus for processing an audio signal, comprising:
an information generation module that receives information about the object and information about mixing and generates information about the processing of mixing downward using information about the object and information about mixing, and generates multi-channel information using information about the object and information about mixing; and
a downmix processing module receiving a downmix signal and downmix processing information and a downmix processing signal using downmix processing information;
moreover, the number of channels of the signal mixed with decreasing is equal to the number of channels of the processed signal mixed with decreasing. 18. An apparatus for processing an audio signal, comprising:
an information generating unit receiving a downmix signal, object information and mixing information, wherein the information generating unit generates downmix processing information using object information and mixing information; and
a downmix processing module, decomposing the downmix signal into a subband signal, processing a subband signal using downmix processing information, and generating an output signal using the processed subband signal,
moreover, the number of channels of the downmix signal is equal to the number of output signals, and the output signal corresponds to a time-domain signal. 19. A method for processing an audio signal, comprising the steps of:
receiving a downmixed signal using a signal of a plurality of objects;
generating information about an object representing a relationship between signals of a plurality of objects using signals of a plurality of objects and a signal mixed downwardly, and
wherein the downmix signal may be a downmix signal so that the number of channels of the downmix signal is equal to the number of processed downmix signals.

Images