RU2485605C2 - Improved method for coding and parametric presentation of coding multichannel object after downmixing - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: audio object encoder, designed to generate encoded object signals using a plurality of audio objects, which includes a downmixing data generator which generates downmixing parameters, having indications for the order of distribution of the plurality of audio objects on at least two downmixing channels, an audio object parameter generator which generates audio object parameters, and an output interface which generates an imported output audio signal using downmixing data and object parameters. An audio synthesiser, which uses downmixing data to generate output data, used to form a plurality of output channels for reproducing an audio signal of a given configuration.

EFFECT: facilitating upmixing on all downmixing channels.

13 cl, 18 dwg

Description

The invention relates to decoding multiple objects by converting an encoded multi-object signal based on available multi-channel down-mix and auxiliary control data.

Recent developments in the field of sound processing technology make it possible to recreate a multi-channel audio signal based on a stereo (or mono) signal and the corresponding control data. These methods for parametric coding of the surround sound environment typically include parameterization. Parametric multi-channel audio decoder (e.g., MPEG Surround ISO / TEC 23003-1, L. Villemoes, J. Herre, J. Breebaart, G. Hotho, S. Disch, H. Pumhagen, and K. Kjorling, "MPEG Surround: The Forthcoming ISO Standard for Spatial Audio Coding, "in the 28th International AES Conference, The Future of Audio Technology Surround and Beyond, Pitea, Sweden, June 30-July 2, 2006; J. Breebaart, J. Herre, L. Villemoes, C. Jin ,, K.Kjorling, J.Plogsties, and J.Koppens, "Multi-Channels goes Mobile: MPEG Surround Binaural Rendering," in the 29th International AES Conference, Audio for Mobile and Handheld Devices, Seoul, Sept 2-4,2006 ) reconstructs M channels based on K received channels, where M> K, using control data. The control data is a parameterization of a multi-channel signal based on the difference in signal intensity between channels (IID) and inter-channel coherence, consistency (ICC). As a rule, such parameters are distinguished at the coding stage and describe the power ratios and the correlation between the pairs of channels used in upmixing. The use of such a coding algorithm allows coding at a data rate much lower than the transmission of the entire set of M channels, with high coding efficiency and at the same time guarantee compatibility with both K channel devices and M channel devices.

A similar coding system is implemented by the corresponding encoder of an audio object [C. Faller, "Parametric Joint-Coding of Audio Sources," Convention Paper 6752 presented at the 120th AES Convention, Paris, France, May 20-23, 2006], [C. Faller, " Parametric Joint-Coding of Audio Sources, "Patent application PCT / EP2006 / 050904, 2006], where several audio objects are mixed down by the encoder and later mixed up using the control commands. The upmixing process can also be considered as the separation of objects mixed in the downmix. The signal obtained as a result of upmixing can be converted for reproduction to a single or multichannel form. Determining more precisely, the publications mentioned above present a method for synthesizing sound channels based on the results of down-mixing (referred to as the sum signal), statistical information about the sources, and characteristics that specify the required output format. If several signals obtained by down-mixing are used, these signals consist of subsets of different objects, and up-mixing should be carried out individually for each down-mixing channel. The novelty of the proposed method lies in the implementation of up-mix simultaneously on all channels of down-mix. The object coding methods presented prior to the present invention did not offer the option of decoding the downmix results on several channels simultaneously.

A first aspect of the invention relates to an audio object encoder generating an encoded audio object signal using a plurality of audio objects, including:

a downmix information (data) generator generating distribution parameters of a plurality of audio objects over at least two downmix channels;

an audio object parameter generator and an output interface for generating an encoded audio object signal using downmix characteristics and object parameters.

A second aspect of the invention relates to a method for encoding an audio object. generating a coded signal of an audio object using a combination of audio objects, including:

generating down-mix data characterizing the distribution order of a plurality of audio objects over at least two down-mix channels;

generating parameters of audio objects and generating encoded signals of an audio object using downmix data and object parameters.

A third aspect of the invention relates to a sound synthesizer (audio synthesizer) generating output using an encoded audio object signal, including:

an output synthesizer used to represent a plurality of output channels with a given configuration of an output audio signal representing a plurality of audio objects, where the output synthesizer recognizes down-mix characteristics for distributing a plurality of audio objects over at least two down-mix channels and audio object parameters.

A fourth aspect of the invention relates to a method for synthesizing sound, which allows generating output using an encoded signal of an audio object, including:

generating output data for generating a plurality of output channels with a given configuration of an output audio signal representing a plurality of audio objects using an output synthesizer capable of reading down-mix characteristics to distribute a plurality of audio objects over at least two down-mix channels and audio object parameters.

A fifth aspect of the invention relates to an encoded audio object signal comprising downmix characteristics indicating the distribution order of a plurality of audio objects over at least two downmix channels, and object parameters allowing reconstruction of audio objects using object parameters and at least two downmix channels mixing.

A sixth aspect of the invention relates to computer software for implementing a method for encoding an audio object or a method for decoding an audio object on a computer.

Further, the invention will be presented by illustrative material, not limiting it either in form or in essence, with explanations of the accompanying drawings, where:

on figa presents a block diagram of the encoding algorithm for a spatial audio object, including encoding and decoding;

on fig.1b presents a block diagram of the encoding algorithm for a spatial audio object using the MPEG Surround decoder;

figure 2 presents the algorithm of the encoder spatial audio object;

figure 3 presents a diagram of the algorithm of the extractor (separator) parameters of the audio object in the mode of power differentiation;

figure 4 presents a diagram of the algorithm of the extractor (extractor) parameters of the audio object in the prediction mode;

figure 5 presents a diagram of the device transcoder SAOC - MPEG Surround;

figure 6 schematically shows the various modes of operation of the down-mixer for down-mixing;

7 is a schematic diagram of an MPEG Surround decoder for down-mixing a stereo signal;

on Fig is a diagram of a special case of implementation using the SAOC encoder;

Fig.9 is a diagram of an embodiment of an encoder;

10 is a diagram of an embodiment of a decoder;

figure 11 presents a table of optimal modes of operation of the decoder / synthesizer;

on Fig presents a block diagram of a methodology for calculating some spatial parameters of the up-mix;

on figa presents a block diagram of a methodology for calculating additional spatial parameters up-mixing;

on fig.13b presents a block diagram of a calculation method using prediction parameters;

Fig. 14 is a general schematic diagram of an encoder / decoder system;

on Fig presents a block diagram of an algorithm for calculating the prognostic parameters of the object; and

Fig. 16 illustrates a stereo presentation (rendering) method.

Embodiments of the invention described below are nothing more than an illustration of the principles of an improved encoding method and a parametric representation of the encoding of a multi-channel object after downmixing. It is understood that for specialists in this field, the possibility of making changes and improvements to the layout and elements of the described construction is obvious. Therefore, the presented descriptions and explanations of the embodiments of the invention are limited only by the scope of patent requirements, but not by specific details.

Preferred embodiments provide an encoding method that combines the functionality of an object encoding algorithm with the capabilities of the audio presentation (audio rendering) of a multi-channel decoder. Forwarded control data refers to individual objects and, therefore, allows you to control the spatial position and signal level during playback. Thus, the control information is directly related to the so-called 'scene description', which gives information about the location of objects in the surrounding space. The scene description can be controlled either from the decoder side in interactive mode with the listener or from the encoder side from the sound source.

The essence of the invention lies in the fact that a transcoder is introduced in order to convert (transcode) the object-related control information and the down-mix signal into control data and the down-mix signal intended for a reproduction system, for example, an MPEG Surround decoder. In the presented encoding method, objects can be arbitrarily distributed over the available channels of the downstream mixing of the encoder. The transcoder accurately utilizes multi-channel down-mix parameters, providing a transcoded down-mix signal and object-related control data. Due to this, up-mixing on the decoder is not performed individually for each channel, as suggested in [C. Faller, "Parametric Joint-Coding of Audio Sources," Convention Paper 6752 presented at the 120th AES Convention, Paris, France, May 20-23, 2006], and all downmix channels are processed simultaneously in a single upmix process. According to the new scheme, the parameters of multichannel down-mixing should be part of the control data and encoded by the object encoder.

The distribution of objects on the down-mix channels can be performed automatically, or it can be a constructive decision related to the encoder. In the latter case, the system of down (down) mixing can be included in an existing multi-channel playback system (for example, in stereo), emphasizing playback, omitting the stage of transcoding and multi-channel decoding. This is another advantage over earlier coding algorithms known in the art for a single downmix channel or multiple downmix channels containing subsets of source objects.

While encoding algorithms of an object of the prior art describe decoding technology using only a single downmix channel, the present invention does not have such a limitation, since it offers a method for simultaneously decoding downmix material containing downmix signals on several channels. The quality of object separation increases as the number of down-mix channels increases. Thus, the invention successfully fills the gap between the object coding algorithm for a single mono channel down-mix and the multi-channel coding algorithm, where each object is transmitted on a dedicated channel. Thus, the proposed method allows flexible quality management when separating objects depending on the requirements, application requirements and operational properties of the transmission system (such as channel capacity).

In addition, the advantage of using more than one channel is that it also allows you to take into account the correlation between different objects, as opposed to a description that takes into account only the difference in the intensity of sound signals, as in object encoding algorithms in earlier practice. Earlier practice proceeded from the premise that all objects are independent from each other and mutually inconsistent (zero cross-correlation), while in reality it is unlikely that objects cannot be correlated, such as the left and right channels of a stereo signal. In accordance with the concept of the present invention, the inclusion of correlation parameters in the description (control data) makes it more complete and thus contributes to the creation of an additional possibility of separation of objects. Preferred embodiments include at least one of the following features.

A system for transmitting and creating many separate audio objects using multi-channel down-mixing and auxiliary control data describing these objects, including:

a spatial audio object encoder encoding a plurality of audio objects for multi-channel down-mix, multi-channel down-mix information and object parameters; or a spatial audio object decoder that decrypts the multi-channel down-mix data, the multi-channel down-mix information, the object parameters and the audio rendering matrix (presentation matrix) of the object into a second multi-channel audio signal suitable for audio playback.

Fig. 1a shows a spatial audio object (SAOC) encoding algorithm including an SAOC encoder 101 and an SAOC decoder 104. The spatial audio object encoder 101 encodes N objects into object down-mix data about K> 1 audio channels in accordance with the encoder parameters. Information about the applied downmix weight matrix D is output by the SAOC encoder along with auxiliary data regarding the power and correlation of the downmix. Matrix D is often, but not necessarily always, constant in time and frequency and therefore contains relatively little information. Finally, the SAOC encoder captures the parameters of each object as a time-frequency function with a resolution depth that is determined based on the principles of perception (perceptual coding). The decoder of spatial audio objects 104 receives the input data of the down-mix channels of the objects, information about the down-mix, and the parameters of the objects (generated by the encoder) and generates output data containing M audio channels for presentation to the user. Audio rendering of N objects in M audio channels is performed by means of an audio rendering matrix, which is a set of parameters entered by the user into the SAOC decoder.

Fig. 1b shows a block diagram of an encoding algorithm for a spatial audio object followed by the use of an MPEG Surround decoder. The SAOC 104 decoder used in the present invention can be implemented as a SAOC transponder - MPEG Surround 102 in combination with a down-mix MPEG Surround 103 decoder to a stereo signal. A user-controlled audio rendering matrix A of dimension M × N determines a predetermined conversion ratio of N objects into M audio channels. The functions of this matrix can depend on both settings and frequency indicators, and this is the final result of the most user-friendly interface for managing audio objects (where, in addition, a scene description can be entered from the outside). In the case of applying the settings for the speaker system 5. 1, the number of output audio channels will be M = 6. The task of the SAOC decoder is to perceptually recreate the original audio objects as the final result of the audio rendering. At the input, the SAOC-MPEG Surround 102 transcoder receives an audio rendering matrix A, object down-mix data, down-mix results, including a down-mix weight matrix D, and a description of the object, and generates stereo down-mix and MPEG Surround information. If the transcoder is implemented in accordance with the present invention, the next MPEG Surround 103 decoder, having received this data at the input, gives an M-channel acoustic signal with the required characteristics at the output.

The SAOC decoder introduced in the present invention consists of a SAOC transcoder - MPEG Surround 102 and an MPEG Surround 103 decoder with down-mixing to a stereo signal. A user-controlled audio rendering matrix A of dimension M × N determines a predetermined conversion ratio of N objects into M audio channels. This matrix can depend on both settings and frequency, which is an indicator of a more user-friendly interface for managing audio objects. When applying the settings for the 5.1 speaker system, the number of audio output channels will be M = 6. The SAOC decoder is designed to perceptually recreate source audio objects as the final result of audio rendering. At the input, the SAOC-MPEG Surround 102 transcoder receives an audio rendering matrix A, object down-mix data, down-mix results, including a down-mix weight matrix D, and an object description and generates stereo down-mix and MPEG Surround information. If the transcoder is implemented in accordance with the present invention, the next MPEG Surround 103 decoder, having received this data at the input, gives an M-channel acoustic signal with the required characteristics at the output.

Figure 2 presents the algorithm of the encoder spatial audio object (SAOC) 101, introduced by the present invention. N audio objects are input into the downmixing unit 201, as well as into the extractor (extractor) of the parameters of the audio object 202. The downmixing unit 201 mixes the objects into the stream of the final downmixing data of the object, consisting of K> 1 audio channels, in accordance with the encoder parameters, and also outputs downmix information. This information includes a description of the applied downmix weight matrix D, and additionally, if the sequentially used audio object parameter extractor operates in prediction mode, parameters describing the power and correlation of the object's downmix results.

As will be discussed in one of the following paragraphs, the role of such additional parameters is to provide access to the energy and correlation indicators of the subsets of the converted audio channels in those cases when the parameters of the objects are expressed only with respect to the downmix, and the main example here is the rear / front signals for acoustic systems 5.1. An audio object parameter extractor 202 extracts object parameters according to encoder parameters. The encoder controls by time-frequency changes determine which of the two encoder modes is applied, on an energy or prognostic basis. In the power differentiation mode, the encoder parameters further contain information on the grouping of N audio objects into P stereo objects and N-2P mono objects. Each mode will be described further in FIGS. 3 and 4.

Figure 3 presents a diagram of the algorithm of the extractor parameters of the audio object 202 in the mode of power differentiation. Grouping 301 into P stereo objects and N-2P mono objects is carried out according to the grouping information contained in the encoder parameters. For each given time-frequency interval, the following operations are then performed. Two power indicators of the object and one normalized correlation are extracted by the stereoparameter extractor 302 for each of the P stereo objects. One energy indicator is allocated by the extractor of parameters 303 for each of the N-2P monoobjects. Then, a complete set of N power parameters and P normalized correlation parameters are encoded in 304 together with grouping data, forming object parameters. Coding may include a normalization operation, taking into account the highest indicator of the power of the object or taking into account the amount of allocated power of the object.

Figure 4 presents a diagram of the algorithm of the extractor parameters of the audio object 202 in prediction mode. For each given time-frequency interval, the following operations are then performed. For each of N objects, a linear combination of K channels for downmixing objects is output, which corresponds to this object using the least squares method. K weights of this linear combination are called object prediction coefficients (OPC), and they are calculated by the OPC 401 extractor. A complete set of OPC in the amount of N-K is encoded in 402 with the formation of the object parameters. Coding may include a reduction in the total number of OPCs based on linear interdependencies. A distinctive feature of this invention is that this total number can be reduced to a maximum of {K · (N-K), 0}, there is a weight matrix down-mixing D has a full rank.

FIG. 5 is a schematic diagram of a device of a SAOC-MPEG Surround 102 transcoder according to the present invention. For each time-frequency interval, down-mix information and object parameters are combined with an audio rendering matrix with a parameter counter 502 to form MPEG Surround parameters of the CLD type (channel level difference), CPC (channel prediction coefficient) and ICC (inter-channel consistency) and the G matrix of the down converter 2 × K mixing. The down-mix result converter 501 converts the down-mix of an object into a stereo down-mix using the matrix operation in accordance with the matrix G. In the simplified transcoder mode for K-2, this matrix works as a single matrix, and the down-mix of the object passes without changing as a stereo down-mix. In the diagram, this mode is shown as switch 503 in position A, while in normal operation the switch is in position B. An additional advantage of the transcoder is its suitability for use as a stand-alone device where MPEG Surround parameters are ignored, and the output of the down converter mixes are used directly as stereo audio rendering.

6 schematically illustrates various modes of operation of the downmix data converter 501 according to the present invention. Given that the transmitted downstream object in bitstream format is the output of a K-channel audio encoder, this bitstream is first decrypted by audio decoder 601 into K time-domain audio signals. Then, all these signals are converted to the frequency domain by the hybrid filter bank QMF (quadrature mirror filter) MPEG Surround in the T / F block (time / frequency) 602. The operation of the time and frequency variation matrix determined by the data of the converter matrix is performed on the resulting signals of the hybrid QMF region a matrixing unit 603 that outputs a stereo signal in the hybrid QMF region. The hybrid synthesis unit 604 converts the stereo signal of the hybrid QMF region to the stereo signal of the QMF region. The hybrid QMF region is defined to improve the frequency resolution towards lower frequencies by subsequently filtering the QMF subbands. When such filtering is subsequently performed using Nyquist filter banks, the conversion from hybrid to standard QMF is to simply sum the signal groups of the hybrid subbands, see [E. Schuijers, J. Breebart, and H. Purnhagen "Low complexity parametric stereo coding" Proc 116 ^th AES convention Berlin. Germany 2004, Preprint 6073]. This signal is the first possible output format of the down-mix converter, which corresponds to the position A of switch 607. A similar QMF domain signal can be fed directly to the corresponding QMF interface of the MPEG Surround decoder, and this is the most preferred mode of operation in terms of delay, complexity and quality . Another possibility is to generate a time-domain stereo signal using QMF 605 filter bank synthesis. When position B of the switch 607, the converter outputs a digital stereo signal, which can also be input to the time-domain interface of a subsequent MPEG Surround decoder or fed directly to a stereo playback device. A third possibility at position C of the switch 607 is to encode the stereo signal of the music domain using the stereo audio encoder 606. In this case, the output format of the down-mix converter will be a stereo audio stream compatible with the center decoder, which is a component of the MPEG decoder. This third mode of operation is applicable when the SAOC - MPEG Surround transcoder is blocked by an MPEG decoder due to a connection limiting the data transfer rate, or when the user needs to save an image of a specific object for future playback.

7 is a schematic diagram of an MPEG Surround decoder for downmixing a stereo signal. The stereo down-mix using the two-to-three window (TTT) is divided into three intermediate channels. Next, each intermediate channel using three one-to-two windows (OTT) is divided into two with the formation of six channels 5.1-channel configuration.

On Fig given a diagram of a special case of implementation using the encoder SAOC. The 802 audio mixer produces a stereo signal (left and right), which is usually created by mixing the signals at the mixer input (here, input channels 1-6) and arbitrary additional input from electronic effects such as a reverb, etc. In addition, the mixer has one individual output channel (here channel 5). This channel can be used, for example, for normal mixer functions, such as "direct output" or "additional transfer" to output individual data without involving any intermediate processes (such as dynamic processing and equalizer). The stereo signal (left and right) and the individual output channel (obj5) are input to the SAOC 801 encoder, which is only a special case of the SAOC 101 encoder in FIG. However, it serves as a typical application example, when the obj5 audio object (containing, for example, speech) should be completely controlled by the user with the right to make corrections at the decoder input, however, remaining part of the mixed stereo (with left and right channels). It is also obvious from the concept that two or more audio objects can be connected to the “object input” panel in frame 801, and in addition, the stereo can be expanded by a multi-channel connection, for example a 5.1-channel device.

The following is a brief mathematical description of the invention. For discrete complex signals x, y, the complex internal product and the squared norm (energy) are determined by:

where y (k) denotes the complex conjugate signal y (k). All the signals considered here are subband samples from a modulated filter bank or FFT window analysis (fast Fourier transform) of discrete time signals. It is understood that these subbands must be converted back to a discrete time domain using the appropriate synthesis filter bank operations. A block of signals from L samples represents a signal in the time-frequency interval, which is part of the perceptually motivated mosaic filling (tiling) of the time-frequency plane used to describe the properties of the signal. With this partitioning, certain audio objects can be represented as N rows of length L in the matrix,

Downmix weight matrix D of dimension K × N,

where K> 1 defines the K-channel down-mix signal in the form of a matrix with K rows of matrix multiplication

User-controlled audio rendering matrix of object A of dimension M × N defines an M-channel audio rendering with specified performance of audio objects in the form of a matrix with M rows of matrix multiplication

Unless temporarily taking into account the effects of the main audio coding stream, the task of the SAOC decoder is to generate close to the desired perception Y as a result of audio rendering of the original audio objects based on the audio rendering matrix A, the downmix X, the downmix D, and the object parameters.

The parameters of the object in the energy mode according to the present invention carry information about the covariance of the original objects. In the deterministic version, convenient for obtaining sequential results, as well as visual for describing typical encoder operations, covariance appears in an abnormal form as the product of SS * matrices, where the asterisk denotes an operation with a complex conjugate transposed matrix. Thus, the parameters of the object obtained in the energy mode provide a positive semidefinite matrix A of dimension N × N in such a way that, possibly up to the scaling factor,

The prior art coding of audio objects often considers an object model where all objects are not correlated. In this case, the matrix E is diagonal and contains only an approximation to the energies of the object S _n = || S _n || ² for n = 1, 2, ..., N. According to figure 3, the object parameter extractor makes a significant adjustment to this idea, which is especially true in cases where objects are represented by stereo signals for which the assumption of the absence of correlation does not work. The grouping P of selected stereo pairs of objects is described by sets of indices {(n _p , m _p ), p = 1, 2, P}. For these stereopairs, the correlation <S _n , S _m > is calculated, and the complex, real or absolute value of the normalized correlation (ICC)

highlighted by the stereoparameter extractor 302. After that, the ICC data in the decoder can be combined with energy indicators to form the matrix E, 2P separated from the diagonal elements. For example, for the total number of objects N = 3, of which the first two make up a single pair (1, 2), the transmitted energy and correlation data have the form:

S ₁ , S ₂ , S ₃ and p ₁ .

In this case, combining into matrix E gives:

The parameters of the object in the prediction mode according to the present invention are intended to form an object prediction coefficient (OPC) matrix C of dimension N × K, accessible to the decoder in such a way that

In other words, for each object, there is a linear combination of the down-mix channels so that the object can be restored approximately according to:

In a preferred embodiment, an object prediction coefficient (OPC) extractor 401 solves the normal equations

or, for a more attractive real estimate of the object prediction coefficient (OPC), it decides:

In both cases, if we take the really estimated weight matrix of the downmix D and the non-singular covariance of the downmix, then from the left multiplication with D it follows that

where I is the identity matrix of dimension K.

If D has full rank, then, according to elementary linear algebra, the set of solutions for (9) can be parameterized with max {K · (N-K), 0} parameters. This principle is used in 402 when jointly encoding OPC data. The full prediction matrix C can be reconstructed in the decoder from a reduced set of parameters and a downmix matrix.

As an example, consider the case of down-mix with obtaining a stereo down-mix (K = 2), which includes three objects (N = 3>) - a stereo music record (s ₁ , s ₂ ) and a central panned single musical instrument or s ₃ vocal track.

The downmix matrix has the form:

That is, the left channel of the downmix is x ₁ = s ₁ + s ₃ / √2, and the right channel is x ₂ = s ₂ + s ₃ / √2.

The object prediction coefficients (OPC) for a single track tend to approach s ₃ ≈c ₃₁ x ₁ + c ₃₂ x ₂ , in which case equation (11) can be solved to obtain with ₁₁ = 1-c ₃₁ / √2, s ₁₂ = -c ₃ 2 / √2, with ₂₁ = -c ₃₁ / √2 and with ₂₂ = 1-c ₃₂ / √2.

It follows that a sufficient number of prediction coefficients of the object (OPC) is determined through K (N-K) = 2 · (3-2) = 2.

OPC c ₃₁ , c ₃₂ can be found from normal equations

SAOC Transcoder - MPEG Surround

As for figure 7, M = 6 output channels of the configuration 5.1 are

(y ₁ , y ₂ , ..., y ₆ ) = (I _f , I _s , r _f , r _s , c, lfe).

The transcoder should give the output of the stereo down-mix (l ₀ , r ₀ ) and parameters for the TTT and OTT configurations. Since attention is now focused on stereo downmix, it will be assumed that K = 2. Since both the object parameters and the MPS TTT parameters exist both in the energy and in the prognostic mode, it is necessary to consider all four combinations.

The power mode is effective, for example, when the down-mix audio encoder is not a wave encoder in the frequency range in question. It is understood that the parameters of MPEG Surround, which will be discussed below, must be properly quantized and encoded before being sent. To further clarify the four above combinations, it should be recalled that these are:

1) the parameters of the object in the energy mode and the transcoder in the prediction mode;

2) the parameters of the object in the energy mode and the transcoder in the energy mode;

3) the parameters of the object in the prediction mode (prediction coefficient of the OPC object) and the transcoder in the prediction mode;

4) the parameters of the object in the prediction mode (OPC) and the transcoder in the energy mode.

If in the frequency range under consideration the down-mix audio encoder is a wave-type encoder, the object parameters can be fixed both in the energy mode and in the prediction mode, while the transcoder should preferably operate in the prediction mode. If in the considered frequency range the down-mix audio encoder is not a wave-type encoder, both the object encoder and the transcoder should work in the energy mode. The fourth combination is less relevant, as a result of which the further description will affect only the first three combinations.

Object parameters in energy mode

In the energy mode, the data available for the transcoder is described by a triple of matrices (D, E, A). The OTT MPEG Surround parameters are formed by evaluating the energy and correlation indicators during virtual audio rendering of the transferred parameters and the audio rendering matrix A with a dimension of 6 × N. The given six-channel covariance is represented as

Introduction (5) to (13) gives an approximation

which is completely determined by the available data. Let f _a denote the elements of F. Then the parameters CLD and ICC are determined from:

where J> is either the absolute value <p (z) = \ z \, or the operator of the real value <p (z) -Pe {z}. As an illustrative example, consider the case with three objects described earlier in relation to equation (12). Imagine the audio rendering matrix as

Thus, the task of audio rendering is to place object 1 between the right front and right panoramic positions, object 2 - between the left front and left panoramic positions and object 3 - in front of the right, in the center and along the low-frequency optimization channel (lfe). To simplify, we also assume that all three of these objects are uncorrelated and have the same energy so that

In this case, the right side of the formula (14) takes the form

Substituting the corresponding values in formulas (15) - (19) we obtain:

As a reaction, the MPEG Surround decoder will be instructed to introduce some decorrelation between the right front and right panoramic positions, but not allow decorrelation between the left front and left panoramic positions.

For the TTT parameters of MPEG Surround in prediction mode, the first step is to form a 3 × N reduced audio rendering matrix A ₃ for combined channels (l, r, qc), where q = 1 / √2. This implies that A ₃ = D ₃₆ A, where the partial downmix matrix from 6 to 3 is determined using

The incomplete down-mix weights w _p , p = 1, 2, 3 are adjusted so that the energy w _p (y _2p-1 + y _2p ) is equal to the sum of the energies || y _2p-1 || ² + || y _2p || ² to the limit coefficient. All the data necessary to derive a partial downmix matrix, D _{36, is} available in F. Then a 3 × 3 prediction matrix C ₃ is formed so that

It is more preferable to derive such a matrix, previously taking into account the normal equations C ₃ (DED *) = A ₃ S.

The result of solving the normal equations best suits the waveform for (21), taking into account the covariance model of object E. It is recommended to perform some post-processing of the C ₃ matrix, including in-line coefficients, for full or selective compensation of the predicted channel losses.

In order to illustrate and clarify the above steps, it is necessary to continue consideration of the example of audio rendering of the six channels identified earlier. When considering the elements of the matrix F, it should be taken into account that the downmix weights are solutions of the equations

, p=1, 2, 3,

, p = 1, 2, 3,

which in a particular example takes the form

Thus, (w ₁ , w ₂ , w ₃ ) = (1 / √1, √3 / 5.1 / √2).

Substitution in (20) gives:

After which, by solving the system of equations C ₃ (DED *) = A ₃ ED * we find (switching now to achieve the ultimate accuracy),

The C ₃ matrix contains the best weights to approximate the desired result of the audio rendering of the object through the combined channels (l, r, qc) during downstream mixing. This general type of matrix operation cannot be performed by the MPEG Surround decoder, which is bound by the limited TTT matrix space due to the use of only two parameters. The purpose of the down-mix converter (down-mix result) of this invention is to pre-process the down-mix of an object so that the combined effect of the pre-processing and the TTT MPEG Surround matrix matches the desired up-mix result (upmix) described with C ₃ .

In MPEG Surround, the TTT matrix for predicting (l, r, qc) based on (l ₀ , r ₀ ) is parameterized in three parameters (α, β, γ) through

The matrix G of the down-mix converter (down-mix results) according to the present invention is formed by selecting 7 = 1 and solving the system of equations

It is easily confirmed that D _TTT C _TTT = I, where I is the two-on-two identity matrix and

Thus, multiplying the matrices on the left by D _{TTT of} both sides (23) yields

In the general case, G is invertible, and (23) has a unique solution for C _TTT satisfying the condition C _TTT G _TTT = I.

TTT parameters (α, β) are determined by this solution.

For the particular example considered earlier, one can easily confirm that the solutions correspond

.

.

It should be noted that the main volume of the stereo down-mix stream with this transformation matrix changes position between the left and right sides, reflecting the fact that during the audio rendering shown in the example, the objects passing through the left down-mix channel are moved to the right side of the acoustic scenes and vice versa. A similar phenomenon is not possible when using the MPEG Surround decoder in stereo.

If it is not possible to use a down-mix converter, the method described below, which is close to optimal, can be developed. When operating in power mode, the TTT MPEG Surround parameters require the distribution of energy of the combined channels (α, β). Therefore, the corresponding parameters of the channel level difference CLD can be derived directly from the elements F through

In this case, it is advisable to use only the diagonal matrix G with positive cells for the down-mix converter. It is functionally important to achieve the correct energy distribution of the downmix channels before the upmix of the TTT. With a downmix matrix from six channels to two D ₂₆ = D _TTT D ₃₆ and definitions from

just get out

Further observation shows that such a diagonal down-mix converter can be skipped on the way from the object to the MPEG Surround transcoder and implemented by setting the parameters of the arbitrary amplification of the down-mix (ADG) of the MPEG Surround decoder. In this case, the increments in the logarithmic region will correspond to ADG _i = 10log ₁₀ (w _n / z _n ) for i = 1, 2.

Object parameters in prediction mode (OPC)

In the object prediction mode, the available data is represented by three matrices (D, C, A), where C is an N × 2 matrix containing N pairs of OPC object prediction coefficients. Due to the relative nature of the prediction coefficients, in order to estimate the energy parameters of MPEG Surround, access to the approximation indices for the covariance matrix of a 2 × 2 downmixing object will be necessary

It is preferable if this information comes from the object encoder as part of the down-mix information, however, it can also be evaluated on the transcoder based on the measurements of the adopted down-mix, or indirectly derived from (D, C) through analysis of an approximate model of the object. In the presence of Z, the covariance of the object can be estimated by introducing the prediction model Y = CX, resulting in

and all the parameters of the OTT MPEG Surround and the TTT of the energy mode can be estimated based on E, as is the case with the energy parameters of the object. However, the greatest advantage of using the OPC object prediction coefficients is manifested in combination with the TTT MPEG Surround parameters in the prediction mode. In this case, the approximation of the waveform D ₃₆ Y≈А ₃ СХ immediately gives a reduced prediction matrix:

C ₃ = A ₃ C,

based on which, the remaining steps to the formation of the TTT parameters (α, β) and the down-mix converter are similar to obtaining the object parameters in the energy mode. In fact, the steps from formula (22) to formula (25) are completely identical.

The resulting matrix G is fed to the down-mixer, and the TTT parameters (α, β) are sent to the MPEG Surround decoder.

Standalone downmix converter for stereo downmix

In all of the cases described above, the stereo down-mix converter 501 output provides data close to 5.1-channel stereo down-mix as a result of the audio rendering of the original audio objects. This stereo audio rendering can be expressed by an A ₂ matrix of 2 × N dimension, defined as A ₂ = D ₂₆ A. In many implementations, this down-mix is of interest in its own right, while the possibility of direct control of the stereo audio rendering of A ₂ draws attention. As an illustrative example, we again consider the case of a stereo phonogram superimposed in the center of a panned monophonic voice track encoded in a special case of the technique briefly described in the description of figure 8 with explanations in the context of formula (12). The user can control the dynamic range of the voice through audio rendering according to

where ν is the regulation of the voice-music ratio. The structure of the transformation matrix of the downmix results is based on the expression

For the parameters of the object obtained on the basis of the prediction, it is only necessary to substitute the approximation S≈CDS and obtain the transducer matrix G = A ₂ C. For the parameters of the object based on energy indicators, normal equations

FIG. 9 is a diagram of a preferred embodiment of an audio object encoder in accordance with one aspect of the present invention. The encoder audio objects 101 as a whole has already been described in the explanation of the previous graphic schemes. An audio object encoder generating an encoded object signal uses a plurality of audio objects 90, shown in FIG. 9 as input to the downmixer 92 and an object parameter generator 94. In addition, the audio object encoder 101 includes a downmix information generator 96 generating the downmix parameters 97 , fixing the distribution order of the plurality of audio objects over at least two down-mix channels, indicated in the diagram as paths 93, coming from the down-mixer 92.

The object parameter generator is designed to generate the parameters of audio objects 95, and the object parameters are calculated in such a way that reconstruction of the audio object is possible using the object parameters and at least two down-mix channels 93. It is important that the reconstruction is carried out not from the encoder, but from the decoder. However, a full reconstruction from the side of the decoder is possible due to the calculation of the parameters of the objects 95 performed by the generator of parameters of the objects of the encoder.

In addition, the audio object encoder 101 includes an output interface 98 for generating the encoded signal of the audio object 99 using downmix data 97 and object parameters 95. Depending on the purpose, the downmix channels 93 can also be used and encoded as an audio object signal. In this case, situations may arise in which the output interface 98 generates an encoded signal of the audio object 99, which does not contain down-mix channels. Such a situation may occur when any down-mix channels that should be used by the decoder are already at the decoder's disposal so that down-mix information and audio object parameters are transmitted separately through the down-mix channels. The benefit from this situation can be obtained when the down-mix channels of the objects 93 can be purchased separately from the parameters of the objects and information on the down-mix for a smaller amount of money, and the parameters of the objects and information on the down-mix can be bought for additional funds in order to provide the user on the side decoder features to get added value.

In the absence of object parameters and downmix information, the user can convert the downmix channels into a stereo or multichannel signal depending on the number of channels involved in the downmix. Naturally, the user can also generate a monophonic signal by simply adding at least two transmitted down-mix channels of objects.

Object parameters and down-mix data provide the user with the flexibility of acoustic transformations and increase the quality and sound quality of acoustic objects, allowing for multi-purpose audio rendering for later playback of audio material on any type of audio equipment - on stereo systems, on multi-channel systems or even on wave field synthesis systems. If the installation of wave field synthesis is still not very popular, then multichannel systems of the 5.1 or 7.1 format are increasingly spreading to the consumer market.

Figure 10 presents a diagram of a sound synthesizer for generating output data. To perform its functions, the audio synthesizer contains an output data synthesizer 100. The output data synthesizer receives down-mix data 97 and parameters of the audio object 95, as well as, possibly, characteristics of the intended sound source, such as the spatial arrangement of sound sources or the user-defined dynamic range of a particular source in audio rendering result using 101.

The output synthesizer 100 is intended to generate the output data necessary for generating a plurality of output channels with a given configuration of the output audio signal reconstructing a plurality of audio objects. In the best way, the output data synthesizer 100 realizes its functionality using the down-mix parameters 97 and the audio object parameters 95. According to the explanations to FIG. 11 given below, the output data are numerous indicators for various purposes, including specific rendering of the output channels or simple recreation of the original signals , or the recoding of parameters into spatial transformation characteristics with the formation of a spatial configuration for increasing shirovaniya without any rendering of output channels, for example for storage or shipment of these spatial parameters.

The General implementation scheme of the present invention is shown in Fig.14. Here, the encoder unit 140 includes an audio object encoder 101 that receives N audio objects at the input.

At the output of the preferred embodiment of the audio object encoder, in addition to the down-mix information and the object parameters not shown in Fig. 14, the number K of down-mix channels is formed. In accordance with the present invention, the number of downmix channels should be greater than or equal to two.

The downmix channels are transmitted to a decoder unit 142, which includes a spatial boost mixer 143. The spatial boost mixer 143 may include an audio synthesizer that is part of the present invention if the audio synthesizer is in transcoder mode. However, if the audio synthesizer 101, as shown in FIG. 10, operates in a spatial upmix mode, in this implementation both the spatial upmixer 143 and the audio synthesizer are the same device. The spatial boost mixer generates M output channels for playback through the M speakers. These speakers are located at predetermined points in the surrounding space and collectively form an acoustic output signal of a given configuration. The output channel of the output audio signal of a given configuration can be considered as a digital or analog electrodynamic acoustic signal transmitted from the output of the spatial boosting mixer 143 to the input of the speaker with a predetermined positioning in the environment of a number of configured audio output sources in a certain way. Depending on the specific situation, if stereo audio rendering is performed, the number of M output channels may be equal to two. When performing multi-channel audio rendering, the number of M output channels will be more than two.

The most common situation is when the number of down-mix channels is less than the number of output channels due to the technical requirements of data transmission paths. In such cases, the number M can be significantly larger than the number K, exceeding it by two or even more times.

On Fig additionally given a matrix representation of the functions performed by the encoder unit and the decoder unit in the framework of the present invention. In most cases, blocks of sample values are processed. Therefore, as can be seen from equation (2), the audio object is displayed in the form of a series of L sample values. Matrix S contains N rows corresponding to the number of objects, and L columns corresponding to the number of samples. Matrix E is calculated according to equation (5) and includes N columns and N rows. Matrix E contains the parameters of the object when the parameters of the object are given in the energy mode. For uncorrelated objects, the matrix E, as shown in the context of equation (6), has only the main diagonal elements, each of which displays the energy of the audio object. All off-diagonal elements, as mentioned earlier, represent the correlation of two audio objects, which is especially important when several objects represent two channels of a stereo signal.

Depending on the design features, equation (2) represents a time-domain signal. After that, a single energy indicator is generated for the entire range of audio objects. However, it is preferable if the audio objects are processed by a time-frequency converter based, for example, on a conversion algorithm or a filter bank, and, in the latter case, equation (2) is valid for each subband, as a result of which matrix E is generated for each subband and definitely for every time interval.

The matrix of X down-mix channels has K rows and L columns and is calculated by equation (3). As can be seen from equation (4), the M output channels are calculated based on N objects using the so-called audio rendering matrix A for N objects. Depending on the situation, N objects can be reconstructed by a decoder unit using the results of down-mixing and object parameters, while audio rendering can be applied directly to the signals of reconstructed objects.

On the other hand, the down-mix array can be directly converted to the output channel signals without accurately calculating the source signals. The audio rendering matrix A mainly individually positions the sources in accordance with a predetermined configuration of the output audio signals. Suppose there are six objects and six output channels, then each object can be associated with each output channel, and this scheme will be reflected in the audio rendering matrix. However, if necessary, arrange all objects inside the acoustic space between two speakers, the audio rendering matrix A, reflecting the new positioning, will take a different look.

The audio rendering matrix, or in a more general sense, the planned spatial localization of objects, as well as the estimated ratio of the dynamic ranges of sound sources, can generally be calculated by the encoder and transmitted to the decoder in the form of a so-called scene description. However, in other embodiments, such a description of the scene can be performed directly by the user in order to generate a boost mixing set by himself to obtain a configuration of acoustic output signals set by himself. Thus, the transfer of the description of the scene is not a mandatory procedure, such a description of the scene can be implemented by the user to achieve the satisfaction of his own requests. The user can, for example, optionally localize some audio objects in places other than the positions in which these objects were originally located and which were generated for them. There are also cases when audio objects are implemented as such, without the presence of the “original” and its location relative to other, real, objects. In such situations, sound sources are initially positioned relative to each other by the user.

Returning to Fig. 9, we consider a downmixer 92. It is intended to reduce when mixing a phonogram of a plurality of audio objects to the number of downmix channels, the number of audio objects exceeding the number of downmix channels, while the downmix is coupled to the downmix information generator so that the distribution audio objects through a plurality of downmix channels is performed in accordance with the downmix indicators. The downmix indicators generated by the downmix information generator 96 of FIG. 9 can be automatically generated or manually controlled. It is recommended that downmix data be processed with a lower resolution than object parameters. Due to this, bits of overhead information can be saved without loss of quality, since fixed indicators of down-mixing for individual parts of the phonogram or a single slowly changing state of down-mixing, which does not require mandatory frequency selectivity, are quite sufficient. An embodiment of the invention is possible in which the downmix information is a downmix matrix having K rows and N columns.

The indicator in the row of the downmix matrix has a definite value when an audio object corresponding to this indicator in the downmix matrix is present in the downmix channel presented in the row of the downmix matrix. When an audio object is included in more than one downmix channel, more than one row of the downmix matrix has a specific value. Moreover, it is preferable if the quadratic values during addition for a single audio object give a total of no more than 1.0. However, other meanings are possible.

In addition, audio objects can be introduced into one or more downmix channels with different levels, and these levels can be indicated within the downmix matrix by weights other than unity and not totaling 1.0 for a particular audio object.

When the downmix channels are included in the encoded audio object signal generated by the output interface 98, the encoded audio object signal may be, for example, a time division multiplexed signal in a specific format. Conversely, the encoded signal of the audio object can be any signal that allows using the decoder unit to separate the parameters of the objects 95, the parameters of the downmix 97 and the channels of the downmix 93. In addition, the data output interface 98 may include encoders of the parameters of the objects, information downmix or downmix channels. The encoders for object parameters and for downmix data can be differential encoders and / or entropy encoders, and the encoders for downmix channels can be mono or stereo audio encoders, such as MP3 or AAC encoders (advanced audio codec). All these encoding operations result in additional data compression in order to subsequently reduce the data rate necessary for the encoded signal of the audio object 99.

Depending on the particular application of the downmixer 92, its functions provide for stereo representation of the musical background through at least two downmix channels and the introduction of a voice phonogram in these at least two downmix channels in a predetermined ratio. With this version of the implementation, the first channel of the background music passes through the first channel of the downmix and the second channel of background music goes through the second channel of the downmix. The result of this arrangement is the optimal stereo playback of musical background on stereo equipment. In this case, the user has the ability to position the voice phonogram between the left stereo speaker and the right stereo speaker. Alternatively, the first and second background music channels may pass through one downmix channel, and the voice phonogram may be conducted through another downmix channel.

Thus, excluding one channel of down-mixing, you can completely separate the voice phonogram from the background of musical accompaniment, which, in particular, meets the requirements of karaoke. However, at the same time, the quality of reproduction of the channels of the stereo soundtrack of musical accompaniment suffers due to the parameterization of the object, which, of course, is a lossy compression method.

The downmixer 92 has a configuration that allows the accumulation of countdown by counting over the time domain. For such a summation, samples of audio objects intended for down-mixing to one down-mixing channel are used. If an audio object is introduced into the downmix channel in a certain percentage, preliminary weighing must be performed before summing the samples. In addition, the summation can be performed in the frequency domain, or in the subband, that is, in the area following the time-frequency conversion. Thus, downmixing can be performed even in the filter bank region when the time-frequency transform is performed in the filter bank, or in the transform domain when the time-frequency transform is FFT (fast Fourier transform, FFT), MDCT (modified discrete cosine transform , MDKP), or any other conversion.

According to one aspect of the present invention, the object parameter generator 94 generates energy parameters and, optionally, correlation parameters between two objects when two audio objects together represent a stereo signal, as can be seen from the following equation (6). On the other hand, the parameters of the object are parameters of the prediction mode.

On Fig presents a block diagram of an algorithm or method for calculating the prognostic parameters of an audio object. As already explained with respect to equations (7) through (12), some statistical information regarding down-mix channels in matrix X and audio objects in matrix S are subject to calculation. In particular, block 150 shows the first step of calculating the real part S · X * and the real part X X *. These real parts are not just numbers, but matrices, and these matrices in one of the implementation options are determined through the notation system in equation (1) when considering the implementation following equation (12). In most cases, the values of step 150 can be calculated using the data available in the encoder of the audio objects 101. Then, as shown in step 152, the prediction matrix C is calculated. In particular, as is customary in the art, it is necessary to solve the system of equations in such a way that All values of the prediction matrix With the dimension of N rows and K columns were obtained. Basically, the weighting factors c _{n, i} , as in equation (8), are calculated so that a weighted linear summation of all down-mix channels reconstructs the corresponding audio object with the highest possible quality. Such a prediction matrix gives the better result of reconstruction of audio objects, the more the number of down-mix channels is activated.

Next, in more detail will be considered 11. In particular, FIG. 7 shows several types of output data used to create a plurality of output channels with a given output signal configuration. Line 111 shows the situation in which the output of the output synthesizer 100 is the reconstructed sound sources.

The input data required by the output data synthesizer 100 to reconstruct the audio sources includes downmix information, downmix channels, and audio object parameters. Moreover, for further reproduction of the reconstructed sources, there is no need to create the configuration of the output signal and pre-position the acoustic sources themselves within the spatial configuration of the output audio signal. In the mode indicated in FIG. 11 by number 1, reconstructed sources of audio signals will be generated at the output of the output data synthesizer 100. In the case of using the prediction parameters as parameters of the audio object, the output data synthesizer 100 operates according to the definition formulated in equation (7). When the parameters of the object are fixed in the energy mode, to recreate the original signals, the output synthesizer uses the inverse of the downmix matrix and the energy matrix.

Alternatively, the output synthesizer 100 may act as a transcoder, as shown, for example, in block 102 of FIG. 1b. When the output signal synthesizer is operating in transcoder mode generating surround mixer parameters, down-mix data, audio object parameters, output signal configuration and planned spatial localization of sound sources are required. In particular, the configuration of the output signal and the planned spatial positioning are provided using the audio rendering matrix A. However, to generate the parameters of the surround mixer, there is no need for down-mix channels, a more detailed explanation will be given in the context of Fig. 12. Depending on the situation, the surround mixer parameters generated by the output synthesizer 100 can then be directly used by the surround mixer like MPEG Surround to up-mix the down-mix channels. With this version of the design, the adjustment of the channels for down-mixing of objects is not necessary, it is sufficient to use a simple transformation matrix with only diagonal elements, which was described in relation to equation (13). In format 2, on line 112 in FIG. 11, the output synthesizer 100 outputs the parameters of the surround mixer and, preferably, the conversion matrix G according to equation (13), including gain factors that can be used as arbitrary gain parameters of downmix (ADG ) MPEG-surround decoder.

In format 3 on line 113 of FIG. 11, the output contains the surround mixer parameters in the form of a conversion matrix, such as shown in the context of equation (25). In this context, the output synthesizer 100 does not have to actually convert the downmix of the object into stereo downmix. The number 4 on line 114 in FIG. 11 designates a different format of the output data synthesizer 100 shown in FIG. 10. In this case, the transcoder operates as an element 102 in FIG. 1b, and outputs not only the surround mixer parameters, but also additional converted downmix results.

This eliminates the need for outputting the conversion matrix G in addition to the converted downmix. The output of the converted downmix and surround mixer parameters are sufficient, which is obvious from fig.1b.

Format 5 characterizes another application of the output data synthesizer 100 shown in FIG. 10. Under the conditions indicated on line 115 in FIG. 11, the output generated by the output synthesizer does not contain any surround mixer parameters, but only include, for example, the conversion matrix G according to equation (35) or actually contain directly stereo output signals, as shown in line 115. With this embodiment, only stereo audio rendering is of interest, and any surround mixer parameters are not required. However, to generate a stereo output, all available input information is required, as shown in FIG. 11.

Another mode of operation of the output synthesizer is displayed in format 6 on line 116. In this case, the output data synthesizer 100 generates a multi-channel output and is analogous to component 104 in FIG. 1b. For this, the output data synthesizer 100 needs all available input information, on the basis of which it generates a multi-channel output signal consisting of more than two output channels to be reproduced using the appropriate number of acoustic speakers localized in space in accordance with a given configuration of the output audio signal. This multi-channel output can be a 5.1-channel output, a 7.1-channel output, or a 3.0-channel output with left, center, and right speakers.

Next, reference is made to FIG. 11 for the purpose of explaining an example of calculating several parameters taken from an MPEG-surround decoder based on the parameterization principle presented in FIG. 7. As already mentioned, FIG. 7 illustrates a parameterization process using an MPEG-Surround decoder unit, starting from inputting a stereo down-mix 70 containing left I ₀ and right r ₀ down-mix channels. Schematically, both down-mix channels are input into the so-called two-to-three block 71. The two-to-three block is controlled by several input parameters 72. The block 71 generates three output channels 73a, 73b, 73c. Each output channel is inserted in a one-to-two block. This means that channel 73a is input to block 74a, channel 73b is input to block 74b, and channel 73c is input to block 74c. Each block has two output channels. Block 74a outputs the left front l _f and left panoramic l _s channels. Simultaneously unit 74b outputs a right front r _f and right panoramic r _s channels. At the same time, block 74c provides a central channel c and a low frequency optimization channel (lfe) at the output. It is important that the whole up-mix process from down-mix channels 70 to the output channels is carried out using a matrix operation, and the tree structure shown in Fig. 7 does not have to be implemented step-by-step, but can be implemented through one or several operations on matrices. Moreover, the intermediate signals, designated 73a, 73b and 73c, are not specifically calculated by any particular implemented device, but are shown in FIG. 7 for illustrative purposes only. At the same time, blocks 74a, 74b receive some residual signals res ₁ ^OTT , res ₂ ^OTT , which can be used to introduce a certain moment of randomness into the output signals.

и

. Вместе с тем, в блок 71 может быть введен показатель степени корреляции, ICC_TTT, который, однако, является лишь вспомогательной характеристикой, не обязательной к использованию в одном и том же варианте технического решения изобретения. На фиг.12 и 13 представлен алгоритм и/или необходимые средства вычисления всего комплекса параметров объекта 95 на фиг.9 - CPC/CLD_TTT, CLD0, CLD1, ICC1, CLD2, ICC2, информация по понижающему микшированию 97 на фиг.9 и планируемое пространственное позиционирование источников звука, например, описание сцены 101, как отображено на фиг.10. Эти параметры представляют собой предварительно задаваемый формат вывода звукового сигнала для 5.1-канальной системы звукового окружения.As is known from the description of the MPEG-surround decoder, block 71 is controlled using either CPC prediction parameters or CLDrrr energy parameters. For up-mixing from two channels to three channels, at least two prediction parameters CPC1, CPC2 or at least two energy parameters are required

and

. However, in block 71, an indicator of the degree of correlation, ICC _TTT , which, however, is only an auxiliary characteristic, not required for use in the same embodiment of the technical solution of the invention, can be entered. On Fig and 13 presents the algorithm and / or necessary means of calculating the entire complex of parameters of the object 95 in Fig.9 - CPC / CLD _TTT , CLD0, CLD1, ICC1, CLD2, ICC2, information on down-mix 97 in Fig.9 and the planned spatial positioning of sound sources, for example, a description of a scene 101, as shown in FIG. 10. These parameters are a predefined audio output format for a 5.1-channel surround sound system.

Naturally, such a special calculation of the parameters for this particular technical solution can be adapted to other signal output formats or types of parameterization in accordance with the concept of the present invention. Moreover, the sequence of steps of the algorithm or the layout of the funds in FIGS. 12 and 13a, b are given only as a good example and can undergo changes in the boundaries of the logic of mathematical equations.

Step 120 provides the functioning of the audio rendering matrix A. The audio rendering matrix positions each source from a plurality of sources in the acoustic space, taking into account a predetermined output signal configuration.

Step 121 provides the formation of a partial downmix matrix D ₃₆ in accordance with equation (20). This matrix allows downstream mixing from six output channels to three channels and has a 3 × N dimension. If necessary, generate a larger number of output channels than with 5.1 configuration, for example, when creating an 8-channel output signal format (7.1), the matrix presented in block 121 will become D ₃₈ matrix.

Step 122 provides the formation of a reduced matrix of audio rendering A ₃ by multiplying the matrix D ₃₆ and the full matrix of audio rendering, as defined in step 120.

Step 123 provides the introduction of the downmix matrix D. This downmix matrix D can be extracted from the encoded signal of the audio object when the matrix is entirely contained in this signal. Alternatively, the downmix matrix can be parameterized, for example, to introduce special downmix data and generate the downmix matrix G.

Step 124 provides in addition to this an energy matrix of the object. This energy matrix of the object is reflected in the object parameters for N objects and can be extracted from imported audio objects or reconstructed using a specific set of rules. Such a set of recovery rules may include entropy decoding and the like.

Step 125 provides the formation of a “reduced” C ₃ prediction matrix. The values of this matrix can be calculated by solving a system of linear equations according to step 125. In particular, the elements of the C ₃ matrix can be calculated by multiplying both sides of the equation by inversion (DED *).

Step 126 provides the calculation of the conversion matrix G. The conversion matrix G of dimension K * K is formed according to equation (25). To solve the equation in step 126, a special matrix D _{TTT is} needed, which is formed in step 127. An example for this matrix is given in equation (24), and the definition can be obtained from the corresponding equation for C _TTT , which is described by equation (22). Thus, equation (22) determines the procedure in step 128. Step 129 defines the equations for calculating the matrix _TTT . As soon as the matrix C _TTT is determined based on the equation of block 129, the parameters α, β, and γ, which are the parameters of the CPC (channel prediction coefficient), can be derived. It is recommended to set the γ value to 1, after which α and β will remain the only input parameters of the CPC in block 71.

The remaining parameters necessary for the algorithm in Fig. 7 are the parameters input to blocks 74a, 74b and 74c. The calculation of these parameters is described in the context of figa. Step 130 provides the formation of an audio rendering matrix A. The dimension of the audio rendering matrix A is N rows for the number of audio objects and M columns for the number of output channels. This audio rendering matrix contains information based on the scene vector when the scene vector is used. Most often, the audio rendering matrix includes information about a specific location in a given output signal configuration. If we consider the audio rendering matrix A, for example, in the context below equation (19), it becomes clear how a certain localization of objects in the structure of the audio rendering matrix can be encoded. Naturally, other methods of strictly defined positioning can be used, such as for values not equal to 1. In addition, using values, on the one hand, less than 1 and, on the other hand, more than 1, you can control the volume level of specific audio objects.

A design variant is possible in which the audio rendering matrix is formed by the decoder module without using any information from the encoder.

This enables the user to place audio objects arbitrarily at will, without taking into account their mutual spatial location, recorded by the encoder data.

A version of the technical solution is also possible in which the relative or absolute positioning of the acoustic sources can be encoded by the encoder module and transmitted to the decoder in the form of a certain scene vector. Then, on the decoder module, information regarding the localization of sound sources, preferably independent of the specified audio rendering settings, is processed to form an audio rendering matrix reflecting the spatial arrangement of the audio sources oriented to the specific configuration of the output audio signal.

Step 131 provides the formation of the matrix E of energy indicators of the object, which has already been considered in connection with step 124 in Fig. 12. This matrix has the dimension N × N and contains the parameters of the audio object. One of the embodiments of the invention provides a similar matrix of energy parameters of the object for each subband and each module of time samples or samples of subbands.

Step 132 provides a calculation of the matrix of energy parameters of the output signal F.

F is the covariance matrix of the output channels. Since the output channels remain uncertain in this case, the matrix F of energy parameters of the output signal is calculated using the matrix of audio rendering and the matrix of energy characteristics. These matrices are formed in steps 130 and 131 with direct access to the matrices in the decoder module. After that, using special equations (15), (16), (17), (18) and (19), the channel level difference indicators CLD ₀ , CLD ₁ , CLD ₂ and the interchannel coherence characteristics ICC ₁ and ICC ₂ are calculated with the aim of obtaining parameters for blocks 74a, 74b, 74c. It is important that the spatial characteristics are calculated by combining specific elements of the matrix of energy indicators of the output signal F.

In step 133, all parameters for the spatial boost mixer, such as, for example, schematically shown in FIG. 7, are prepared.

In the previously described implementations of the invention, the parameters of the object were presented as energy characteristics. However, when the parameters of the objects are given in the prognostic representation, that is, in the form of an object prediction matrix C, shown under paragraph 124a in Fig. 12, for calculating the reduced prediction matrix C _3, a simple matrix multiplication is sufficient according to the illustration of block 125a and explanations in the context of equation (32) . The matrix used in block 125a is the same matrix A ₃ that is mentioned in block 122 in FIG. 12.

When the object prediction matrix C is generated by the audio object encoder and transmitted to the decoder, additional calculations are required to prepare the parameters for blocks 74a, 74b, 74c. These auxiliary steps are presented in fig.13b. Again, the object prediction matrix C is formed as block 124a in FIG. 13b, which is similar to the description of block 124a in FIG. 12. Then, as described in connection with equation (31), the covariance matrix Z of the down-mix of the object is calculated using the transmitted down-mix or generated and transmitted as additional overhead information. After transmitting data about the Z matrix, the decoder should not perform any calculations of energy parameters, leading, in essence, to the resumption of the delayed processing of some data and to increase the total load of the decoder block. However, when these issues are not critical for a particular application, the bandwidth can be saved, and the covariance matrix Z of the downmix of an object can also be calculated using downmix samples, which are certainly available in the decoder module. Once the actions of step 134 are completed and the object down-mix covariance matrix is ready, the object energy parameters matrix E can be calculated according to the instructions of step 135 using the prediction matrix C and the down-mix covariance matrix or the “down-mix energy” matrix Z. At the end of step 135, all the above steps related to FIG. 13a can be performed, namely 132, 133, with the aim of generating all the necessary parameters for blocks 74a, 74b, 74c in FIG. 7.

On Fig presents yet another design solution that implements only stereo audio rendering. Stereo audio rendering is the formation of an output signal in accordance with mode number 5 or line 115 of FIG. 11. Here, the output synthesizer 100 in FIG. 10 does not require any spatial up-mix parameters, it mainly needs a special conversion matrix G to convert the downmix of the object into a functional and, of course, quickly tuned and easily controlled stereo down-mix.

Step 160 in FIG. 16 includes calculating a partial downmix matrix from M to 2 channels. In the variant with six output channels, the partial downmix matrix will act as a downmix matrix from six to two channels, while maintaining the possibility of using other downmix matrices. The calculation of such a partial downmix matrix can be performed, for example, by deducing from the partial downmix matrix D ₃₆ , as was the case in step 121, and the _TTT matrix D, as was done in step 127 of Fig. 12.

In addition, based on the result of step 160, the stereo audio rendering matrix A _{2 is} generated, and the “large” audio rendering matrix A is presented in step 161. The audio rendering matrix A is the same matrix that was considered in connection with block 120 of FIG. 12.

Next, at step 162, the stereo-audio rendering matrix can be parameterized by the localization factors µ and κ. If both μ and κ are set to 1, then equation (33) is obtained, which makes it possible to vary the dynamic range of the voice, which has already been described in the example given in the context of equation (33). At the same time, when changing other parameters, such as µ and κ, the arrangement of sources can also vary.

Then, as shown in step 163, the conversion matrix G is calculated using equation (33).

Corrections made to the description

In particular, the matrix (DED *) can be calculated, inverted, and the inverted matrix can be multiplied by the right side of the equation of block 163. Of course, other methods of solving the equation of block 163. can also be applied. After the conversion matrix G, which reduces the mixing, is obtained object X can be transformed by multiplying the conversion matrix and down-mixing the object, which is displayed in block 164. After that, stereo audio rendering of the converted down-mixing X 'can be performed using I eat two stereo speakers. Depending on the technical solution, for μ, ν and κ, certain values can be set for calculating the conversion matrix G. Or, the conversion matrix G can be calculated using all these three parameters as variables so that the parameters are set in accordance with the requirements user after passing step 163.

In preferred embodiments of the invention, solutions have been found to the problem of transmitting several independent audio objects (using multi-channel downmixing and auxiliary control data describing objects and audio rendering of objects for a given playback system (speaker configuration)). A method for converting object-related control data into control data compatible with a reproduction system is introduced. The following are suitable coding methods based on the MPEG Surround coding algorithm.

Depending on the technical requirements of a particular embodiment, the input methods and the resulting signal may take the form of implementation in hardware or in software. This part of the invention can be implemented using a digital storage medium, in particular a disk or CD, designed to store control signals in an electronically readable form compatible with a programmable computer system so that the input methods can be performed. Thus, in a general sense, the present invention is a computer program product with software code assigned to it stored on a machine-readable drive, and designed to perform at least one of the invented methods when starting this software product on a computer. Stated differently, the invented methods are, therefore, a computer program having program code designed to implement the invented methods when running this program on a computer.

In other words, the embodiment of the present invention is an audio object encoder for generating an encoded audio object signal as one of a plurality of audio objects, including a downmix information generator for generating downmix information showing a distribution order of a plurality of audio objects between at least two downmix channels;

generator of audio object parameters; and an output interface for generating a coded signal of the audio object using downmix information and object parameters.

Alternatively, the data output interface may generate an encoded audio signal, additionally using a plurality of downmix channels.

In addition, or instead, the parameter generator is characterized in that it is capable of generating object characteristics with primary time and frequency resolution, and in cases where the down-mixing information generator has the function of generating down-mixing information with secondary time and frequency resolution, the secondary resolution is time and frequency lower than the primary.

In addition, the down-mix information generator is characterized in that it is able to generate down-mix information so that the down-mix parameters uniformly cover the entire frequency range of audio objects.

In addition, the downmix information generator is characterized in that it is capable of generating downmix information such that the downmix information may comprise a downmix matrix, defined as:

X = DS

where S is a matrix representing audio objects and containing the number of rows equal to the number of audio objects,

where D is a downmix matrix, and

where X is a matrix representing a plurality of downmix channels and containing a number of lines equal to the number of downmix channels.

In addition, information about a part of an object may have an indicator less than 1 and greater than 0.

In addition, the downmixer is characterized in that it is able to generate a stereo background representation of the musical background through at least two downmix channels and enter a voice phonogram in at least these two downmix channels in a predetermined ratio.

In addition, the downmixer is characterized in that it is capable of adding up the signal samples for further introduction into the downmix channel according to the downmix data.

In addition, the data output interface is characterized in that it is capable of compressing downmix data and object parameters before generating an encoded audio object signal.

In addition, a plurality of audio objects may include a stereo object represented by two audio objects with some non-zero correlation and containing grouping data generated by a downmix information generator pointing to these two audio objects forming this stereo object.

In addition, the object parameter generator is characterized in that it is able to generate the prediction parameters of audio objects, calculating them in such a way that a weighted addition of down-mix channels to the original object controlled by the prediction parameters, or simply to the original object, results in an approximation of the source object.

In addition, the prediction parameters can be formed based on the frequency band, and audio objects cover the entire frequency range.

In addition, the number of audio objects can be equal to N, the number of down-mix channels is K, and the number of object prediction parameters calculated by the object parameter generator is equal to or less than N · K.

In addition, the object parameter generator is characterized in that it is able to calculate the largest number of object prediction parameters K · (N-K).

In addition, the object parameter generator may include an upmixer to increase the number of channels obtained by downmixing using various combinations of controlled object prediction parameters;

at the same time, the encoder of audio objects included in the upmixer includes an iterative controller designed to detect the prediction parameters of the object to be tested, which minimizes the deviation of the signal reconstructed by the upmixer from the corresponding original signal among various sets of monitored object prediction parameters .

In addition, the output synthesizer is characterized in that it is able to determine the conversion matrix using the downmix information, the conversion matrix being calculated in such a way that the arrangement of the downmix channels is at least partially changed when the audio object contained in the first downmix channel representing the first half of the stereoscopic plane should be reproduced in the second half of the stereoscopic plane.

In addition, the audio synthesizer may include an audio renderer of channels designed to perform audio rendering of the output audio channels to produce an acoustic signal of a predetermined configuration through the use of spatial parameters and at least two down-mix channels or converted down-mix channels.

In addition, the output data synthesizer is characterized in that it is capable of generating output audio channels of a given configuration, additionally involving at least two down-mix channels.

In addition, the output synthesizer is characterized in that it is able to calculate the actual downmix weights for the partial downmix matrix so that the energy of the weighted sum of the two channels is equal to the energies of the channels within the constraint.

In addition, the downmix weights for the partial downmix matrix can be determined as follows:

, p=1, 2, 3,

, p = 1, 2, 3,

where w _p is the weight of the downmix, p is an integer index variable, f _{j, i} is the cell of the energy characteristics matrix, representing the approximation of the covariance matrix of the output channels, a predefined output signal configuration.

In addition, the output synthesizer is characterized in that it is able to calculate the individual coefficients of the prediction matrix by solving a system of linear equations.

In addition, the output synthesizer is different in that it is able to solve a system of linear equations based on:

C ₃ (DED *) = A ₃ ED *,

where C ₃ is a two-to-three prediction matrix, D is a down-mix matrix derived from down-mix information, E is an energy characteristics matrix derived from the original audio objects, and A ₃ is a reduced down-mix matrix, and where "*" denotes a complex conjugate operation.

In addition, the prediction parameters for up-mixing two-to-three can be obtained by parameterizing the prediction matrix so that the prediction matrix is determined by only two parameters, and

the output synthesizer is characterized in that it is able to pre-process at least two down-mix channels in such a way that the result of the pre-processing and the parameterized prediction matrix correspond to the desired up-mix matrix.

In addition, the parameterization of the prediction matrix can be as follows:

where the TTT index is the parameterized prediction matrix, and α, β, and γ are the coefficients.

In addition, the downmix conversion matrix G can be calculated as follows:

G = D _TTT C ₃

where C ₃ is a two-to-three prediction matrix, where D _TTT and C _TTT are equal to I, where I is a two-to-two prediction matrix, and where C _{TTT is} based on:

where α, β, γ are constant coefficients.

Further, the prognostic parameters for up-mixing two-to-three can be defined as α and β, with γ being set as 1.

In addition, the output synthesizer is characterized in that it is able to calculate the energy parameters for up-mixing three-to-six using the matrix of energy characteristics F based on:

YY * ≈F = AEA *,

where A is the matrix of audio rendering, E is the matrix of energy characteristics formed on the basis of audio source objects, Y is the matrix of the output channel, and "*" is an indicator of a complex conjugate operation.

In addition, the output data synthesizer is characterized in that it is able to calculate energy parameters by combining elements of the matrix of energy characteristics.

In addition, the output synthesizer is characterized in that it is able to calculate energy parameters based on the equations below:

,

,

,

,

,

,

,

,

,

,

where φ is the absolute value of φ (z) = | z | or the operator of the real value φ (z) = Pe {z},

where CLD ₀ is the first energy parameter of the channel level difference, where CLD ₁ is the second energy parameter of the channel level difference, where CLD ₂ is the third energy parameter of the channel level difference, where ICC ₁ is the first energy parameter of interchannel coherence, and ICC ₂ is the second energy parameter inter-channel coherence, and where f _{i, j} are the elements of the matrix of energy characteristics F at positions i, j in this matrix.

In addition, the first group of parameters may contain energy parameters, and the output data synthesizer is characterized in that it is able to generate energy parameters by combining elements of the matrix of energy characteristics F.

In addition, energy parameters can be obtained on the basis that:

,

,

,

,

- первый энергетический параметр первой группы, и где

- второй энергетический параметр первой группы параметров.Where

is the first energy parameter of the first group, and where

- the second energy parameter of the first group of parameters.

In addition, the output synthesizer is characterized in that it is able to calculate weights for weighting the down-mix channels, weights for controlling the arbitrary down-mix gain (ADG) of the spatial decoder.

In addition, the output data synthesizer is characterized in that it is able to calculate weighting factors based on:

Z = DED *,

W = D ₂₆ ED * ₂₆ ,

,

,

where D is the down-mix matrix, E is the energy characteristics matrix obtained from the source audio objects, where W is the intermediate matrix, where D ₂₆ is the partial down-mix matrix to reduce the number of channels from 6 to 2 with the given output signal configuration, and where G is a transformation matrix containing arbitrary gain coefficients of the down-mix of the spatial decoder.

In addition, the output data synthesizer is characterized in that it is able to calculate the matrix of energy characteristics based on:

E = CZC *,

where E is the matrix of energy characteristics, C is the matrix of the prediction parameter, and Z is the covariance matrix of at least two downstream channels.

In addition, the output data synthesizer is characterized in that it is able to calculate the conversion matrix based on:

G = A ₂ · C,

where G is the transformation matrix, A ₂ is the incomplete matrix of audio rendering, and C is the matrix of prediction parameters.

In addition, the output data synthesizer is characterized in that it is able to calculate the conversion matrix based on:

G (DED *) = A ₂ ED *,

where G is the matrix of energy characteristics formed on the basis of the sound source in the phonogram, D is the matrix of downmixing, obtained on the basis of information on downmixing, A ₂ is the reduced matrix of audio rendering, and "*" is an indicator of the complete conjugate operation.

In addition, a parametrized stereo audio rendering matrix A ₂ can be formed as follows:

where µ, ν, and κ are actual parameters specified in accordance with the location and dynamic range of one or more source audio objects.

Claims (13)

1. An audio synthesizer (104) designed to generate output using an encoded audio object signal (95, 97), characterized in that it includes an output data synthesizer (100) that generates output parameters that are suitable for representing a variety of output channels from a preliminary a predetermined configuration of the output audio signal displaying a plurality of audio objects, the output data synthesizer provides for the possibility of using downmix information containing a decree distribution of a plurality of audio objects along at least two down-mix channels, and an object parameter for audio objects, wherein the output data synthesizer (100) transcodes (502) the audio object parameters into spatial parameters for a predetermined configuration of the output audio signal, additionally using a predetermined location ( A) audio objects (90) in the configuration of the output audio signal. 2. The audio synthesizer according to claim 1, characterized in that the output data synthesizer (100) is designed to convert a plurality of down-mix channels into a stereo down-mix for a predetermined configuration of the output audio signal using a transform matrix (G) generated based on a given spatial location audio objects (A). 3. The audio synthesizer according to claim 1, characterized in that the spatial parameters include a first group of parameters for up-mixing two-to-three and a second group of energy parameters for up-mixing "three-to-six", and the synthesizer output (100) provides for computing prediction parameters for predicting matrix "two-to-three" using the presentation of the matrix (a) according to a predetermined arrangement of audio objects (90), the partial downmix matrix (D ₃₆₎ describing at "Two-to-three", and the downmix matrix (D) izhayuschee mixing output channels to three channels generated by a hypothetical operation during upmixing. 4. The audio synthesizer according to claim 3, characterized in that in it the parameters of the object are the parameters of the prediction of the object, and as part of which the synthesizer of the output data (100) preliminary calculates the energy matrix (E) using the parameters of the prediction of the object (C), information reducing mixing (D) and energy indicators (Z) corresponding to the downmix channels. 5. The audio synthesizer according to claim 1, characterized in that the output data synthesizer (100) generates (165) two stereo channels for generating a stereo output signal configuration by calculating a parameterized stereo representation matrix (A ₂ ) and a transformation matrix (G) depending on the parameterized stereo representation matrix (A ₂ ). 6. A method of synthesizing sound, comprising generating output data using an encoded signal of an audio object (95, 97), characterized in that it includes generating output data for generating a plurality of output channels with a given configuration of an output audio signal displaying a plurality of audio objects (90), using downmix information indicating the distribution order of a plurality of audio objects over at least two downmix channels and audio parameters object for audio objects, and the parameters of the audio object are transcoded (502) into the spatial parameters of the calculated configuration with additional consideration for the data of the specified location (A) of the audio objects (90) in the configuration of the output audio signal. 7. An encoder of audio objects (101), designed to generate encoded signals of audio objects of a plurality of audio objects (90), characterized in that it includes a down-mix information generator (96) for generating down-mix information (97) reflecting the distribution order of the plurality of audio objects, at least between two downmix channels; wherein the down-mix information generator (96) is configured to generate (150) energy characteristics (XX *) and correlation data (SX *) reflecting the power characteristics and correlation characteristics of these at least two down-mix channels (93); audio object parameter generator (94); and an output interface (98) for outputting the generated encoded signal of the audio object (99), wherein the encoded signal of the audio object contains downmix information, power information, correlation information and object parameters. 8. The audio object encoder according to claim 7, characterized in that it further includes a down-mixer (92) for down-mixing a plurality of audio objects over a plurality of down-mixing channels, where the number of audio objects exceeds the number of down-mixing channels and where the down-mixer (92) is connected to a generator downmix information so that the distribution of the plurality of audio objects across the plurality of downmix channels is performed in accordance with the downmix information vania. 9. The audio object encoder according to claim 7, characterized in that the downmix information generator (96) calculates the downmix parameters in such a way that it contains information about an audio object that is fully or partially distributed over one or more of the many downmix channels, and when distributing an audio object over more than one down-mix channel, about a part of audio objects distributed over one down-mix channel from several down-mix channels . 10. A method of encoding audio objects (101) with generating an encoded signal of a plurality of audio objects, characterized in that it includes generating down-mix information (97) containing instructions for distributing the plurality of audio objects (90) over at least two down-mix channels, generating ( 150) energy indicators (XX *) and correlation data (SX *), reflecting the power characteristics and correlation characteristics of these at least two down-mix channels; development of parameters of audio objects (94); and issuing the encoded signal of the audio object (99), wherein the encoded signal of the audio object contains power information, correlation information, downmix information and object parameters. 11. A computer-readable storage medium with an encoded audio object signal stored on it, characterized in that it contains downmix information defining the distribution order of the plurality of audio objects over at least two downmix channels, energy metrics (XX *) and correlation data (SX *), reflecting the power characteristics and correlation characteristics of these at least two down-mixing channels, and the parameters of objects, which, in combination with, at To the least, recreate audio objects with these two channels of reduction. 12. Machine-readable medium with a computer program stored thereon for implementing the method according to claim 6. 13. A machine-readable medium with a computer program stored thereon for implementing the method of claim 10.

Images