RU2439719C2 - Device and method to synthesise output signal - Google Patents

Abstract

FIELD: information technologies. ^ SUBSTANCE: device to synthesise an output signal comprises a decorrelator (356) and a convergence assembly (364). The decorrelator (356) provides for a decorrelated single-channel signal or decorrelated signals of the first and second channels on the basis of a step-down mixing signal, which contains signals of step-down mixing of the first and second audio objects, and is a result of step-down mixing of multiple signals of audio objects in compliance with information on step-down mixing (354). The convergence assembly (364) performs weighted convergence of a step-down mixing signal (352) and a decorrelated signal (358) with application of weight ratios (P, Q, C0, G), calculation of which is carried out on the basis of data on step-down mixing (354), specified characteristics of audio rendering (360), determining a virtual position of audio objects in a virtual sound space, and parametric data on audio objects (362), which describe these audio objects. ^ EFFECT: development of a method for audio object coding, in which quality of objects reproduction in a stereo soundtrack will be most close to reference stereo sound transfer. ^ 28 cl, 24 dwg

Description

The present invention relates to synthesizing an output signal using audio rendering, for example a stereo output signal or an output multi-channel audio signal generated as a result of downmixing using additional control data. Moreover, the multi-channel down-mix is a down-mix of a plurality of signals of an audio object.

Recent developments in the field of audio technology contribute to the improvement of multi-channel playback of an audio signal based on a stereo (or mono) signal and the corresponding control commands. These methods for parametric coding of the surround sound environment typically include parameterization. A parametric multichannel audio decoder (e.g. MPEG Surround decoder described by ISO / IEC 23003-1 (L. Villemoes, J. Herre, J. Breebaart, G. Hotho, S. Disch, H. Purnhagen, and K. Kjörling, "MPEG Surround: The Forthcoming ISO Standard for Spatial Audio Coding, "in the 28th International AES Conference, The Future of Audio Technology Surround and Beyond, Pitee, Sweden, June 30 - July 2, 2006 (" MPEG Surround: preparing a new ISO standard for spatial audio coding ", Proceedings of the 28th AES International Conference" Advanced Audio Technologies, Current State and Prospects ", Pete, Sweden, June 30 - July 2, 2006, and Breebaart, J. Herre, L. Villemoes, C. Jin, K. Kjörling , J.Plogsties, and J.Koppens, "Multi-Channels goes Mobi le: MPEG Surround Binaural Rendering, "in the 29th International AES Conference, Audio for Mobile and Handheld Devices, Seoul, Sept 2-4, 2006" Mobile multi-channel devices: stereo presentation of MPEG Surround format ", 29th AES International Conference," Sound for mobile and portable devices, "Seoul, September 2-4, 2006), reconstructs M channels using K transmitted channels, where M> K, using control data. The control data contains multichannel signal parameterization data derived from IID (inter-channel power difference) and ICC (inter-channel coherence). These parameters, obtained, as a rule, at the coding stage, describe the power ratio and correlation of the channel pairs involved in the upmixing process. The use of such a coding algorithm allows coding at a data rate much lower than the transmission of all M channels, providing very high coding efficiency and guarantee compatibility with devices with K channels and with devices with M channels.

A similar encoding system is the corresponding audio object encoding device, which provides for down-mixing of several audio objects during encoding and then performing up-mixing using C.Faller control data, "Parametric Joint-Coding of Audio Sources," Convention Paper 6752 presented at the 120th AES Convention, Paris , France, May 20-23, 2006 (Combined Parametric Coding of Sound Sources. Report 6752 at the 120th AES Conference, Paris, France, May 20-23, 2006; C. Faller, Parametric Joint-Coding of Audio Sources, "Patent application PCT / EP2006 / 050904, 2006" Combined full parametric coding of sound sources ”). Upmixing can also be considered as the separation of objects mixed during the downmixing process (The resultant upmixing signal can be converted for playback in single or multi-channel mode). More specifically, the indicated sources propose a method for synthesizing audio channels based on a downmix (the so-called total signal obtained as a result of down-mixing), statistical information about the source objects and parameters describing the required output format. If several signals obtained by downmixing are used, these signals consist of subsets of different objects, and upmixing should be carried out individually for each channel of downmixing.

Based on the prior art, it is known that when down-mixing a stereo object and converting the object to stereo format or when generating a stereo signal for subsequent processing, for example, using the MPEG Surround decoder, significant operational advantages are achieved due to pair processing of channels using a time-frequency matrix-based matrixing algorithm. Outside of the encoding of audio objects, a similar procedure is used to partially convert one stereo audio signal to another stereo audio signal in WO 2006/103584. In addition, it is known that for a standard coding system of an audio object, it is necessary to additionally introduce a decorrelation procedure into the audio rendering process in order to perceptually reproduce a given scene. Moreover, the prior art does not contain a description of the comprehensive optimization of coding using a matrix system (matrixing) and decorrelation. A simple combination of well-known technical solutions leads either to an ineffective and inflexible use of the capabilities of a multi-channel down-mix of an object, or to a poor quality of the stereo object of the source object resulting from audio rendering.

An object of the present invention is an adjusted concept for synthesizing an output signal using audio rendering.

This problem is solved by means of a device synthesizing an output signal converted by audio rendering in accordance with claim 1, by a method of synthesizing an output signal using audio rendering according to claim 27, or by a computer program according to claim 28.

The present invention provides the synthesis of an output signal converted by audio rendering, integrating the signals of two or more audio channels. In this case, when converting a plurality of audio objects, the number of synthesized signals of the audio channels is always less than the number of source audio objects. However, if the number of audio objects is not large (for example, 2), and the number of output channels is 2, 3 or more of the output audio channels may exceed the number of audio objects. The synthesis of the output signal using audio rendering is carried out without performing the operation of full decoding of the audio object before decoding the components of the audio objects and without subsequent rendering of the synthesized audio objects according to the specified parameters. Instead, convertible output signals are calculated in the parameter value range based on the downmix characteristics, based on predefined audio rendering characteristics and on the basis of audio objects describing these audio objects in the form of energy and correlation parameters. Thus, the number of decorrelators, significantly complicating the implementation of the synthesis device, can be reduced so that it will be less than the number of output channels and even significantly less than the number of audio objects. More specifically, high-quality sound can be synthesized when synthesizers are implemented with only one or two decorrelators. Moreover, due to the absence of the need for full decoding of audio objects and subsequent audio rendering according to the specified parameters, a saving in memory and computing resource is provided. In addition, each operation potentially introduces additional artifacts. Because of this, in accordance with the present invention, preference is given to calculations only in the range of parameter values, so that at least the two audio signals represented, not as parameters, but in the form of, for example, time-domain signals or frequency-domain signals signal, which are obtained as a result of down-mixing of the object. In acoustic synthesis, they are introduced into the decorrelator both in the form of a downmix, if one decorrelator is used, and in a mixed form, if a separate decorrelator is used for each channel. Other actions performed in the time domain or in the area of the filter bank or with mixed channel signals are only weighted combinations, such as weighted addition or weighted subtraction, that is, linear operations. This prevents the introduction of artifacts during the complete decoding of an audio object and subsequent audio rendering according to specified parameters.

Information about the audio object is preferably given in the form of energy parameters and correlation data, for example, in the form of a covariance matrix of the object. In addition, such a matrix should preferably be applicable for each subband and each time interval, which would be reflected in the time-frequency map, where each slot contains a covariance matrix of an audio object that describes the energy of the corresponding audio objects in this subband and the correlation between the corresponding pairs of audio objects in the corresponding subrange. Naturally, this information is tied to certain time frames of a frequency signal or an audio signal.

The preferred result of audio synthesis is a converted stereo output signal that integrates the signal of the first or left sound channel and the signal of the second or right sound channel. Thanks to this, it is possible to achieve such a level of coding of an audio object at which the quality of reproduction of objects in a stereo phonogram will be as close as possible to a reference stereo sound transmission.

In many cases, when encoding audio objects, it is very important that the stereo-audio rendering of objects is as close as possible to the reference stereo audio transmission. The high quality of acoustics obtained as a result of stereo-audio rendering, close to the reference stereo-phonogram, is important both in the case when stereo-audio rendering provides the final output signal of the object decoder, and in the case when the stereo signal is transmitted to the next contour element, such as the MPEG Surround decoder operating in stereo downmix.

The present invention provides a method for the most efficient combination of matrixing and decorrelation, which enables the audio object decoder to fully utilize the potential of the audio object encoding algorithm using the down-mix signal of the object simultaneously on several channels.

The implementation of this invention is characterized in that it includes:

- an audio object decoder, which is intended for audio rendering of a plurality of individual audio objects using multi-channel down-mix, control data describing audio objects, control data describing down-mix, and audio rendering parameters, including in its construction:

- a stereo processor, which includes an integrated matrix coding unit (complex matrixing module), which linearly converts the channels of the multi-channel down-mix to obtain the original raw (dry mix = dry mix) signal and the decorrelator input signal, followed by the transfer of the decorrelator input signal to the decorrelator, the output signal of which is linearly reduced to a signal which, after summing with the original unprocessed signal, forms the stereo output signal of the module in the channel complexly go matrixing; or

- matrix calculation unit (matrix calculator), which calculates based on control data describing audio objects, down-mix and stereo audio rendering parameters, weights for linear information, which are then used by the integrated matrix encoding unit.

The invention is further presented by illustrative material, which does not limit it either in form or in essence, with a description of the accompanying drawings,

where figure 1 is a flowchart of a process for encoding an audio object, including encoding and decoding;

on figa given a flowchart of the decoding process of an audio object to a stereo;

2B is a flowchart of a decoding process of an audio object;

on figa given a block diagram of the operation of the stereo processor;

on figb given a block diagram of a device for synthesizing a converted audio rendering output signal;

on figa presents a first embodiment of the present invention, including a matrix C ₀ mixing the original signal, matrix Q pre-correlation signal mixing and matrix P up-mixing signal of the decorrelator;

on figv presents an embodiment of the present invention without the use of a matrix of pre-correlation signal mixing;

on figs presents a variant of the technical solution of the present invention without the use of matrix up-mixing decorrelated signal;

on fig.4D presents a version of the implementation of the present invention using the additional matrix G gain compensation;

FIG. 4E illustrates an embodiment of a downmix matrix Q of a decorrelator signal and a matrix P of upmix of a decorrelator signal using a single decorrelator;

on fig.4F presents an embodiment of the matrix With ₀ mixing the original signal;

on fig.4G is given a detail of the process of mixing the mixed source signal and the signal of the decorrelator or signal obtained as a result of up-mixing of the signal of the decorrelator;

figure 5 is a block diagram of a multi-channel decorrelation using multiple decorrelators;

Fig.6 is a summary map of several audio objects having an individual identifier, an audio file of an object, and a common matrix E of parameters of audio objects;

in Fig.7 is given the decoding of the covariance matrix E of the objects shown in Fig.6;

Fig. 8 shows a downmix matrix and an audio object encoder controlled by a downmix matrix D;

figure 9 shows the matrix A of the audio rendering parameters, presented, as a rule, by the user, and a particular example of the audio rendering script for the given parameters;

figure 10 shows the algorithm for preliminary calculations of the matrix elements for four different options presented on figa-4D;

11 shows an algorithm for calculating matrices for the first embodiment;

Fig. 12 shows a matrix calculation algorithm for a second embodiment;

on Fig shows a matrix calculation algorithm for the third embodiment; and

on Fig shows a matrix calculation algorithm for the fourth embodiment.

Embodiments of the present invention are presented below solely for the interpretation of the basic principles of the DEVICE AND METHOD FOR SYNTHESIS OF THE OUTPUT SIGNAL. It is understood that for specialists in this field, the possibility of making changes and improvements to the layout and elements of the described construction is obvious. Therefore, the descriptions and explanations provided for the embodiments of the invention are limited only by the scope of patent requirements, and not by specific details.

Figure 1 is a flowchart of an audio object encoding process in which an object encoder 101 and an object decoder 102 are involved. A spatial audio object encoder 101 encodes N objects into downlink object mixing data over K> 1 audio channels in accordance with the encoder parameters. The object encoder displays information about the applied downmix weight matrix D in combination with auxiliary power data and downmix correlation. Matrix D is often, but not necessarily always, constant in time-frequency characteristics, and therefore information about it requires a relatively small amount. Finally, the object encoder extracts the parameters of each object in the form of a time-frequency function with a resolution determined by the conditions of perception. The decoder of the spatial audio object 102 receives input down-mix channels of the object, down-mix data and object parameters (generated by the encoder) and generates an output signal consisting of M sound channels for listening. Audio rendering of N objects in M audio channels is performed using a sound transmission matrix (audio rendering) inputted to the object decoder as user-defined characteristics.

On figa shows the constituent elements of the decoder of the audio object 102 in the configuration, the output of which is set as a stereo. The downmix of the audio object is inserted into the stereo processor 201 to process the signals to produce a stereo signal. The signal processing depends on the characteristics of the matrix calculated by the matrix calculator 202. The information on the matrix is derived from the object parameters, down-mix data and user-defined characteristics of the object's audio rendering, describing the sound transmission requirements of N objects in stereo format using the audio rendering matrix.

FIG. 2B shows the constituent elements of an audio object decoder 102 in a configuration, the output of which is defined as a common multi-channel audio signal. The downmix of the audio object is inserted into the stereo processor 201 to process the signals to produce a stereo signal. The signal processing depends on the characteristics of the matrix calculated by the matrix calculator 202. The information on the matrix is derived from the object parameters, downmix data, and the object’s audio rendering characteristics processed and transmitted by the audio rendering data mixing unit 204. The object’s audio rendering information contains a description of the set conversion characteristics of N objects to a stereophonogram using an audio rendering matrix and is formed on the basis of audio rendering data describing the conversion N objects in M sound channels transmitted to the decoder of the audio object 102, object parameters, and object down-mix data. The auxiliary processor 203 converts the stereo signal received from the stereo processor 201 into a final multi-channel audio output signal generated based on audio rendering information, downmix information and object parameters. Typically, the role of the basic element of the auxiliary processor 203 is played by an MPEG Surround format decoder operating in a stereo downmix mode.

3A is a diagram of a stereo processor 201 device. If an object downmix is considered as a binary data stream coming from a K-channel audio encoder, this bitstream is first decoded by an audio decoder 301 into K time-domain audio signals. Further, using the time-frequency converter 302, all these signals are transformed into the frequency domain. The improved time-frequency conversion matrixing of the invention, the characteristics of which are defined by matrix data supplied to the stereo processor 201, is performed by the complex matrixing module 303 using the resulting frequency domain X signals. Received at the output of module 303, the stereo signal Y ′ in the frequency domain is converted using the time-frequency converter 304 into a time-domain signal.

FIG. 3B shows a device synthesizing an audio-converted output signal 350, including in the case of stereo-audio rendering, a signal of a first audio channel and a signal of a second audio channel or signals of a larger number of audio output channels in the case of multi-channel audio rendering. However, for a larger number of audio objects, for example three or more, the number of output channels should be less than the number of original audio objects integrated in the signal obtained by downmix 352. Namely, the signal 352 obtained by downmix contains at least a downmix signal of the first the object and the downmix signal of the second object, the signal obtained by downmixing is the result of downmixing a plurality of signals of audio objects, corresponding to the downmix information entered 354. Namely, the decorrelator 356 incorporating a decorrelated signal that includes a decorrelated single channel signal or, in the case of two decorrelators, a first decorrelated channel and the signal of the second decorrelated channel, or, in an embodiment with three or more decorrelators, including signals of the number of decorrelated channels, b more than two. At the same time, fewer decorrelators and, therefore, fewer signals of decorrelated channels are preferable, since each decorrelator significantly complicates the design. The preferred number of decorrelators is less than the number of audio objects integrated in the downmix signal 352 and equal to the number of channels in the output signal 352 or less than the number of audio channels in the audio output converted signal 350. However, for a small number of audio objects (for example, 2 or 3) the number of decorrelators can be equal to or greater than the number of audio objects.

As shown in FIG. 3B, the decorrelator receives a downmix signal 352 at the input and generates an output decorrelated signal 358. The downmix information 354 is supplemented with data set for audio rendering 360 and parameters of the audio object 362. In particular, the parameters of the audio object are used at least information unit 364 and can optionally be used decorrelator 356, which will be described below. The parameters of the audio object 362 should advantageously include energy and correlation data parametrically describing the audio object by a number within the range of values between 0 and 1, or by a number in a predetermined range of values, serving as an indicator of energy, power, or correlation between two audio objects, as will be discussed later.

The mixer 364 is designed for weighted input of the downmix signal 352 and the decorrelated signal 358. In addition, based on the parameters of the downmix 354 and the specified characteristics of sound transmission (audio rendering) 360, the information unit 364 calculates weighting factors for the weighted information. The set audio rendering parameters simulate the actual location of audio objects in a virtual sound environment and position audio objects, determining for playback each of them the first or second output channel, that is, the left output channel or the right output channel in stereo audio rendering. At the same time, with multi-channel audio rendering, the approximate position relative to the left panned (surround / spatial) or right panned or central acoustic channel, etc., is additionally determined in the set sound transmission characteristics for each channel. It is possible to implement any audio rendering scenarios, which, however, will differ from each other by sound transmission parameters, predefined mainly in the form of an audio rendering matrix, provided, as a rule, by the user, which will be described later.

And finally, the information block 364 uses the parameters of the audio object 362 containing a description of the audio objects, mainly in the form of energy indicators and correlation data. As an embodiment, it is allowed that the parameters of the audio object are presented in the form of a covariance matrix of the audio object for each “cell” of the time-frequency plane. Formulating differently, for each subband and for each time interval within which this subband is selected, the parameters of the audio object 362 determine the complete covariance matrix of the object, that is, the matrix containing power / energy information and correlation information.

When comparing figv and figa or 2B shows that the decoder of the audio object 102 in figure 1 corresponds to a device for synthesizing the output signal using audio rendering.

In addition, the decorrelation phase 356 shown in Fig. 3B is included in the functions of the stereoprocessor 201. At the same time, the mixing unit 364 includes a matrix calculating unit 202 shown in Fig. 2A. At the same time, if the decorrelator 356 performs a downmix operation, here, part of the functions of the matrix calculator 202 lies on the decorrelator 356, and not on the information block 364.

However, the binding of the performed functions to specific components of the element base is not critical in the implementation of the present invention, since its scope extends to software, and to special means of digital signal processing, or even to general-purpose personal computers. Therefore, correlation of a specific function with a specific module is only one of the approaches to the technical implementation of this invention. However, if all the schematic diagrams of constructive solutions are considered as block diagrams illustrating the algorithms of working operations, certain functions can be easily correlated with certain blocks depending on the requirements for implementation or programming.

Moreover, when comparing FIG. 3B and FIG. 3A, it becomes apparent that the function of the weighting unit 364 for calculating the weighting factors for weighted information can be transferred to the matrix calculating unit 202. In other words, the matrix data is a set of weighting factors used by the complex module matrix coding 303, which is implemented in the information block 364, but which can also partially function as a decorrelator 356 (taking into account the matrix Q, which will be discussed later). Thus, the complex matrixing module 303 performs the operation of converting the subbands of these at least two downmix signals of the object (signals obtained by downmixing the object), where the matrix information includes weighting factors for weighting these at least two downmix signals or a signal de-correlated before performing a downmix operation.

The following describes in detail preferred options for constructive solutions of the information unit 364 and decorrelator 356. In particular, some versions of the implementation of the functionality of decorrelator 356 and information unit 364 are presented using figa-4D. FIGS. 4E-4G show particular cases of arrangement of the elements shown in FIGS. 4A to 4D. Before a detailed consideration of figa-4D should explain the General principle of presentation of these drawings. The contour in each drawing has an upper branch line for the passage of the decorrelated signal, and a lower branch line for the passage of the original unprocessed signal. Further, the output signals of each branch, that is, the signal at the output of line 450 and the signal at the output of line 452, are reduced by a data unit 454 to generate an audio signal converted by audio rendering 350. In general, the circuit in Fig. 4A includes three matrix processors 401, 402 , 404. 401 - mixer of the original signal. These at least two downmix signals of the audio object 352 are weighted and / or mixed to obtain two source signals of the object corresponding to the signal path branch signals, which ends with the input to the adder 454. The source signal path may be provided with an additional the matrix processor-compensator gain 409 shown in fig.4D at the output of the mixer of the original signal 401.

In addition, the mixing unit 364 may optionally include an upmixing unit of the decorrelator 404 using the upmixing matrix P of the decorrelated signal.

Naturally, the separation of the matrixing units 404, 401 and 409 (FIG. 4D) and the mixing unit 454 is conditional, although a similar design solution is certainly possible. At the same time, the functions of these matrices can be realized with the help of one “large matrix”, which receives a decorrelated signal 358 and a downmix signal 352 at the input and provides two, three or more output channels 350 converted by audio rendering at the output. large matrix "signals along lines 450 and 452 may not pass, and these lines themselves as intermediate components may not be structurally implemented, while the functions of such a" large matrix "will be performed in the form of work operations of matrix blocks full coding 404, 401 or 409 and information block 454.

In addition, a pre-correlation mixing block of signal 402 can be arbitrarily inserted into decorrelator 356. FIG. 4B shows an embodiment not containing this block. In particular, this is applicable when two decorrelators are used to convert signals through two down-mix channels, when special down-mix is not required. Naturally, in each particular implementation, certain gain factors can be applied to both downmix channels, or two downmix channels can be mixed before entering the decorrelator. At the same time, the functions of the matrix Q can be performed by the matrix P. This means that the matrix P in Fig. 4B is different from the matrix P in Fig. 4A, despite the fact that the same result is achieved. In view of this, decorrelator 356 may not contain any matrix at all, and the full calculation of the matrix parameters, as well as their use, will be carried out inside the information block. However, in order to more clearly display the technical capabilities hidden behind mathematical calculations, in the future, the present invention will be described in the framework of the matrix algorithms shown in figa-4D.

FIG. 4A shows a constructive solution of the invention in the form of a complex matrix coding module 303. Input X, including parameters of at least two channels, is input into the mixer of the original signal 401, which performs the matrix operation in accordance with the matrix C of mixing the original signal and outputs the original stereo up-mix signal Y ′. The input data X is also input into the pre-correlation mixing block 402, which performs the matrix operation in accordance with the pre-correlation bias matrix Q and gives the output of the channel signal N _d , which is fed to the decorrelator 403. Then, the resulting decorrelated signal Z of the channel N _d is input into the upmixing block the decorrelator signal 404, which performs a matrix operation in accordance with the upmix matrix P of the decorrelator signal and outputs a decorrelated stereo signal. Finally, the decorrelated stereo signal is mixed by simply adding in the channel to the unprocessed up-mixing stereo signal Y ′ to form the output signal Y ′ of the complex matrix. All three mixing matrices C, Q, P are described by matrix data calculated and sent by the matrix calculator 202 to the stereo processor 201. One embodiment of the prior art system contains only the lower branch of the source signal line. Such a system works unsatisfactorily in the simple case when a stereo musical object passes through one channel of a downmix of an object, and a monophonic voice object passes through another channel of a downmix of an object. This is because the audio rendering of music to a stereo phonogram is entirely based on frequency selective panning, although it is known that the parametric principle of conversion to stereo format, including decorrelation, gives a much higher quality of perceived sound. In this particular case, the best result could be achieved by a completely different system, related to the prior art, in which decorrelation is applied, but which is based on downmixes of two separate monophonic objects, however, on the other hand, the achieved quality will remain the same as in the first example with a system that produces an unprocessed stereo signal for backward compatible downmixing, in which the music remains in true stereo format and the voice is mixed in equal weights with these two channels Amps downmix an object. An example is the target audio rendering for karaoke, reproducing only a stereo music object. Subsequent separate processing of each channel down-mixing does not allow optimal suppression of the speech object as in combined processing, when the received information about the stereo-audio object, such as cross-channel correlation, is taken into account. A key feature of the present invention is the achievement of the highest possible sound quality, not only in both of these simple situations, but also with much more complex combinations of down-mixing an object with audio rendering.

On figv, as mentioned above, in contrast to figa shows a layout in which the matrix Q pre-correlation mixing is not required or integrated into the matrix P up-mixing signal of the decorrelator.

FIG. 4C shows an arrangement in which the pre-correlator matrix Q is provided and implemented in a block with decorrelator 356, and in which the upmix matrix P of the decorrelator signal is not required or integrated into the matrix Q.

In addition to this, FIG. 4D shows an arrangement in which the matrices shown in FIG. 4A are present, but in which an additional gain compensation matrix G is provided, which is especially applicable in the third embodiment of the technical solution, which will be discussed in the context of FIG. 13, and a fourth embodiment of the design, which will be discussed in the context of FIG.

The decorrelator block 356 may include a single decorrelator or two decorrelators. FIG. 4E illustrates an arrangement in which a single decorrelator 403 is provided and in which the downmix signal is a two-channel downmix signal of an object, and the output signal is a two-channel audio output signal. In this case, the downmix matrix Q of the decorrelator signal has one row and two columns, and the upmix matrix of the decorrelator signal has one column and two rows. However, when the signal received by the downmix has more than two channels, the number of columns Q will be equal to the number of channels of the downmix signal, and when the synthesized audio-converted output signal has more than two channels, the number of rows of the upmix matrix P of the decorrelated signal will be equal to the number channels of the output converted by audio rendering.

FIG. 4F shows an electric circuit resembling embodiment of a source signal mixer 401, designated C ₀ , which, in a two-on-two design, has two rows in two columns. The cells of the matrix are displayed in the diagram as weights c _ij . In addition, as can be seen in FIG. 4F, weighted channel mixing is performed using adders. However, when the number of down-mix channels is different from the number of channels of the audio rendering of the output signal, the mixing matrix of the original signal C ₀ will not be a quadratic matrix, but will have a number of rows other than the number of columns.

In FIG. 4G, the summing step 454 in FIG. 4A is shown in detail. In particular, when there are two output channels, for example, the left stereo channel signal and the right stereo channel signal, two different summing links 454 are provided that perform the output signals of the upper branch related to the decorrelator signal and the lower branch related to the original signal, as shown in figg.

The cells of the gain compensation matrix G 409 are located only along its diagonal. With the two-on-two design shown in FIG. 4f for the source signal mixing matrix C ₀ , the gain for compensating for the gain of the left source signal should be at position ₁₁ , and the gain for compensating for the gain of the right source signal will be position C _{22 of the} matrix C ₀ in fig.4F. The values of c ₁₂ and C ₂₁ must be 0 for the gain matrix G, as shown under 409 in FIG. 4D.

Figure 5 shows the operation of the multi-channel decorrelator 403 of the prior art. A similar tool is used, for example, in the framework of the MPEG Surround standard. N _d signals - signal 1, signal 2, ..., signal N _{d are} separately fed to decorrelator 1, decorrelator 2, ..., decorrelator N _d . Each decorrelator usually consists of a filter designed to generate an output signal that is maximally uncorrelated with the input signal while maintaining the input signal power. At the same time, the filters of various decorrelators are selected so that the output signal of decorrelator 1, the output signal of decorrelator 2, ..., the output signal of decorrelator N _d also do not correlate as much as possible in pairs. Since decorrelators are typically characterized by high computational complexity compared to other components of an audio object decoder, the number N _d should preferably be as small as possible.

The present invention provides technical solutions with a number N _d equal to 1, 2 or more, but preferably less than the number of audio objects. More specifically, the preferred number of decorrelators in the implementation should correspond to the number of audio channel signals converted by the audio rendering of the output signal, or less than the number of audio channel signals converted by the audio rendering of the output signal 350.

The following is a brief mathematical description of the present invention. All the signals considered here are samples of subbands of modulated filter banks or window FFT analysis (fast Fourier transform) of discrete time signals. It is understood that these subbands must be converted back to the discrete time domain using the appropriate synthesis filter bank operations. A signal packet of L samples displays the signal in a time-frequency interval that is part of the perceptually motivated fragmentation of the time-frequency plane used to describe the characteristics of the signal. With this approach, audio objects can be represented by a matrix containing N lines of length L,

Figure 6 is a summary map of N audio objects. In Fig.6, each object has an individual identifier (ID) of the object, the corresponding sound file of the object, and necessarily the parameters of the audio object, which preferably should contain indicators of the energy of the audio object and its mutual correlation with other audio objects. Namely, the parameters of the audio object include an object covariance matrix E for each subband and for each time block.

An example of such a matrix E of the acoustic parameters of the object is shown in Fig.7. Diagonally located cells e _ij enclose information about the power or energy of the audio object i in the corresponding subband and in the corresponding time block. For this, a subband signal representing a certain audio object i is input into a power or energy calculator, which is designed, for example, to perform the autocorrelation function (acf) to obtain the value of e ₁₁ with or without normalization. Or, the energy can be calculated as the sum of the squares of the signal over a specific length (that is, as a vector product: ss *). The acf function can in some sense describe the spectral distribution of energy, but since the use of the time-frequency conversion for frequency selection is preferable in any case, the energy can be calculated without using acf separately for each subband. Thus, the diagonal elements of the matrix E of the acoustic parameters of the object indicate the measured level of power or energy of the audio object in a particular subband in a specific time interval.

However, the off-diagonal element e _ij denotes the corresponding degree of tightness of the correlation between the audio objects i, j in the corresponding subband and time block. As can be seen in Fig.7, in the matrix E, the cells with the actual value are located symmetrically with respect to the main diagonal. Typically, such a matrix is a Hermitian matrix. The correlation tightness index (correlation coefficient) e _ij can be calculated, for example, by cross-correlation of two subband signals of the corresponding audio objects, thereby determining the degree of cross-correlation tightness, which can be arbitrarily normalized. Other methods of measuring correlation are possible, which do not include the operation of determining cross-correlation, but involve other methods of calculating the degree of tightness of the correlation between the two signals. For practical reasons, all elements of the matrix E are normalized to values between 0 and 1, where 1 is an indicator of maximum power, or maximum correlation, 0 indicates the minimum power (zero power), and -1 indicates the minimum (not matching in phase) correlation.

The K × N downmix matrix D, where K> N, generates a K-channel downmix signal according to a matrix with an end-to-end number of K lines in matrix multiplication

Fig. 8 shows an example of a downmix matrix D with a cell d _ij . Element d _ij indicates whether or not, and to what extent, object j is included in the downmix signal of object i. For example, if d ₁₂ is zero, then object 2 is not included in the downmix signal of object 1. However, a value of d ₂₃ equal to 1 means that object 3 is fully included in the downmix signal of object 2.

The cells of the downmix matrix cells are between 0 and 1. In particular, a value of 0.5 means that an object is participating in the downmix signal with half of its energy. Thus, if, for example, the audio object 4 is evenly distributed over both channels of the downmix signal, the elements d ₂₄ and d ₁₄ will be 0.5. This down-mix method is energy-efficient and can be recommended for a number of cases. However, a down-mix technique that is not economical can also be used when the entire audio object is inserted into the left down-mix channel and into the right down-mix channel so that the energy of this audio object doubles relative to other audio objects inside this down-mix signal.

At the bottom of FIG. 8 is a schematic diagram of the encoder of the object 101 shown in FIG. In particular, two different components 101a and 101b are inserted into the encoder of the object 101. Component 101a is a downmixer that predominantly performs weighted linear mixing of audio objects 1, 2, ..., N, the second component 101b of the encoder of object 101 is a calculator of parameters of audio objects, such as matrix E, for each time block or frequency subband in order to produce correlation data and sound powers, which are parametric information and therefore can be transmitted with a low bit rate or stored in a small amount of memory.

A user-defined object rendering matrix A with a dimension of M × N forms an M-channel rendering of audio objects according to given parameters in the form of a matrix with an end-to-end number of M rows in matrix multiplication

During the subsequent differentiation, it will be assumed that M = 2, since the main attention is paid to stereo audio rendering (/ stereo audio transmission). Taking into account that the primary audio rendering is carried out on more than two channels, and, knowing the rules for down-mixing these several channels into two channels, a qualified specialist will easily calculate the appropriate 2 × N rendering matrix A for stereo audio rendering. Such a reduction is performed by the audio rendering data converging unit 204. In addition, for simplicity, we assume that K = 2, that is, that the downmix of the object is also a stereo signal. Moreover, the down-mix of a stereo object is the most important case of an application scenario.

Figure 9 is a detailed discussion of the matrix A of audio rendering parameters. Depending on the purpose, the matrix A of the audio rendering parameters may be presented by the user. The user is absolutely free to choose the final location of the audio object in a reproducible virtual environment. The strength of the audio object concept is that the downmix information and the audio object parameters do not affect the desired spatial distribution of the audio objects. Such positioning of audio objects is set by the user in the form of information of spatially coordinated audio rendering. Information of such a target audio rendering may be presented in the form of a matrix A of audio rendering parameters, a variant of which is shown in FIG. 9. In particular, the audio rendering matrix A has M rows and N columns, where M is equal to the number of channels of the output converted by audio rendering, and where N is equal to the number of audio objects. M is equal to two according to a preferred stereo audio rendering scenario, however, if M-channel audio rendering is performed, matrix A has M rows.

In particular, the matrix cell a _ij is included or not, and to what extent, object j is in the down-mix signal of object i. The lower part of Fig. 9 shows a simple example of a matrix of audio rendering parameters for a scenario in which six audio objects from A01 to A06 are involved, and only the first five audio objects must be precisely positioned in space, and the sixth audio object should not affect rendering at all.

According to the user's acoustic scenario, audio object A01 should sound on the left. Therefore, this object is placed in the left speaker in the (virtual) audience, as a result of which the first column of the audio rendering matrix A takes the form (10). The second audio object has indicators a ₂₂ equal to one, and a ₁₂ equal to zero, which means that the second audio object should sound on the right.

Audio object 3 should sound in the center, between the left and right speakers, so that 50% of the level or signal of this audio object passes through the left channel and 50% of the level or signal passes through the right channel, respectively, the third column of the matrix A of the audio rendering parameters has the form (0.5 space 0.5).

Similarly, using the matrix of audio rendering parameters, any position between the left and right speakers can be set. Since the matrix cell a _{24 is} larger than a ₁₄ , the audio object 4 is shifted to the right. Similarly, the fifth audio object A05 is shifted to the left, which is reflected by the elements of the matrix of audio rendering parameters a ₁₅ and a ₂₅ . Matrix A of audio rendering parameters additionally provides for the ability not to play some audio objects at all. This is shown by the example of the sixth column of the matrix A of audio rendering parameters containing zero cells.

In the course of the subsequent differentiation, it will be assumed that M = 2, since the focus is on stereo audio rendering. Taking into account that the primary audio rendering is carried out on more than two channels, and, knowing the rules for down-mixing these several channels into two channels, a qualified specialist will easily calculate the appropriate 2 × N rendering matrix A for stereo audio rendering. Such a reduction is performed by the audio rendering data converging unit 204. In addition, for simplicity, we assume that K = 2, that is, that the downmix of the object is also a stereo signal. Moreover, the down-mix of a stereo object is the most important case of an application scenario.

If we temporarily neglect audio coding losses during downmixing of an object, the main task of an audio object decoder is to bring the original audio objects closer to the desired spatial perception during the audio rendering according to the given parameters (target audio rendering) Y, based on the audio rendering matrix A, the downmix data X, the downmix matrix D mixing, and object parameters. An apparatus of the integrated matrix coding module 303 of the invention is shown in FIG. Given the number N _{d of} mutually orthogonal decorrelators in element 403, three mixing matrices can be distinguished:

- With a dimension of 2 × 2, which performs the mixing of the original signal;

- Q dimension N _d × 2, which performs pre-correlation mixing of the signal;

- P dimension 2 × N _d , which performs upmixing of the decorrelated signal.

Assuming decorrelators are energy-efficient, the decorrelated signal matrix Z has a covariance matrix R _z = ZZ ^* with a diagonal of N _d × N _d whose diagonal values are equal to the diagonal values of the covariance matrix

downmix of the object that went through the process of pre-correlation mixing. (Hereinafter, an asterisk denotes matrix operations with complex conjugate transposition. In addition, it is understood that the determinate covariance matrices of the UV ^* form, used universally for the convenience of calculations, can be replaced by mathematical expectations).

In addition, all decorrelated signals can be considered uncorrelated when downmixing objects. Thus, the covariance R ′ of the reduced output signal of the complex matrixing module 303 related to the invention.

смешанного исходного сигнала

и результирующей ковариации выхода декоррелятораcan be written as the sum of the covariance

mixed source signal

and resulting covariance of decorrelator output

The parameters of the object usually contain information regarding the power of the object and the selected inter-object correlations. Based on these parameters, an E model with N × N covariance of SS * objects can be constructed.

At the same time, the data required by the audio object decoder is described by a matrix triplet (D, E, A), and the method that is part of the present invention consists in using this data for general optimization of wave matching of the reduced output signal (5) and its covariance (6) with a signal of a given audio rendering (4). The objective of this bias matrix of the original signal is the correct final covariance R ′ = R, which can be estimated using

Comparing the error matrix described as

c (6) we obtain the construction requirement

Since the left-hand side of (10) is a positive semidefinite matrix for any matrix P of the decorrelator, it is necessary that the error matrix (9) also be a positive semidefinite matrix. For a detailed explanation of the following formulas, it is necessary to parameterize the covariance of the mixed source signal and the given audio rendering as follows:

For error matrix

the necessary requirement of positive semidefiniteness can be expressed in the form of three conditions:

Next will be considered figure 10. Figure 10 shows the sequence of preliminary calculations necessary for the four implementation options, figure 11-14. One of these preliminary calculations is the calculation of the covariance matrix R of the audio rendering signal according to predetermined parameters, which is indicated by element 1000 in FIG. 10. Block 1000 corresponds to equation (8).

As shown in block 1002, the mixing matrix of the original signal can be calculated using equation (15). In particular, the result of calculating the mixing matrix of the initial signal C ₀ is the best signal matching of a given audio rendering achieved using down-mix signals, and it is assumed that the decorrelated signal is not used at all. Thus, the mixing matrix of the original signal provides the best matching of the output signal shape of the mixing matrix with the audio rendering signal according to the specified parameters without any additional decorrelation of the signal. This property of the mixing matrix of the original signal is especially important to keep the decorrelated signal in the output channel as low as possible. In most cases, a decorrelated signal is a signal that has been significantly altered by a decorrelator. Because of this, such a signal, as a rule, contains artifacts in the form of colorization, time blurring and poor transient response. Therefore, the advantage of this design solution is to provide better quality of the output audio signal due to the low level of decorrelation of the signal. Due to the matching of the waveform, that is, weighing and mixing two or more channels into a down-mix signal with the formation of the original sound signal, as close as possible to the specified characteristics of the audio rendering, minimal signal decorrelation is required.

The function of the information block 364 is to calculate the weighting coefficients so that the result 452 of mixing the down-mix signal of the first object and the down-mix signal of the second object in the waveform is consistent with the result of the audio rendering according to the specified parameters, and this coordination should, as much as possible, prevent the situation, when, during the audio rendering of the original audio objects using the specified audio rendering parameters 360, parametric information about ioobekte 362 would be lost in the absence of acoustic images of audio objects. It follows that an accurate reconstruction of the signal cannot be guaranteed even if there is a non-quantized matrix E. The mean-square error can be minimized. Therefore, one should strive for wave matching, due to which the reconstruction of power levels and mutual correlations is possible.

исходного сигнала. В частности, рекомендуется использовать уравнение, расположенное в правой части фиг.10, то есть C₀DED^*C^* ₀. Благодаря этой формуле при расчете матрицы ковариации

результата смешения исходного сигнала требуются только параметры, а отсчеты частотных поддиапазонов не требуются. При этом, как вариант, ковариационная матрица результата смешения исходного сигнала может быть рассчитана с использованием матрицы С₀ смешения исходного сигнала и микшированных с понижением сигналов, но первое вычисление, которое выполняется в области значений параметров может быть только невысокой сложности.Immediately after calculating the initial mixing signal matrix C ₀ , for example, the covariance matrix can be calculated by the above method

source signal. In particular, it is recommended to use the equation located on the right side of FIG. 10, i.e., C ₀ DED ^* C ^* ₀ . Thanks to this formula, when calculating the covariance matrix

the result of mixing the original signal requires only parameters, and samples of the frequency subbands are not required. In this case, as an option, the covariance matrix of the result of mixing the original signal can be calculated using the matrix C _{0 of} mixing the original signal and the signals mixed with decreasing, but the first calculation, which is performed in the range of parameter values, can only be of low complexity.

исходного сигнала.After the calculation steps 1000, 1002, 1004, the mixing matrix C _{0 of the} original signal, the covariance matrix R of the given audio rendering signal and the covariance matrix can be calculated

source signal.

Four variants of the constructive solution are considered below depending on the application of the matrices Q and P. Additionally, the case of Fig. 4d is described (for example, for the third or fourth embodiment), where the values of the gain compensation matrix G are also determined. For a qualified specialist, the possibility of varying design solutions that help to find the values of these matrices is obvious, due to the existing degree of freedom of choice in calculating matrix weighting coefficients.

In a first embodiment of the present invention, matrix calculator 202 operates on the following algorithm.

The upmix matrix of the original signal is calculated primarily to find a solution using the least squares method to match the waveform

In this case, it is valid. . In addition, the equation holds here:

The solution to this problem is found using

является ортогональной по отношению к аппроксимации. Поэтому перекрестные члены при дальнейшем вычислении обращаются в нуль,which has a known additional solution by the least squares method, and on the basis of (13) it can also be easily confirmed that the error

is orthogonal with respect to the approximation. Therefore, the cross terms in the further calculation vanish,

Therefore

which is obviously non-negative definite; therefore, (10) can be solved. The symbolic decision is

легко определяется с помощью поэлементной операции на диагонали, и матрица T решает матричное уравнение TT^*=ΔR. Для этого уравнения существует большой выбор решений. Особенность метода данного изобретения состоит в том, что начинать следует с разложения по сингулярным числам матрицы ΔR. Для этой симметричной матрицы он сводится к обычному разложению собственного вектора,Here is the second factor

it is easily determined using the elementwise operation on the diagonal, and the matrix T solves the matrix equation TT ^* = ΔR. There is a large selection of solutions for this equation. A feature of the method of this invention is that you should start with the expansion in the singular numbers of the matrix ΔR. For this symmetric matrix, it reduces to the usual expansion of the eigenvector,

where the matrix U of the eigenvector is unitary and its columns contain eigenvectors corresponding to characteristic values sorted in descending order. The first solution option of the present invention with one decorrelator (N _d = 1) is performed by setting λ _min = 0 in (19) and substituting the corresponding natural approximation

in (18). A complete solution with the number of decorrelators N _d = 2 is made by introducing the missing minimum significant complement from the smallest eigenvalue and adding the second column to (20), corresponding to the product of the first factor (19) and the square root of each element of the diagonal eigenvalue matrix. In detail it will look as follows:

The result of calculating the matrix P for the first embodiment will be summarized in the context of Fig. 11. In step 1101, the covariance matrix DR of the mismatch signal is calculated, or, when considering FIG. 4A, the matrix of the correlated signal in the upper branch, using the results of step 1000 and step 1004 in FIG. 10. Then the eigenvalues of this matrix are expanded, which was discussed in connection with equation (19). Next, in accordance with one of the many applicable strategies that will be discussed later, the choice goes to the Q matrix. Based on the selected Q matrix, the covariance matrix R _{z of the} matched decorrelated signal is calculated using the equation written to the right of frame 1103 in FIG. 11, then there is matrix multiplication QDED ^* Q ^* . Then, based on R _z obtained in step 1103, the upmix matrix P of the decorrelator signal is calculated. It is clear that the actual implementation of this matrix up-mixing is optional, since the output of block P 404 in Fig. 4A has more signals than at the input. This operation can be applied in the case of a single correlator, while for two decorrelators the upmix matrix P of the decorrelator signal has two channels at the input and two channels at the output and can be implemented as a matrix of the upmixer of the original signal shown in Fig. 4F .

Thus, the main feature of the first embodiment is that C ₀ and P are calculated. There is an opinion that two decorrelators should be used to ensure a clearly correlated circuit at the output. At the same time, the possibility of using only one decorrelator creates its own advantages. Such a solution is represented by equation (20). In particular, in the implementation can be applied decorrelator with a lower characteristic value.

In a second embodiment of the present invention, matrix calculator 202 operates according to the following algorithm. In its shape, the mixing matrix of the decorrelator signal is limited

With this restriction, the covariance matrix of a single decorrelated signal is a scalar R _z = r _Z , and the covariance of the reduced output signal (6) takes the form

where α = c ² r _z . Full agreement with the given covariance R ′ = R is not technically feasible, but the perceptually determined need for a normalized correlation between the output channels in a number of cases makes it possible to bring it closer to the desired one. Here, the defined correlation is described.

and the correlation achieved in the reduced output signal (23) is represented as

Equation (24) and (25) leads to the quadratic equation in α,

. Если оба решения уравнения (26) окажутся положительными, применено должно быть то, норма с которого меньше. В том случае, когда подобное решение не существует, активность декоррелятора устанавливается на ноль выбором с=0, так как сложные решения с приводят к заметным фазовым искажениям декоррелированных сигналов.

может быть рассчитано двумя способами - или напрямую из сигнала

, или путем объединения ковариационной матрицы объектов с данными понижающего микширования и аудиорендеринга как

. Здесь первый метод даст в результате комплексное

и поэтому в правой части (26) квадрат должен быть взят из вещественной части или величины

соответственно. Тем не менее, даже комплекснозначное

может быть использовано. Такое комплексное значение указывает на корреляцию со специфической фазой, что также может оказаться полезным в специфических приложениях.For those cases where (26) has a positive solution α = α ₀ > 0, the second embodiment of the present invention proposes to use the constant in definition (22)

. If both solutions of equation (26) turn out to be positive, then the one with which the norm is less should be applied. In the case when such a solution does not exist, the decorrelator activity is set to zero by choosing c = 0, since complex solutions with lead to noticeable phase distortions of the decorrelated signals.

can be calculated in two ways - or directly from the signal

, or by combining the covariance matrix of objects with downmix and audio rendering data as

. Here, the first method will result in a comprehensive

and therefore, in the right-hand side of (26), the square should be taken from the real part or the quantity

respectively. However, even complex

can be used. Such a complex value indicates a correlation with a specific phase, which may also be useful in specific applications.

.As can be seen from (25), the peculiarity of this design is that it can only reduce the correlation tightness, in contrast to the correlation of the original signal. In this way,

.

As a result, we obtain that the second embodiment is traced in FIG. It begins by calculating the covariance matrix DR in step 1101, which is identical to step 1101 in FIG. 11. Then equation (22) is applied. Namely, the appearance of the matrix P is predefined, and only the weighting factor c remains to be found, which is the same for both elements of P. In particular, one column of the matrix P shows that a single decorrelator is used in the second version of the implementation. Moreover, the signs of the elements p show that the decorrelated signal is input into one channel — into the left channel of the original signal and is subtracted from the right channel of the original signal. Therefore, maximum decorrelation is achieved by adding a decorrelated signal to one channel and subtracting the decorrelated signal from another channel. To find the value of c, steps 1203, 1206, 1103, and 1208 are taken. In particular, as can be seen from equation (24), the specified correlation series is calculated at step 1203. This value is an indicator of inter-channel cross-correlation between two audio channel signals during stereo audio rendering. Based on the result of step 1203, based on equation (26), the weight coefficient a is determined, as shown in step 1206. In addition, values for the cells of the matrix Q are selected and the covariance matrix is calculated, which in this case is only a scalar quantity R _z , as shown in step 1103 and as can be seen from the equation to the right of frame 1103 in FIG. 12. Finally, find the factor c, as shown in step 1208. Equation (26) is a quadratic equation that can give two positive solutions for α. As mentioned above, in this case, the solution with the lowest norm c is used. However, when such a positive solution is not achieved, c is set to 0.

Thus, in the second embodiment, P is calculated as a special case of the distribution of one decorrelator into two channels, as shown by the matrix P in frame 1201. In some cases, there is no solution and the decorrelator is simply turned off. The advantage of this design solution is that it does not introduce a synthesized signal with a positive correlation. This is a favorable factor, since such a signal can be perceived as a localized phantom source, which is an artifact that reduces the acoustic quality of the output signal converted by the audio rendering. Considering the fact that the power level is not taken into account in this version, the output signal may be out of alignment, which means that the output signal is either higher or lower than the down-mix signal. In such a case, in the selected technical embodiment, additional gain compensation is used to improve the sound quality.

In a third embodiment of the present invention, matrix calculator 202 operates according to the following algorithm. The starting point is compensation for the amplification of the original signal.

является результатом аппроксимации наименьших квадратов

по матрице смешивания, данной в (15). Кроме того, С=GC₀, где G является диагональной матрицей с ячейками g₁ and g₂. В этом случаеwhere, for example, the uncompensated source signal

is the result of least squares approximation

according to the mixing matrix given in (15). In addition, C = GC ₀ , where G is a diagonal matrix with cells g ₁ and g ₂ . In this case

and error matrix

Here, a third embodiment of the invention is required to select compensation factors (g ₁ , g ₂ ) to minimize the weighted sum of power mismatches

идентичен микшированному с повышением исходному сигналу, объем декоррелированного сигнала, добавленного к конечному выходному сигналу, гораздо меньше объема, добавляемого к конечному выходному сигналу по первой версии реализации настоящего изобретения.taking into account the limitations identified in (13). Examples of the choice of weights for (30) are (w ₁ , w ₂ ) = (1, 1) or (w ₁ , w ₂ ) = (R, L). After that, the resulting error matrix ΔR is used to input the decorrelator signal P into the calculation of the mixing matrix of the signal according to the sequence of equations (18) - (21). The advantages of this embodiment include that when an error signal

identical to the up-mixed source signal, the amount of decorrelated signal added to the final output signal is much less than the volume added to the final output signal according to the first version of the implementation of the present invention.

Summing up the third embodiment, given in FIG. 13, it can be noted that the matrix shown in FIG. 4d is taken as the gain matrix G. According to the meaning of equations (29) and (30), the gains g ₁ and g ₂ are calculated using the selected w1, w2, as indicated by the text under equation (30) and taking into account the restrictions on the error matrix obvious from equation (13). By performing steps 1301, 1302, the mismatch covariance matrix ΔR can be calculated using g ₁ and g ₂ according to step 1303. It is understood that this mismatch covariance matrix calculated in step 1303 is different from the covariance matrix ΔR calculated in step 1101 in FIG. .11 and 12. Then, steps 1102, 1103, 1104 are repeated according to an earlier discussion in connection with the first embodiment of FIG. 11.

The third embodiment of the constructive solution has the advantage that the original signal is not only consistent in waveform, but also offset by gain. This helps to further reduce the level of decorrelated signal, which reduces the number of artifacts that occur with it. Thus, in the third embodiment, an attempt is made to the most optimal combination of gain compensation and dosed decorrelation. Again, the goal is to integrate the covariance structure, including channel power levels, and minimize the level of the synthesized signal, for example, using the minimization equation (30).

Let us further consider the fourth embodiment of the invention. At step 1401, a single decorrelator is used. Here we are talking about the least complicated technical solution, since in practice the use of a single decorrelator is most appropriate. The next step 1101, including the calculation of the covariance matrix ΔR, was already displayed in the form of a block diagram and was discussed in the context of step 1101 for the first implementation option. However, the data of the covariance matrix ΔR can also be calculated in accordance with step 1303 of FIG. 13, where gain compensation is combined with waveform matching. Next, check the sign of Δp, which is an off-diagonal element of the covariance matrix ΔR. If step 1402 determines that this sign is negative, then steps 1102, 1103, 1104 of the first embodiment will continue, where step 1103 is not particularly complicated due to the fact that R _z is a scalar quantity, since there is only one decorrelator.

However, if it is determined that the Δp sign is positive, the input of the decorrelated signal will be immediately stopped by setting the matrix element P to zero. An alternative action may be to reduce the level of the introduced decorrelated signal to a value above zero, but below the value that would have been with a negative sign. However, it is recommended that the elements of the matrix P are not just given smaller values, but that they are set to zero, as shown in box 1404 in FIG. Moreover, in accordance with fig.4d for gain compensation, the gains g ₁ , g _{2 are} determined, as shown in box 1406. Namely, the gains are calculated so that the elements of the main diagonal of the matrix on the right side of equation (29) are equal to zero. This means that the covariance matrix of the mismatched signal has zero elements on its main diagonal. Thus, gain compensation is achieved if the decorrelator signal is reduced or completely turned off to avoid phantom artifacts of the source that may occur when a decorrelated signal is input that has specific correlation properties.

So, the fourth embodiment combines the capabilities of the first embodiment and the use of a single decorrelator, while providing for checking the quality of the decorrelated signal so that the decorrelated signal can be reduced or interrupted if the quality indicator, for example, such as Δp in the covariance matrix ΔR The error signal (added signal) becomes positive. The choice of the predecorrelator matrix Q should be based on the conditions of sensory perception, since the second-order theory presented above is insensitive to the use of this specific matrix. This also implies that the grounds on which Q is chosen are in no way connected with the considerations for which the design option is chosen.

The first basis prescribed by the present invention is to use for input into all decorrelators a monophonic down-mixed (monodaunmix) source stereo signal. In the form of matrix elements, this can be represented as follows:

- ячейки матрицы Q, a

- ячейки матрицы С₀.Where

are the cells of the matrix Q, a

- cells of the matrix With ₀ .

The second reason prescribed by the present invention is that the predecorrelator matrix Q should be derived from only one downmix matrix D. Taking the derivative implies here that all objects have specific power and are uncorrelated. This assumption also extends to the formation of the upmix matrix, starting from the source objects to predicting errors for each of them. Then, the square of the pre-correlation weights is selected in proportion to the total energy of the prediction errors of objects for all channels of the downmix. Ultimately, the same weights are used for all decorrelators. In more detail, these weights are obtained by first forming the N × N matrix,

and then, the matrix W _{0 of the} expected energy of the prediction errors of the objects is derived by reducing all off-diagonal values (32) to zero. Denoting the diagonal values of DW ₀ D ^* by t ₁ , t ₂ , which express the total fractions of the energy of the errors of the objects passing through each channel of the downmix, we obtain a finite set of elements of the matrix of the precorrelator using

Depending on the specific design and purpose, any kind of decorrelator, such as reverb and others, can be used. However, a preferred embodiment involves the use of energy-saving decorrelators. This means that the output power of the decorrelator should be equal to the input power of the decorrelator. However, deviations that occur when using an uneconomical decorrelator can also be compensated, for example, taken into account when calculating the matrix P.

As stated previously, in preferred embodiments, the introduction of a positive-correlated synthesized signal should be avoided since such a signal can be perceived as a localized synthesized phantom source. In the second embodiment, this should definitely be avoided due to the specific structure of the matrix P, as shown in box 1201. Nevertheless, this problem was clearly circumvented in the fourth embodiment of the constructive solution using the verification procedure at step 1402. Other methods of determining quality are also available to qualified specialists. decorrelated signal, and in particular correlation characteristics, in order to avoid the appearance of such artifacts of a phantom source, while such methods can be applied in the form of pre rotation of the decorrelated signal by technical means, or in the form of reducing the power of the decorrelated signal and amplifying the original signal to obtain a compensated output signal.

Although all matrices E, D, A are described as complex matrices, they can also be real. However, the value of this invention lies in the fact that it considers complex matrices E, D, A, actually containing complex coefficients with an imaginary part other than zero.

In addition, matrix D and matrix A will often have a much lower spectral and temporal resolution than matrix E, which has the highest time and frequency resolution among all matrices. In particular, the matrix of audio rendering parameters and the downmix matrix will not depend on the frequency, but may depend on the time. This may affect the downmix matrices when performing certain specific optimized downmix operations. With regard to the matrix of audio rendering parameters, this may occur in connection with moving audio objects, which from time to time, possibly, will change their position between the left and right channels. The structural solutions described below are given as an illustration of the basic principles of the present invention. It is understood that for specialists in this field, the possibility of making changes and improvements to the layout and elements of the described construction is obvious. Therefore, the descriptions and explanations provided for the embodiments of the invention are limited only by the scope of patent requirements, and not by specific details. Depending on the specific requirements for the implementation of the methods related to the invention, these methods can be implemented both in hardware and in software. The invention can be implemented using a digital data storage device, in particular a disk, DVD-ROM or CD-ROM containing electronically readable control signals, compatible with programmable computer systems with the aim of implementing methods related to the invention. Thus, in general, the present invention is a computer program product with program code stored on a machine-readable medium, by which the inventive methods are practically executed provided that the computer program product is executed on a computer. In other words, the invented methods are therefore a computer program having the program code necessary to implement at least one of the invented methods, provided that the computer program is executed on a computer.

Claims (28)

1. A device for synthesizing an output signal (350), providing a signal of a first sound channel and a signal of a second sound channel, including: a decorrelator (356) for generating a decorrelated signal (358), providing a decorrelated single channel signal or a decorrelated signal of the first channel and a decorrelated signal a second channel based on the downmix signal, wherein the downmix signal includes a downmix signal of the first audio object and a downmix signal mixing the second audio object and the result of downmixing a plurality of audio objects signals, implemented in accordance with the information about the downmix (354); and a converting unit (364) for performing weighted mixing of the downmix signal (352) and a de-correlated signal (358) using weighting factors (P, Q, C ₀ , G), wherein the mixing unit (364) is used to calculate weighting factors (P , Q, C _0, G) for the weighted information on the basis of the downmix information (354) based on predetermined characteristics rendering (360), determining a virtual position of the audio objects in the virtual sound space, and on the basis of parametric data audioobe max (362) describing the audio objects. 2. The device according to claim 1, in which the information unit (364) is designed to calculate weighting factors for weighted information so that the result 452 of mixing the down-mix signal of the first audio object and the down-mix signal of the second audio object in the form of the signal is consistent with the result of the audio rendering for the given parameters. 3. The device according to claim 1, in which the mixing unit (364) is designed to calculate the mixing matrix C ₀ for mixing the down-mix signal of the first audio object and the down-mix signal of the second audio object based on the equation:
C ₀ = AED * (DED *) ^-1 ,
where C ₀ is the mixing matrix, where A is the matrix of audio rendering parameters defining the characteristics of audio rendering (360), where D is the down mixing matrix defining the parameters of down mixing (354), where * denotes the complex conjugate transpose operation, and where E is the covariance matrix audio objects specifying parameters of audio objects (362). 4. The device according to claim 1, in which the information unit (364) is designed to calculate weighting factors based on the equation:
R = AEA *,
where R is the covariance matrix of the output signal converted by audio rendering (350), obtained by applying the specified characteristics of audio rendering to audio objects, where A is the matrix of audio rendering parameters that defines the characteristics of sound transmission (360), and where E is the covariance matrix of the audio object that defines the parameters of the audio object (362). 5. The device according to claim 3, in which the information unit (364) is designed to calculate weighting coefficients based on the equation:
R ₀ = C ₀ DED * C ₀ *,
where R ₀ is the covariance matrix of the result of the mixing operation (401) of the down-mix signal. 6. The device according to claim 1, in which the information unit (364) is designed to calculate weighting factors for weighted information so that weighted information is feasible by calculating the matrix C _{0 of} mixing the source signals and applying (401) the matrix of mixing the source signals C ₂ to the downmix signal (352), by calculating the matrix P of the subsequent processing of the decorrelator signal and applying (404) the matrix P of the subsequent processing of the decorrelator signal to the decorrelated signal (358), and by reducing (454) the results of the opera tion (404, 401) to obtain the output converted by audio rendering (550). 7. The device according to claim 1, in which the decorrelator unit (356) performs an operation (402) for processing the downmix signal (352) input to the decorrelator (403). 8. The device according to claim 7, in which the pre-correlation operation includes the process of mixing the first channel down-mix of the audio object and the second channel down-mix of the audio object based on the down-mix information (354) containing instructions on the distribution of the audio object in the channels of the down-mix signal. 9. The device according to claim 7, in which the mixing unit (364) performs the operation of mixing the original signal (401) using the first and second down-mix signals of an audio object, comprising a pre-correlation device (402) that performs functions similar to the operations of mixing the original signal (401) ) 10. The device according to claim 9, in which the mixing unit (364) uses the matrix C _{0 of} mixing the original signal, and the pre-correlator (402) performs its functions using the matrix Q of pre-correlation identical to the matrix C _{0 of} mixing the original signal. 11. The device according to claim 6, in which the matrix P of the subsequent processing of the decorrelator signal is based on (1102) decomposing the eigenvalue of the covariance matrix of the decorrelated signal added to the result of mixing the original signal (452). 12. The device according to claim 11, in which the information unit (364) calculates weighting coefficients based on the multiplication (1104) of the matrix (T) formed on the basis of the eigenvalues obtained by decomposition of the eigenvalue (1102) and the covariance matrix of the decorrelator signal (358) . 13. The device according to claim 11, in which the information unit (364) is designed to calculate the weight coefficients so that a single decorrelator (403) is used, while the matrix P of the subsequent processing of the decorrelator signal will contain one column and the number of rows equal to the number of channels in the output converted by audio rendering, or so that two decorrelators (403) are used, and the matrix P of the subsequent processing of the decorrelator signal contains two columns and the number of rows equal to the number of channels of the converted audio renedringom output signal. 14. The device according to claim 11, in which the information unit provides for the calculation of weighting coefficients based on the covariance matrix of the decorrelated signal, which is calculated by the equation:
R _Z = QDED * Q *,
where R _Z is the decorrelation signal covariance matrix (358), Q is the pre-correlation signal mixing matrix, D is the down-mixing matrix that sets the down-mixing parameters (354), E is the covariance matrix of the audio object containing parametric information about the audio object (362). 15. The device according to claim 6, in which the information unit (364) provides for calculating the weighting factors for the weighted information so that the matrix P of the subsequent processing of the decorrelator signal is calculated so that the decorrelated signal is summed with the two resulting channels (452) of the original mixing operation signal with opposite signs (1201). 16. The device according to clause 15, in which the information unit (364) provides for calculating the weighting factors so that the decorrelated signal (358) is weighted using the weighting factor (s) determined by the control signal of the correlation between the two channels of the converted audio rendering of the output signal, this correlation pilot signal is similar to the correlation value established by the virtual operation of a given audio rendering based on the matrix of audio rendering parameters (A) (1203). 17. The device according to clause 16 is configured to solve the quadratic equation (26) for determining the weight coefficient (s), and within which, in the absence of a satisfactory solution to this quadratic equation, the introduction of the decorrelated signal is limited or terminated (1208). 18. The device according to claim 6, in which the information unit (364) provides for calculating weighting factors to obtain weighted information by compensating for gain (409) by weighing the result of mixing the original signal so that the energy error within the result of mixing the original signal when compared with the signal the downmix was minimized (1302). 19. The device according to claim 1, in which the information unit (364) estimates the likelihood of artifacts (1402) when the decorrelated signal is introduced, and containing the information unit (364), which provides for the termination or reduction of the introduction of the decorrelated signal (1404) when recognizing the risk of occurrence artifact, and reduction (1406) of power distortion arising from the reduction or termination of the introduction (1404) of the decorrelated signal. 20. The device according to claim 19, in which the information unit (364) provides for the calculation of weighting coefficients, aimed at optimizing the result of the mixing operation of the original signal (401). 21. The device according to claim 19, in which the information unit (364) provides for the calculation of the data of the covariance matrix R of the mismatch (1104), which represents the correlation structure of the mismatch between the original up-mix signal and the output signal determined by the virtual audio rendering circuit using the specified audio rendering parameters ( 360), and containing the information block (364), which provides for the determination of the sign (1402) of the off-diagonal data element of the covariance mismatch matrix R and termination (1104) and whether input reduction with a positive sign. 22. The device according to claim 1, further comprising: a time-frequency converter (302) for converting the downmix signal into a spectral representation including a plurality of subband down-mix signals: the decorrelation operation (403) and the operation are performed for each subband signal information (364) to generate the entire set of subband signals of the output audio rendering array, and a frequency-time converter (304) for converting a plurality of subband signals audio output to the time domain. 23. The device according to claim 1, also including a controller for generating blocks of sample values of the signal obtained by downmixing, and for controlling decorrelator (356) and mixing unit (364) to process individual blocks of sample values. 24. The device according to p. 22, which generates information about the audio object for each block and for each subband signal, and the specified audio rendering parameters and characteristics of the down-mixing signal of the audio object remain constant in frequency for the time block. 25. The device according to claim 1, in which the mixing unit (364) includes a complex matrixing module (303) for linearly converting the down-mix signal of the first audio object and the down-mix signal of the second audio object to the original signal (452), and in this the information unit (364) also provides a linear reduction of the decorrelated signal (358) into a signal that, after summing in the channel with the original signal, forms the stereo output signal of the integrated matrix coding module (303), and Moreover, the mixing unit (364) includes a matrix calculator (202) for calculating linear weighting coefficients used by the complex matrixing module (303) based on parametric information about the audio object (362), downmix information (354), and preset audio rendering parameters (360). 26. The device according to claim 1, in which the information unit (364) is designed to calculate weighting factors so that the energy fraction of the decorrelated signal (358) in the output signal converted by audio rendering is minimal, and so that the energy fraction of the original signal (452) obtained linear mixing of the down-mix signal of the first audio object and the down-mix signal of the second audio object was maximum. 27. A method for synthesizing an output signal (350) including a signal of a first sound channel and a signal of a second sound channel, comprising generating (356) a decorrelated signal (358) including a decorrelated single-channel signal or decorrelated signal of the first channel and decorrelated signal of the second channel obtained from the down-mix signal, wherein the down-mix signal contains the down-mix signal of the first audio object and the down-mix signal of the second au a dioobject, wherein the downmix signal contains a downmix - the result of downmixing a plurality of audio object signals according to the downmix information (354); and weighted mixing (364) of the downmix signal (352) and decorrelated signal (358) using weights (P, Q, C ₀ , G) based on the calculation of weights (P, Q, C ₀ , G) for weighted information based on the downmix information (354), based on the specified audio rendering characteristics (360) that determine the virtual position of the audio objects in the virtual sound space, and based on the parametric data on the audio objects (362) that describe these audio objects. 28. A computer-readable medium containing a computer program product stored on it with a program code for executing the method of claim 27, provided that the computer program product is executed on the computer.

Images