RU2792114C2 - Integration of high-frequency sound reconstruction techniques - Google Patents

Abstract

FIELD: audio technology.

SUBSTANCE: invention relates to means for processing audio signals. An encoded audio bitstream is received, which contains audio data representing the part of the audio signal related to the lower range, and metadata of the reconstruction of high frequencies. Decoding of audio data is performed to generate a decoded audio signal of the lower range. High-frequency reconstruction metadata is extracted from the encoded audio bitstream, these high-frequency reconstruction metadata contain operating parameters for the high-frequency reconstruction process. The operating parameters include an insertion mode parameter located in a backward compatible extension container of the encoded audio bitstream. The first value of the insertion mode parameter indicates spectral transfer, and the second value of the insertion mode parameter indicates harmonic transposition using frequency range stretching by a phase vocoder. Filtering of the decoded audio signal of the lower range is performed to generate a filtered audio signal of the lower range.

EFFECT: increase in the efficiency of reconstruction of audio signal high frequencies.

14 cl, 7 dwg

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority in European patent application EP18169156.9, filed April 25, 2018, which is incorporated herein by reference.

FIELD OF TECHNOLOGY

Embodiments of the present invention relate to the processing of audio signals, and in particular to the encoding, decoding or transcoding of audio bitstreams with control data specifying whether either a basic form of high frequency reconstruction ("HFR") or an improved form of HFR is performed on the audio data.

BACKGROUND OF THE INVENTION

A normal audio bitstream contains both audio data (eg, encoded audio data) indicative of one or more audio content channels and metadata indicative of at least one characteristic of the audio data or audio content. One well-known format for generating an encoded audio bitstream is the MPEG-4 Advanced Audio Coding (AAC) format described in the MPEG ISO/IEC 14496-3:2009 standard. In the MPEG-4 standard, AAC stands for "Advanced Audio Coding" and HE-AAC stands for "High Efficiency Advanced Audio Coding".

The MPEG-4 AAC standard defines several audio profiles that define which encoding objects and tools are present in a compatible encoder or decoder. Three of these audio profiles are (1) the AAC profile, (2) the HE-AAC profile, and (3) the HE-AAC v2 profile. The AAC profile contains an AAC object type of low complexity (or "AAC-LC"). The AAC-LC object is a low-complexity analog of the MPEG-2 AAC profile with some enhancements, and contains neither a Spectral Band Copy ("SBR") object type nor a Parametric Stereo ("PS") object type. The HE-AAC profile is a superset of the AAC profile and additionally contains the SBR object type. The HE-AAC v2 profile is a superset of the HE-AAC profile and additionally contains the PS object type.

The SBR object type contains a spectral band copy tool, which is an important high frequency reconstruction ("HFR") coding tool that greatly improves the compression efficiency of perceived audio codecs. SBR reconstructs the high frequency components of the audio signal at the receiver side (eg, in a decoder). Thus, the encoder is only required to encode and transmit the low frequency components, which enables much higher audio quality at low bit rates. SBR is based on copying harmonic sequences, previously truncated to reduce the bit rate, from the available bandwidth limited signal and control data received from the encoder. The relationship between tonal and noise-like components is maintained by adaptive inverse filtering and optional addition of noise and sinusoids. In the MPEG-4 AAC standard, the SBR tool performs spectral insertion (also called "line transfer" or "spectral transfer"), in which a number of successive quadrature mirror filter (QMF) subbands are copied (or "pasted") from the transmitted low-band portion of the audio signal into the high-band part of the audio signal that is generated in the decoder.

Spectral insertion, or linear transfer, may not be ideal for some types of sound, such as music content with relatively low crossover frequencies. Therefore, techniques are needed to improve the copying of the spectral band.

Brief description of embodiments of the invention

The first class of embodiments relates to the disclosure of a method for decoding an encoded audio bitstream. The method includes receiving an encoded audio bitstream and decoding the audio data to generate a low band decoded audio signal. The method further includes extracting high frequency reconstruction metadata and filtering the low band decoded audio signal with an analysis filter bank to generate a low band filtered audio signal. The method further includes extracting a flag indicating whether to perform either spectral transfer or harmonic transposition on the audio data, and restoring the high-band portion of the audio signal using the low-band filtered audio signal and high-frequency reconstruction metadata according to the flag. Finally, the method includes combining the filtered low band audio signal and the reconstructed high band portion to form a wideband audio signal.

The second class of embodiments relates to an audio decoder for decoding an encoded audio bitstream. The decoder comprises an input interface for receiving an encoded audio bitstream, where the encoded audio bitstream contains audio data representing a low band portion of an audio signal, and a main decoder for decoding audio data to generate a decoded low band audio signal. The decoder also includes a demultiplexer for extracting high-frequency reconstruction metadata from the encoded audio bitstream, where the high-frequency reconstruction metadata contains operating parameters for a high-frequency reconstruction process that linearly transfers a sequential number of subbands from the low-band portion of the audio signal to the high-band portion of the audio signal. a portion of the audio signal, and an analysis filter bank for filtering the decoded low band audio signal to generate a filtered low band audio signal. The decoder further comprises a demultiplexer for extracting from the encoded audio bitstream a flag indicating whether it is necessary to perform either linear transfer or harmonic transposition on the audio data, and a high frequency restorer for restoring the high-band portion of the audio signal using the low-band filtered audio signal. and high frequency reconstruction metadata according to the flag. Finally, the decoder includes a synthesis filter bank for combining the filtered low band audio signal and the reconstructed high frequency high band portion to form a wideband audio signal.

Other classes of embodiments relate to encoding and transcoding audio bitstreams containing metadata identifying the need to perform enhanced spectral band copying (eSBR) processing.

Brief description of graphic materials

In FIG. 1 shows a block diagram of one embodiment of a system that may be adapted to carry out one embodiment of the method of the invention.

In FIG. 2 shows a block diagram of an encoder, which is one embodiment of an audio processing unit according to the invention.

In FIG. 3 shows a block diagram of a system that includes a decoder, which is one embodiment of an audio processing unit according to the invention, and optionally also a post-processor connected to it.

In FIG. 4 shows a block diagram of a decoder, which is one embodiment of an audio processing unit according to the invention.

In FIG. 5 shows a block diagram of a decoder, which is another embodiment of the audio data processing unit according to the invention.

In FIG. 6 shows a block diagram of another embodiment of an audio processing unit according to the invention.

In FIG. 7 shows a block diagram of an MPEG-4 AAC bitstream data block, including the segments into which it is divided.

Conventions and terminology

Throughout this document, including the claims, the expression performing an operation "on" a signal or data (e.g., filtering, scaling, transforming, or applying a gain factor to signals or data) is used broadly to mean performing an operation directly on the signal or data, or on processed version of the signal or data (for example, on the version of the signal that was subjected to pre-filtering or pre-processing before performing the specified operation on it).

Throughout this document, including the claims, the terms "sound processing unit" or "sound processor" are used in a broad sense to refer to a system, device, or apparatus adapted to process audio data. Examples of audio processing units include, but are not limited to, encoders, transcoders, decoders, codecs, pre-processing systems, post-processing systems, and bitstream processing systems (sometimes referred to as "bitstream processing tools"). Virtually all consumer electronics, such as mobile phones, televisions, laptops, and tablet computers, contain an audio processing unit, or sound processor.

Throughout this document, including the claims, the term "connects" or "connected" is used in a broad sense to mean either a direct or indirect connection. Thus, if a first device is connected to a second device, the connection may be through a direct connection or through an indirect connection through other devices or connections. Moreover, components embedded in or combined with other components are also related to each other.

Detailed description of embodiments of the invention

The MPEG-4 AAC standard stipulates that an encoded MPEG-4 AAC bitstream contains metadata that specifies each type of High Frequency Reconstruction ("HFR") processing to be applied (if any) to the decoder to decode the audio content of the bitstream. , and/or which control this HFR processing, and/or indicate at least one characteristic or parameter of at least one HFR tool to be applied to decode the audio content of the bitstream. In this document, the term "SBR metadata" is used to refer to metadata of the type described or referred to in the MPEG-4 AAC standard for use in spectral band copying ("SBR"). Those skilled in the art will appreciate that SBR is a form of HFR.

SBR is preferably used as a two-rate system in which the base codec operates at half of the original sample rate, while SBR operates at the original sample rate. The SBR encoder operates in parallel with the underlying main codec, albeit at a higher sampling rate. Although SBR is primarily a post-processing in the decoder, important parameters are extracted in the encoder to ensure the most accurate reconstruction of the high frequencies in the decoder. The encoder estimates the spectrum envelope of the SBR band for a time and frequency span/resolution appropriate for the segment characteristics of the current input signal. The spectrum envelope is estimated using complex QMF analysis and subsequent energy calculation. The time and frequency resolution of the spectrum envelopes can be chosen with a higher degree of freedom to provide the best suitable time-frequency resolution for a given input segment. When evaluating the envelope, it must be taken into account that a transient signal originally located mainly in the high frequency region (for example, a cymbal signal) will be only slightly present in the SBR generated high range before the envelope correction, since the high range in the decoder is based on the low range where this transient is much less pronounced compared to the high range. This aspect introduces different requirements for the time-frequency resolution of spectrum envelope data compared to conventional spectrum envelope estimation used in audio coding algorithms.

In addition to the spectrum envelope, several additional parameters are extracted that represent the spectral characteristics of the input signal for different time and frequency domains. Since the encoder naturally has access to the original signal, as well as information on how the SBR block in the decoder will create the upper band, given a specific set of control parameters, the system can handle situations in which the lower band is a strict harmonic series, and the upper the range to be recreated is mostly random signal components, as well as situations in which there are strong tonal components in the original high range that have no counterpart in the low range on which the high range is based. In addition, the SBR encoder works closely with the underlying core codec to judge which frequency range should be covered by SBR at a given time. The SBR data is efficiently encoded before transmission using entropy coding as well as control data channel dependencies, in the case of stereo signals.

Control parameter extraction algorithms usually need to be carefully tuned to the underlying codec at a given bit rate and a given sample rate. This is because a lower bit rate typically implies a wider range of SBRs than a higher bit rate, and different sample rates correspond to different temporal resolutions of the SBR frames.

An SBR decoder usually contains several different parts. It contains a bitstream decoding module, a high frequency reconstruction (HFR) module, an additional high frequency components module, and an envelope corrector module. The system is based on a complex-valued QMF filter bank (for high quality SBR) or a real-valued QMF filter bank (for low power SBR). Embodiments of the present invention are applicable to both high quality SBR and low power SBR. In the bitstream extractor, control data is read from the bitstream and decoded. Before reading the envelope data from the bit stream for the current frame, a time-frequency grid is obtained. The underlying core decoder decodes the audio signal of the current frame (albeit at a lower sampling rate) to obtain audio samples in the time domain. The resulting frame of audio data is used by the HFR module to reconstruct the high frequencies. The decoded low band signal is then subjected to analysis using a QMF filter bank. Then, on discrete values of the subbands of the QMF filter block, high-frequency reconstruction and envelope correction are performed. The high frequencies are reconstructed from the low range in a flexible manner based on the given control parameters. In addition, in order to ensure proper spectral characteristics of a given time/frequency domain, the reconstructed upper band is subjected to adaptive filtering based on the subband channels in accordance with the control data.

The highest level of an MPEG-4 AAC bitstream is a sequence of data blocks ("raw_data_block" elements), each of which is a data segment (referred to in this document as a "block") containing audio data (typically for a time span of 1024 or 960 discrete values), and related information and/or other data. In this document, the term "block" is used to refer to a segment of an MPEG-4 AAC bitstream containing audio data (and associated metadata, and optionally other related data) that defines or specifies one (but no more than one) raw_data_block element.

Each MPEG-4 AAC bitstream block may contain a number of syntax elements (each of which is also implemented in the bitstream as a data segment). The MPEG-4 AAC standard defines seven types of such syntax elements. Each syntax element is identified by a different value of the "id_syn_ele" data element. Examples of syntax elements include "single_channel_element()", "channel_pair_element()", and "fill_element()". The single_channel_element element is a container containing the audio data of a single audio channel (a mono audio signal). The channel_pair_element contains the audio data of two audio channels (ie, a stereo audio signal).

A fill_element is a container of information containing an identifier (eg, the value of the "id_syn_ele" element noted above) followed by data called "fill data". Padding elements have historically been used to correct the instantaneous bit rate of bit streams to be transmitted over a constant rate channel. By adding an appropriate amount of padding data to each block, a constant data rate can be achieved.

According to embodiments of the present invention, the padding data may contain one or more extension payloads that extend the data type (eg, metadata) and may be transmitted in a bitstream. A decoder receiving bitstreams with padding data containing a new data type may optionally be used by a device receiving the bitstream (eg, a decoder) to extend the functionality of the device. Thus, as can be understood by one skilled in the art, padding elements are a special type of data structure and are different from data structures commonly used to convey audio data (eg, payload audio data containing channel data).

In some embodiments of the present invention, the identifier used to identify the padding element may consist of a three-bit unsigned integer with the most significant bit ("uimsbf") having a value of 0x6 transmitted first. There can be several instances of a syntax element of the same type in one block (for example, several padding elements).

Another audio bitstream encoding standard is the MPEG Unified Speech and Audio Coding (USAC) standard (ISO/IEC 23003-3:2012). The MPEG USAC standard describes the encoding and decoding of audio content using spectral band copy processing (including SBR processing as described in the MPEG-4 AAC standard, as well as other advanced forms of spectral band copy processing). This processing uses spectral band copy tools (sometimes referred to herein as "enhanced SBR tools" or "eSBR tools") from an extended and improved version of the SBR toolset described in the MPEG-4 AAC standard. Thus, eSBR (defined in the USAC standard) is an improvement on SBR (defined in the MPEG-4 AAC standard).

In this document, the term "enhanced SBR processing" (or "eSBR processing") is used to refer to spectral band copy processing using at least one eSBR tool (e.g., at least one eSBR tool described or referred to in the MPEG USAC standard), which is not described or mentioned in the MPEG-4 AAC standard. Examples of such eSBR tools are harmonic transposition and pre-processing or "pre-smoothing" QMF inserts.

The integer-order harmonic transposer T maps a sinusoid with a frequency ω into a sinusoid with a frequency Tω while maintaining the signal duration. Typically, three orders, T = 2, 3, 4, are used in sequence to obtain each part of the required output bandwidth, using the smallest possible order of transposition. If output above the fourth order of the transposition range is required, it can be obtained by frequency shifts. In order to minimize computational complexity, when possible, sampled baseband time domains close to critical are created for processing.

The harmonic transposer can be based on either QMF or DFT. With a QMF-based harmonic transposer, the core encoder's time domain bandwidth extension is performed entirely in the QMF domain using a modified phase vocoder design that performs decimation followed by time stretching for each QMF subband. Transposition using multiple transposition coefficients (eg T = 2, 3, 4) is performed in a common QMF analysis/synthesis transformation step. Since the QMF-based harmonic transposer does not have signal-adaptive oversampling in the frequency domain, the corresponding flag in the bitstream ("sbrOversamplingFlag[ch]") can be ignored.

To reduce complexity when using a DFT-based harmonic transposer, transposers with coefficients 3 and 4 (transposers of the 3rd and 4th orders) are preferably embedded in the transposer with a coefficient of 2 (transposer of the 2nd order) by interpolation. For each frame (corresponding to the coreCoderFrameLength core coder samples), the nominal "overall size" of the transposer transform is primarily determined in the bitstream by a signal-adaptive frequency domain oversampling flag ("sbrOversamplingFlag[ch]").

If sbrPatchingMode==1, indicating that a linear transposition is to be used to generate the high range, then an additional step can be introduced to avoid discontinuities in the shape of the envelope spectrum of the high frequency signal injected into the subsequent envelope corrector. This improves the performance of the next stage of envelope correction, resulting in a more stable perceived high-end signal. The operation of additional pre-processing is useful for signal types in which the coarse spectral envelope of the low-band signal used for high-frequency reconstruction exhibits large level variations. However, the meaning of this bitstream element can be determined at the encoder by applying any type of signal dependent classification. The additional preprocessing is preferably activated with a one-bit bitstream element, "bs_sbr_preprocessing". If "bs_sbr_preprocessing" is set to one, preprocessing is enabled. If "bs_sbr_preprocessing" is set to zero, additional preprocessing is disabled. The post-processing preferably uses a preGain curve, which is used by the high frequency generator to scale the lower range, X _Low , for each key. For example, the preGain curve can be calculated using the formula:

where k ₀ is the first QMF subband in the principal band table and lowEnvSlope is calculated using a function that calculates the coefficients of the best fit polynomial (in the least squares sense), such as polyfit(). For example,

can be used (using a third degree polynomial) and where

where x_lowband(k) = [0…k ₀ -1], numTimeSlot is the number of SBR envelope timeslots that exist within a frame, RATE is a constant indicating the number of discrete QMF subband values per time period (for example, 2), ϕ _k is linear prediction filter coefficient (potentially obtained by the covariance method), and where

.

.

A bitstream generated according to the MPEG USAC standard (sometimes referred to herein as a "USAC bitstream") contains encoded audio content and typically contains metadata indicating each type of spectral band copy processing that a decoder needs to apply to decode the audio content of the USAC bitstream, and/or metadata governing such spectral band copy processing and/or indicating at least one characteristic or parameter of at least one SBR and/or eSBR instrument to be applied to decode the audio content of the USAC bitstream.

In this document, the term "enhanced SBR metadata" (or "eSBR metadata") is used to refer to metadata indicating each type of spectral band copy processing that a decoder needs to apply to decode the audio content of an encoded audio bitstream (e.g., a USAC bitstream), and /or which control such spectral band copy processing, and/or specifying at least one characteristic or parameter of at least one SBR and/or eSBR instrument to be applied to decode such audio content, but which is not described or mentioned in the standard MPEG-4 AAC. An example of eSBR metadata is metadata (indicating or controlling spectral band copy processing) that is described or referred to in the MPEG USAC standard, but not in the MPEG-4 AAC standard. Thus, eSBR metadata herein refers to metadata that is not SBR metadata, and SBR metadata herein refers to metadata that is not eSBR metadata.

The USAC bitstream may contain both SBR metadata and eSBR metadata. More specifically, the USAC bitstream may contain eSBR metadata that controls the execution of eSBR processing by the decoder and SBR metadata that controls the execution of SBR processing by the decoder. According to exemplary embodiments of the present invention, eSBR metadata (eg, eSBR specific configuration data) is included (according to the present invention) in an MPEG-4 AAC bitstream (eg, in an sbr_extension() container at the end of the SBR payload).

When performing eSBR processing, during decoding of the encoded bitstream using the eSBR toolkit (comprising at least one eSBR tool), the decoder recovers the upper range of the audio signal based on copying the harmonic sequences that were truncated during encoding. This eSBR processing typically corrects the spectral envelope of the recovered high frequency range, applies inverse filtering, and adds noise and sine components to recreate the spectral characteristics of the original audio signal.

According to conventional embodiments of the present invention, eSBR metadata includes (e.g., includes a small number of control bits representing eSBR metadata) in one or more metadata segments of an encoded audio bitstream (e.g., an MPEG-4 AAC bitstream) that also contains encoded audio data. in other segments. Typically, at least one such metadata segment of each bitstream block is (or contains) a padding element (containing an identifier indicating the start of the padding element), and the eSBR metadata is included in the padding element after this identifier.

In FIG. 1 is a block diagram of an exemplary audio processing chain (audio data processing system) in which one or more system elements may be implemented according to one embodiment of the present invention. The system contains the following elements connected together as shown: an encoder 1, a delivery subsystem 2, a decoder 3 and a post-processing unit 4. Variations of the system shown omit one or more elements or include additional audio data processing units.

In some implementations, encoder 1 (optionally including a preprocessor) is configured to receive as input PCM (time domain) samples containing audio content and output an encoded audio bitstream (having a format compatible with the MPEG-4 AAC standard) , which characterizes the audio content. Bitstream data characterizing audio content is sometimes referred to as "audio data" or "encoded audio data". If the encoder is implemented according to an exemplary embodiment of the present invention, the audio bitstream output from the encoder contains eSBR metadata (and typically other metadata) as well as audio data.

One or more encoded audio bitstreams output from the encoder 1 may be transmitted to the encoded audio delivery subsystem 2. Subsystem 2 is configured to store and/or deliver each encoded bitstream output from encoder 1. The encoded audio bitstream output from encoder 1 may be stored in subsystem 2 (for example, in the form of a DVD or Blu ray disc), or be transmitted by subsystem 2 (which may implement a communication channel or network), or may be both stored and transmitted by subsystem 2.

Decoder 3 is configured to decode an MPEG-4 AAC encoded audio bitstream (generated by encoder 1) that it receives via subsystem 2. In some embodiments, decoder 3 is configured to extract eSBR metadata from each block of the bitstream and decode the bitstream (in including by performing eSBR processing using the extracted eSBR metadata) to generate decoded audio data (eg, PCM decoded audio sample streams). In some embodiments, the decoder 3 is configured to extract SBR metadata from the bitstream (however, ignoring the eSBR metadata included in the bitstream) and decode the bitstream (including by performing SBR processing using the extracted SBR metadata) to generate decoded audio data. (eg, PCM decoded audio sample streams). Typically, decoder 3 contains a buffer that stores (eg, non-volatilely) segments of the encoded audio bitstream received from subsystem 2.

The post-processing block 4 of FIG. 1 is configured to receive a stream of decoded audio data from the decoder 3 (eg, decoded PCM audio samples) and perform post-processing on them. The post-processing unit may also be configured to present post-processed audio content (or decoded audio received from decoder 3) for playback by one or more speakers.

In FIG. 2 shows a block diagram of an encoder (100) which is one embodiment of an audio processing unit according to the invention. Any of the components or elements of encoder 100 may be implemented as one or more processes and/or one or more circuits (eg, ASICs, FPGAs, or other integrated circuits), in hardware, software, or a combination of hardware and software. The encoder 100 includes an encoder 105, a bit rate generator/formatter stage 107, a metadata generation stage 106, and a memory buffer 109 connected as shown. Typically, encoder 100 also contains other processing elements (not shown). The encoder 100 is configured to convert the input audio bitstream into an encoded MPEG-4 AAC output bitstream.

Metadata generator 106 is connected and configured to generate (and/or pass to stage 107) metadata (including eSBR metadata and SBR metadata) for inclusion by stage 107 in the encoded bitstream for output from encoder 100.

The encoder 105 is connected and configured to encode the input audio data (for example, by performing compression) and pass the resulting encoded audio to stage 107 for inclusion in the encoded bitstream for output from stage 107.

Stage 107 is configured to multiplex the encoded audio from encoder 105 and metadata (including eSBR metadata and SBR metadata) from generator 106 to generate an encoded bitstream for output from stage 107, preferably such that the encoded bitstream has a format specified by one of embodiments of the present invention.

Memory buffer 109 is configured to store (eg, non-volatilely) at least one encoded audio bitstream block from stage 107, and a sequence of encoded audio bitstream blocks is then transmitted from buffer memory 109 as output from encoder 100 to the delivery system.

In FIG. 3 shows a block diagram of a system comprising a decoder (200), which is one embodiment of an audio processing unit according to the invention, and optionally also a post-processor (300) connected to it. Any of the components, or elements, of decoder 200 and post processor 300 may be implemented as one or more processes and/or one or more circuits (eg, ASICs, FPGAs, or other integrated circuits), in hardware, software, or a combination of hardware and software. The decoder 200 includes a memory buffer 201, a bitstream payload deformatter (parser) 205, an audio decoding subsystem 202 (sometimes referred to as a "main" decoding stage or a "main" decoding subsystem), an eSBR processing stage 203, and a control generation stage 204. bits connected as shown. Typically, decoder 200 also contains other processing elements (not shown).

Buffer memory (buffer) 201 stores (eg, non-volatilely) at least one block of the encoded MPEG-4 AAC audio bitstream received by decoder 200. In operation of decoder 200, a sequence of bitstream blocks is transferred from buffer 201 to deformatter 205.

In variations of the embodiment of FIG. 3 (or the embodiment of FIG. 4 to be described), an APU that is not a decoder (eg, APU 500 of FIG. 6) contains a buffer memory (eg, a buffer memory identical to buffer 201) that stores (eg, non-volatile) at least one block of an encoded audio bitstream (eg, MPEG-4 AAC audio bitstream) of the same type as received by buffer 201 of FIG. 3 or fig. 4 (i.e., an encoded audio bitstream containing eSBR metadata).

Also with reference to FIG. 3, a deformatter 205 is coupled and configured to demultiplex each block of the bitstream to extract SBR metadata (including quantized envelope data) and eSBR metadata (and typically other metadata) therefrom in order to transmit at least eSBR metadata and metadata. SBR to the eSBR processing stage 203, and also typically to pass other extracted metadata to the decoding subsystem 202 (and optionally also to the control bit generator 204). The deformatter 205 is also connected and configured to extract audio data from each block of the bitstream and transmit the extracted audio data to a decoding subsystem 202 (decoding stage).

The system shown in Fig. 3 optionally also includes a post processor 300. The post processor 300 includes a storage buffer (buffer) 301 and other processing elements (not shown), including at least one processing element coupled to buffer 301. Buffer 301 stores (e.g., non-volatile) at least one block (or frame) of decoded audio data received by post processor 300 from decoder 200. Processing elements of post processor 300 are connected and configured to receive and adaptively process a sequence of blocks (or frames) of decoded audio output from buffer 301 using metadata output from decoding subsystem 202 (and/or deformatting device 205) and/or outputting control bits from stage 204 of decoder 200.

The audio decoding subsystem 202 of the decoder 200 is configured to decode the audio data extracted by the parser 205 (such decoding can be referred to as a "main" decoding operation) to generate decoded audio data and pass the decoded audio data to the eSBR processing stage 203. Decoding is performed in the frequency domain and typically includes inverse quantization followed by spectral processing. Typically, the last stage of processing in subsystem 202 applies a frequency domain to time domain transform to the decoded audio data in the frequency domain such that the output of the subsystem is the decoded audio data in the time domain. Stage 203 is configured to apply SBR tools and eSBR tools indicated by SBR and eSBR metadata (extracted by parser 205) to decoded audio data (i.e., perform SBR and eSBR processing on the output of decoding subsystem 202 using SBR and eSBR metadata) to generate fully decoded audio data output (eg, to post processor 300) from decoder 200. Typically, decoder 200 includes a storage device (accessible to subsystem 202 and stage 203) that stores the formatted audio data and metadata output from the device 205 deformatting, and stage 203 is configured to access this audio data and metadata (including SBR and eSBR metadata) as needed during SBR and eSBR processing. The SBR processing and eSBR processing at stage 203 can be considered post-processing on the output of the main decoding subsystem 202 . Optionally, the decoder 200 also includes a post-upmix subsystem (which can apply the parametric stereophony ("PS") tools defined in the MPEG-4 AAC standard using the PS metadata extracted by the deformatter 205 and/or control bits generated in the subsystem 204) coupled and configured to perform upmixing on the output of stage 203 to generate fully decoded and upmixed audio that is output from decoder 200. Alternatively, post processor 300 is configured to perform upmixing on the output of decoder 200 (e.g., with using the PS metadata extracted by the deformatter 205 and/or control bits generated in the subsystem 204).

In response to the metadata extracted by the format remover 205, the control bit generator 204 may generate control data, and this control data may be used in the decoder 200 (e.g., in a post-upmix subsystem) and/or transmitted as an output of the decoder 200 (e.g., to the post processor 300 for use in post processing). In response to the metadata extracted from the input bitstream (and optionally also in response to control data), stage 204 may generate (and pass to post processor 300) control bits indicating that the decoded audio data output from eSBR processing stage 203 should be post-process a specific type. In some implementations, decoder 200 is configured to pass the metadata extracted by deformatter 205 from the input bitstream to post processor 300, and post processor 300 is configured to perform post processing on the decoded audio data output from decoder 200 using this metadata.

In FIG. 4 is a block diagram of an audio processing unit ("APU") (210) which is another embodiment of an audio processing unit according to the invention. APU 210 is a conventional decoder without the ability to perform eSBR processing. Any of the components or elements of APU 210 may be implemented as one or more processes and/or one or more circuits (eg, ASICs, FPGAs, or other integrated circuits), in hardware, software, or a combination of hardware and software. The APU 210 includes a memory buffer 201, a bitstream payload deformatter (parser) 215, an audio decoding subsystem 202 (sometimes referred to as a "main" decoding stage or a "main" decoding subsystem), and an SBR processing stage 213 connected as shown. Typically, APU 210 also contains other processing elements (not shown). APU 210 may represent, for example, an audio encoder, decoder, or transcoder.

The elements 201 and 202 of the APU 210 are identical to the identically numbered elements of the decoder 200 (of FIG. 3) and their description above will not be repeated. In operation of the APU 210, a block sequence of an encoded audio bitstream (MPEG-4 AAC bitstream) received by the APU 210 is transmitted from the buffer 201 to the deformatter 215 .

Deformatter 215 is coupled and configured to demultiplex each block of the bitstream to extract SBR metadata (including quantized envelope data) and generally other metadata from it, but ignoring eSBR metadata that may be included in the bitstream according to any of the embodiments of the present invention. The format remover 215 is configured to transmit at least SBR metadata to the SBR processing stage 213 . The deformatter 215 is also connected and configured to extract audio data from each block of the bitstream and transmit the extracted audio data to a decoding subsystem 202 (decoding stage).

The audio decoding subsystem 202 of the decoder 200 is configured to decode the audio data extracted by the deformatter 215 (such decoding can be referred to as a "main" decoding operation) to generate decoded audio data and transmit the decoded audio data to the SBR processing stage 213. The decoding is performed in the frequency domain. Typically, the last stage of processing in subsystem 202 applies a frequency domain to time domain transform to the decoded audio data in the frequency domain such that the output of the subsystem is the decoded audio data in the time domain. Stage 213 is configured to apply the SBR tools (but not the eSBR tools) indicated by the SBR metadata (extracted by the format remover 215) to the decoded audio data (i.e., perform SBR processing on the output of the decoding subsystem 202 using the SBR metadata) to generate fully decoded audio data output (e.g., to post processor 300) from APU 210. Typically, APU 210 includes a storage device (accessible to subsystem 202 and stage 213) that stores the formatted audio data and metadata output from deletion device 215 formatting, and stage 213 is configured to access this audio data and metadata (including SBR metadata) as needed during SBR processing. The SBR processing at step 213 can be considered post-processing on the output of the main decoding subsystem 202 . Optionally, the APU 210 also includes a post-upmix subsystem (which can apply Parametric Stereo (“PS”) tools defined in the MPEG-4 AAC standard using the PS metadata extracted by the format remover 215) coupled and configured to perform the upmix. with respect to the output of the stage 213 for generating the fully decoded and upmixed audio that is output from the APU 210. Alternatively, the post processor is configured to perform upmixing with respect to the output of the APU 210 (for example, using the PS metadata extracted by the deformatter 215, and/ or control bits generated in the APU 210).

Various implementations of encoder 100, decoder 200, and APU 210 are configured to perform different embodiments of the method of the invention.

In some embodiments, the eSBR metadata is included (eg, a small number of control bits representing the eSBR metadata are included) in an encoded audio bitstream (eg, in an MPEG-4 AAC bitstream) such that conventional decoders (which are implemented without the ability to perform syntactic parsing the eSBR metadata or using whatever eSBR tool to which the eSBR metadata refers) can ignore that eSBR metadata but still decode the bitstream to the extent possible without using the eSBR metadata or whatever eSBR tool to which the eSBR refers. this eSBR metadata is generally without any significant loss in the quality of the decoded audio. However, eSBR decoders configured to parse the bitstream to identify eSBR metadata, and use at least one eSBR tool in response to that eSBR metadata, will benefit from using at least one such eSBR tool. Therefore, embodiments of the present invention provide a means for efficiently transmitting control data or Enhanced Spectral Band Copy (eSBR) metadata in a backwards compatible manner.

Typically, eSBR metadata in a bitstream indicates (eg, indicates at least one characteristic or parameter thereof) one or more of the following eSBR tools (which are described in the MPEG USAC standard and which may or may not have been applied by the encoder during bitstream generation). flow):

• harmonic transposition; And

• additional pre-processing (pre-smoothing) QMF inserts.

For example, eSBR metadata included in the bitstream may indicate parameter values (described in the MPEG USAC standard and the present invention): sbrPatchingMode[ch], sbrOversamplingFlag[ch], sbrPitchInBins[ch], sbrPitchInBins[ch], and bs_sbr_preprocessing.

In this document, the notation "X[ch]", where X is a parameter, means that this parameter refers to the channel ("ch") of the audio content of the encoded bitstream to be decoded. For simplicity, the expression [ch] is sometimes omitted and the corresponding parameter is assumed to refer to an audio content channel.

In this document, the notation X[ch][env], where X is a parameter, means that this parameter refers to the envelope ("env") of the SBR channel ("ch") of the audio content of the encoded bitstream to be decoded. For simplicity, the expressions [env] and [ch] are sometimes omitted and the corresponding parameter is assumed to refer to the SBR envelope of the audio content channel.

During decoding of the encoded bitstream, the harmonic transposition operation in the eSBR decoding processing stage (for each channel, "ch", of the audio content indicated by the bitstream) is controlled by the following eSBR metadata parameters: sbrPatchingMode[ch]: sbrOversamplingFlag[ch]; sbrPitchInBinsFlag[ch]; and sbrPitchInBins[ch].

The value "sbrPatchingMode[ch]" indicates the type of transposer used in eSBR: sbrPatchingMode[ch] = 1 indicates insertion by linear transposition as described in section 4.6.18 of the MPEG-4 AAC standard (when used with either high quality SBR or SBR low power); sbrPatchingMode[ch] = 0 indicates SBR harmonic patching as described in section 7.5.3 or 7.5.4 of the MPEG USAC standard.

The value "sbrOversamplingFlag[ch]" indicates that the eSBR uses signal-adaptive frequency domain oversampling in combination with DFT-based SBR harmonic insertion, as described in section 7.5.3 of the MPEG USAC standard. This flag controls the size of the DFTs used in the transposer: 1 indicates that signal-adaptive frequency domain resampling is enabled, as described in section 7.5.3.1 of the MPEG USAC standard; 0 indicates that signal-adaptive frequency domain resampling is disabled, as described in section 7.5.3.1 of the MPEG USAC standard.

The value "sbrPitchInBinsFlag[ch]" controls the interpretation of the sbrPitchInBins[ch] parameter: 1 indicates that the value in sbrPitchInBins[ch] is valid and greater than zero; 0 indicates that the value of sbrPitchInBins[ch] is set to zero.

The value "sbrPitchInBins[ch]" controls the addition of cross product multipliers in the SBR harmonic transposer. The value of sbrPitchinBins[ch] is an integer in the range [0,127] and represents the distance, measured in frequency bins, for a 1536-line DFT acting on the core encoder sample rate.

In the case where an MPEG-4 AAC bitstream characterizes a pair of SBR channels whose channels are not connected (rather than a single SBR channel), the bitstream characterizes two instances of the above syntax (for harmonic or non-harmonic transposition), one for each channel sbr_channel_pair_element() .

The harmonic transposition of an eSBR instrument tends to improve the quality of decoded music signals at relatively low crossover frequencies. Non-harmonic transposition (i.e. traditional spectral insertion) tends to improve speech signals. Thus, the starting point in deciding which type of transposition is preferable for encoding a particular audio content is the choice of transposition method depending on the detection of speech / music, with harmonic transposition applied to musical content, and spectral insertion to speech content. .

The operation of pre-aliasing during eSBR processing is controlled by the value of a one-bit eSBR metadata parameter known as "bs_sbr_preprocessing", in the sense that pre-aliasing is either performed or not performed, depending on the value of this single bit. When using the SBR insertion QMF algorithm described in section 4.6.18.6.3 of the MPEG-4 AAC standard, a pre-smoothing step may be performed (if specified by the "bs_sbr_preprocessing" parameter) in an attempt to avoid discontinuities in the shape of the envelope spectrum of the high-frequency signal input to the envelope corrector ( envelope corrector performs another eSBR processing stage). Pre-smoothing tends to improve the performance of the next stage of envelope correction, resulting in a more stable perceived high-end signal.

It is expected that the overall bit rate requirements for including in an MPEG-4 AAC bitstream eSBR metadata indicating the aforementioned eSBR tools (harmonic transposition and pre-smoothing) are in the order of several hundred bits per second, since according to some embodiments of the present invention, only differential control data required to perform eSBR processing. Legacy decoders may ignore this information as it is included for backwards compatibility (as will be explained later). Therefore, the negative impact on bit rate associated with the inclusion of eSBR metadata is negligible, for a variety of reasons, including the following:

• The data bit rate loss (due to the inclusion of eSBR metadata) is a very small fraction of the total bit rate, since only the differential control data necessary to perform eSBR processing is transmitted (and no SBR control data is simultaneously transmitted); And

• The setting of the control information related to the SBR is generally independent of the details of the transposition. Examples of how the control data actually depends on the operation of the transposer are discussed in this application below.

Thus, embodiments of the present invention provide a means for efficiently transmitting control data or enhanced spectral band copying (eSBR) metadata in a backwards compatible manner. This efficient transfer of eSBR control data reduces the memory requirements of decoders, encoders, and transcoders employing aspects of the present invention, while not having a measurable negative impact on the bit rate. In addition, the complexity and processing requirements associated with performing an eSBR according to the embodiments of the present invention are also reduced because the SBR data only needs to be processed once, rather than transmitted simultaneously, as would be the case if the eSBR were processed as a completely separate entity type. into MPEG-4 AAC instead of integration into the MPEG-4 AAC codec for backwards compatibility.

Next, with reference to FIG. 7, block elements ("raw_data_block") of an MPEG-4 AAC bitstream in which eSBR metadata is included according to some embodiments of the present invention are described. In FIG. 7 is a block diagram ("raw_data_block") of an MPEG-4 AAC bitstream showing some of its segments.

An MPEG-4 AAC bitstream block may comprise at least one single_channel_element() (e.g., the single channel element shown in FIG. 7) and/or at least one channel_pair_element() (not specifically shown in FIG. 7, although it may be present) containing audio data for an audio program. The block may also contain a number of "fill_elements" (eg fill element 1 and/or fill element 2 in FIG. 7) containing data (eg metadata) related to the program. Each "single_channel_element()" contains an identifier (eg, "ID1" in FIG. 7) indicating the start of a single channel element, and may contain audio data characterizing a single channel of a multi-channel audio program. Each "channel_pair_element" contains an identifier (not shown in FIG. 7) indicating the start of a channel pair element, and may contain audio data characterizing the two channels of the program.

The fill_element (herein referred to as the fill element) of an MPEG-4 AAC bitstream contains an identifier ("ID2" in FIG. 7) indicating the start of the fill element and the fill data following that identifier. ID2 may consist of a three-bit unsigned integer with the most significant bit (“uimsbf”) transmitted first, having the value 0x6. The padding data may contain an extension_payload() element (sometimes referred to in this document as extension payload), the syntax of which is shown in table 4.57 of the MPEG-4 AAC standard. There are several extension payload types, which are identified by the "extension_type" parameter, which is a four-bit unsigned integer with the most significant bit ("uimsbf") passed first.

The padding data (eg, their extension payload) may contain a header, or an identifier (eg, "Header 1" in FIG. 7) indicating a padding segment indicating an SBR object (i.e., the header initializes the "SBR object" type, in the MPEG-4 AAC standard called sbr_extension_data()). For example, a Spectral Band Copy (SBR) extension payload is identified by the value "1101" or "1110" for the extension_type field in the header, with "1101" identifying the extension payload with SBR data and "1110" identifying the extension payload with data SBR and cyclic redundancy check (CRC) to check the correctness of the SBR data.

If the header (eg, the extension_type field) initializes the type of the SBR object, the SBR metadata (sometimes referred to in this document as "spectral band copy data" and called sbr_data() in the MPEG-4 AAC standard) follows the header, and the SBR metadata may be followed by at least one spectral band copy extension element (eg, the "SBR extension element" of padding element 1 in FIG. 7). Such a spectral band copy extension element (bitstream segment) in the MPEG-4 AAC standard is called a "sbr_extension()" container. The spectral band copy extension element optionally contains a header (eg, "SBR extension header" of padding element 1 in FIG. 7).

The MPEG-4 AAC standard provides that the spectral band copy extension element may contain PS (parametric stereo) data for program audio data. The MPEG-4 AAC standard stipulates that if a fill element header (e.g., its extension payload) initializes an SBR object type (as "Header 1" in FIG. 7 does) and the fill element spectral band copy extension element contains PS data, then this padding element (eg, its extension payload) contains spectral band copy data and a parameter "bs_extension_id" whose value (i.e., "bs_extension_id" = 2) indicates that the PS data is included in the fill element's spectral band copy extension element.

According to some embodiments of the present invention, eSBR metadata (eg, a flag that indicates whether enhanced spectral band copy (eSBR) processing is to be performed on the audio content of the block) is included in the fill element spectral band copy extension element. For example, such a flag is indicated in pad 1 in FIG. 7, with the flag appearing after the header ("SBR extension header" of fill element 1) of the "SBR extension element" of fill element 1. Optionally, such a flag and additional eSBR metadata are included in the spectral band copy extension element after the spectral band copy extension element header (e.g. , to the SBR extension element of padding element 1 in Fig. 7 after the SBR extension header). According to some embodiments of the present invention, the padding element containing eSBR metadata also contains a "bs_extension_id" parameter whose value (for example, bs_extension_id = 3) indicates that eSBR metadata is included in the padding element and that eSBR processing is to be performed on the audio content of the corresponding block.

According to some embodiments of the present invention, the eSBR metadata is included in a padding element (e.g., padding element 2 in FIG. 7) of an MPEG-4 AAC bitstream other than the spectral band copy extension element (SBR extension element) of the padding element. This is because padding elements containing extension_payload() with SBR data or SBR data with CRC do not contain any other extension payload of any other type of extension. Therefore, in embodiments where the eSBR metadata stores its own extension payload, a separate padding element is used to store the eSBR metadata. Such a padding element contains an identifier (eg, "ID2" in FIG. 7) indicating the start of the padding element, and padding data after the identifier. The padding data may contain an extension_payload() element (sometimes referred to in this document as extension payload), the syntax of which is shown in table 4.57 of the MPEG-4 AAC standard. The padding data (eg, their extension payload) contains a header (eg, "Header 2" of padding element 2 in FIG. 7) indicating an eSBR object (i.e., the header initializes the Enhanced Spectral Band Copy (eSBR) object type), and padding data (such as its extension payload) contains eSBR metadata after the header. For example, infill element 2 in FIG. 7 contains such a header ("Header 2") and also contains, after the header, eSBR metadata (i.e., the "flag" in padding element 2 that indicates whether enhanced spectral band copying (eSBR) processing is to be performed on the audio content. block). Optionally, additional eSBR metadata is also included in the padding data of padding element 2 in FIG. 7, after "Heading 2". In the embodiments described in this paragraph, the header (e.g., "Header 2" in FIG. 7) has an identification value that is not one of the normal values specified in Table 4.57 of the MPEG-4 AAC standard and instead indicates an extension payload. eSBR (so the header's extension_type field indicates that the padding data contains eSBR metadata).

In a first class of embodiments, the present invention is an audio processing unit (eg, a decoder) comprising:

a memory (eg, buffer 201 shown in FIG. 3 or 4) configured to store at least one encoded audio bitstream block (eg, at least one MPEG-4 AAC bitstream block);

a bitstream payload format remover (eg, element 205 of FIG. 3 or element 215 of FIG. 4) coupled to the storage device and configured to demultiplex at least one portion of said bitstream block; And

a decoding subsystem (for example, elements 202 and 203 shown in Fig. 3, or elements 202 and 213 shown in Fig. 4), connected and configured to decode at least one part of the audio content of the specified bitstream block, where this block contains:

a padding element containing an identifier indicating the start of the padding element (e.g., the identifier "id_syn_ele", having the value 0x6, in Table 4.85 of the MPEG-4 AAC standard), and padding data after the identifier, where the padding data contains:

at least one flag identifying whether enhanced spectral band replication (eSBR) processing is to be performed on the audio content of the block (eg, using spectral band replication data and eSBR metadata included in the block).

The flag is eSBR metadata, and an example of a flag is the sbrPatchingMode flag. Another example of a flag is the harmonicSBR flag. Both of these flags indicate whether a basic form of spectral band copying or an enhanced form of spectral copying should be performed on the block's audio data. The basic form of spectral copying is spectral insertion, and the enhanced form of spectral band copying is harmonic transposition.

In some embodiments, the padding data also contains additional eSBR metadata (ie, eSBR metadata other than the flag).

The storage device may be a buffer storage device (eg, an implementation of the buffer 201 shown in FIG. 4) that stores (eg, non-volatilely) at least one block of the encoded audio bitstream.

The complexity of performing eSBR processing (using harmonic transposition and eSBR pre-smoothing) by an eSBR decoder in the course of decoding an MPEG-4 AAC bitstream that contains eSBR metadata (indicating these eSBR tools) is estimated to be as follows (for normal decoding with the specified parameters) :

• harmonic transposition (16 kbps, 14400 / 28800 Hz)

○ DFT-based: 3.68 WMOPS (weighted million operations per second);

○ Based on QMF: 0.98 WMOPS;

• pre-processing (pre-smoothing) QMF insertion: 0.1 WMOPS.

It is known that, for transient signals, DFT-based transposition generally performs better than QMF-based transposition.

According to some embodiments of the present invention, the padding element (of the encoded audio bitstream) containing the eSBR metadata also contains a parameter (e.g., the "bs_extension_id" parameter) whose value (e.g., bs_extension_id = 3) signals that the eSBR metadata is included in the given fill element and that eSBR processing needs to be performed on the audio content of the corresponding block, and/or a parameter (for example, the same "bs_extension_id" parameter) whose value (for example, bs_extension_id = 2) signals that the fill element's sbr_extension() container contains PS data. For example, as indicated in Table 1 below, such a parameter having a value of bs_extension_id = 2 may signal that the padding element's sbr_extension() container contains PS data, and such a parameter having a value of bs_extension_id = 3 may signal that the padding element's sbr_extension() container contains the eSBR metadata:

Table 1

According to some embodiments of the present invention, the syntax of each spectral band copy extension element containing eSBR metadata and/or PS data is as indicated in Table 2 below (in which "sbr_extension()" denotes a container that is a spectral band copy extension element, "bs_extension_id ' is as described in Table 1 above, 'ps_data' stands for PS data and 'esbr_data' stands for eSBR metadata).

table 2

In the exemplary embodiment, esbr_data() referenced in Table 2 above specifies the values of the following metadata parameters:

1. one-bit metadata parameter "bs_sbr_preprocessing"; And

2. for each channel ("ch") of the audio content of the encoded bitstream to be decoded, each of the above parameters: "sbrPatchingMode[ch]"; "sbrOversamplingFlag[ch]"; "sbrPitchInBinsFlag[ch]"; and "sbrPitchInBins[ch]".

For example, to specify these metadata options, some embodiments of esbr_data() may have the syntax shown in Table 3.

Table 3

The above syntax allows for an efficient implementation of an improved form of spectral band copying, such as harmonic transposition, as an extension to a conventional decoder. In particular, the eSBR data in Table 3 contains only those parameters needed to perform an enhanced form of spectral band copying that are either not already supported in the bitstream or are not directly derivable from the parameters already supported in the bitstream. All other parameters and processing data necessary to perform an improved form of spectral band copying are derived from pre-existing parameters at already defined locations in the bitstream.

For example, an MPEG-4 HE-AAC or HE-AAC v2 compliant decoder can be extended to include an improved form of spectral band copying, such as harmonic transposition. This enhanced form of spectral band copying is an addition to the basic spectral band copying form already supported by the decoder. In the context of an MPEG-4 HE-AAC or HE-AAC v2 compliant decoder, this basic form of spectral band copying is the SBR spectral QMF paste tool as defined in section 4.6.18 of the MPEG-4 AAC standard.

By performing an improved form of spectral band copying, the extended HE-AAC decoder can again use many of the bitstream parameters already included in the SBR extension payload of the bitstream. Specific parameters that can be used again include, for example, various parameters defining a table of main frequency bands. These parameters include bs_start_freq (a parameter that specifies the start of the main frequency table parameter), "bs_stop_freq" (a parameter that specifies the end of the main frequency table), "bs_freq_scale" (a parameter that specifies the number of frequency bands per octave), and "bs_alter_scale" (a parameter that changes the scale frequency bands). Parameters that can be reused also include parameters specifying a noise band table ("bs_noise_bands") and limiter band table parameters ("bs_limiter_bands"). Accordingly, in various embodiments, at least some of the equivalent parameters specified in the USAC standard are not included in the bitstream, thereby reducing control information overflow of the bitstream. In general, if a parameter defined in the AAC standard has an equivalent parameter defined in the USAC standard, then the equivalent parameter defined in the USAC standard has the same name as the parameter defined in the AAC standard, such as the E _OrigMapped envelope scale factor. However, the equivalent parameter defined in the USAC standard typically has a different value that is "tuned" to handle enhanced SBR as defined in the USAC standard rather than to handle SBR as defined in the AAC standard.

To improve the subjective quality of audio content with harmonic frequency structure and strong tonal characteristics, particularly at low bit rates, Enhanced SBR is recommended to be activated. The values of the corresponding bitstream element (i.e. esbr_data()) driving these tools can be determined in the encoder by applying a signal dependent classification mechanism. In general, using the harmonic patching mode (sbrPatchingMode == 1) is preferred for encoding music signals with very low bit rates, where the main codec can be severely limited in audio bandwidth. This is especially true if these signals have a pronounced harmonic structure. In contrast, the use of the conventional SBR insertion method is preferred for speech and mixed signals because it provides better preservation of speech temporal structure.

To improve the performance of the harmonic transposer, a pre-processing step (bs_sbr_preprocessing == 1) can be activated, which tries to avoid introducing disturbances in the homogeneity of the signal spectrum going to the subsequent envelope corrector. The operation of this tool is useful for signal types in which the coarse spectral envelope of the lower range signal used for high frequency reconstruction exhibits large level variations.

To improve the transient response of the SBR harmonic insert, you can apply signal-adaptive frequency domain oversampling (sbrOversamplingFlag == 1). Because signal-adaptive frequency domain resampling increases the computational complexity of the transposer but only benefits frames containing transient signals, the use of this tool is controlled by a bitstream element that is transmitted once per frame and on an independent SBR channel.

A decoder operating in the proposed enhanced SBR mode typically needs to be able to switch between legacy and enhanced SBR insertion. Therefore, a delay may be introduced, which may be as long as the duration of one basic frame of audio, depending on the settings of the decoder. Typically, the delay will be the same for both traditional and enhanced SBR inserts.

In addition to the numerous parameters, the enhanced HE-AAC decoder may reuse other data elements when performing an improved form of spectral band copying according to embodiments of the present invention. For example, envelope data and noise floor data can also be extracted from bs_data_env (envelope scaling factors) and bs_noise_env (noise floor scaling factors) data and used in an improved form of spectral band copying.

As such, these embodiments use the configuration parameters and envelope data already supported by a legacy HE-AAC or HE-AAC v2 decoder in the SBR extension payload to provide an improved form of spectral band replication requiring as little additional transmission data as possible. The metadata was originally set up for a basic form of HFR (eg, an SBR spectral transfer operation), but is used in embodiments for an enhanced form of HFR (eg, an eSBR harmonic transposition). As discussed earlier, the metadata typically represents operating parameters (e.g., envelope scaling factors, noise floor scaling factors, time/frequency grid parameters, sine addition information, variable frequency/crossover range, inverse filtering mode, envelope resolution, leveling mode, frequency interpolation) tuned and intended for use with the basic form of HFR (for example, with linear spectral transfer). However, this metadata, in combination with additional metadata parameters specific to the enhanced form of HFR (eg, harmonic transposition), can be used to efficiently and efficiently process audio data using the enhanced form of HFR.

Accordingly, extended decoders that support the enhanced spectral band copying form can be constructed in an extremely efficient manner by relying on already defined bitstream elements (e.g., elements in the SBR extension payload) and adding only those parameters necessary to support the improved spectral copying form. stripes (in the padding extension payload). This feature of data compression, combined with placing the newly added parameters in a reserved data field such as an extension container, greatly reduces the barriers to building a decoder that supports the enhanced spectral band copy form by making the bitstream backwards compatible with a traditional decoder that does not support the enhanced copy form. spectral band.

In Table 3, the number in the right column indicates the number of bits of the corresponding parameter from the left column.

In some embodiments, the SBR object type defined in MPEG-4 AAC is updated to contain the SBR tool and enhanced SBR tool (eSBR) aspects as indicated by the signals in the SBR extension element (bs_extension_id == EXTENSION_ID_ESBR). If the decoder detects and supports this SBR extension element, then the decoder uses signaled aspects of the enhanced SBR tool. An SBR object type updated in this way is referred to as "SBR enhancements".

In some embodiments, the present invention is a method including the step of encoding audio data to generate an encoded bitstream (e.g., an MPEG-4 AAC bitstream) containing eSBR metadata in at least one segment of at least one encoded bitstream block and audio data in at least one other segment of that block. In conventional embodiments, the method includes the step of multiplexing audio data with eSBR metadata in each encoded bitstream block. In conventional decoding of an encoded bitstream in an eSBR decoder, the decoder extracts the eSBR metadata from the bitstream (including by parsing and demultiplexing the eSBR metadata and audio data) and uses the eSBR metadata to process the audio data to generate a decoded audio data stream.

Another aspect of the present invention is an eSBR decoder capable of performing eSBR processing (eg, using at least one of the eSBR tools known as harmonic transposition or pre-smoothing) during decoding of an encoded audio bitstream (eg, an MPEG- 4 AAC) that does not include eSBR metadata. An example of such a decoder will be described with reference to FIG. 5.

The eSBR decoder (400) shown in FIG. 5 includes a memory buffer 201 (identical to the memory 201 shown in FIGS. 3 and 4), a bitstream payload deformatter 215 (identical to the format remover 215 shown in FIG. 4), an audio decoding subsystem 202 (sometimes called the "main" decoding stage or "main" decoding subsystem, which is identical to the main decoding subsystem 202 shown in Fig. 3), the eSBR control data generation subsystem 401, and the eSBR processing stage 203 (identical to the stage 203 shown in Fig. 3), connected as shown. Typically, decoder 400 also includes other processing elements (not shown).

In operation of the decoder 400, a block sequence of an encoded audio bitstream (MPEG-4 AAC bitstream) received by the decoder 400 is transmitted from the buffer 201 to the deformatter 215 .

A deformatter 215 is coupled and configured to demultiplex each bitstream block to extract SBR metadata (including quantized envelope data) and typically other metadata therefrom. The format remover 215 is configured to transmit at least the SBR metadata to the eSBR processing stage 203 . The deformatter 215 is also connected and configured to extract audio data from each block of the bitstream and transmit the extracted audio data to a decoding subsystem 202 (decoding stage).

The audio decoding subsystem 202 of the decoder 400 is configured to decode the audio data extracted by the deformatter 215 (such decoding can be referred to as a "main" decoding operation) to generate decoded audio data and pass the decoded audio data to the eSBR processing stage 203. The decoding is performed in the frequency domain. Typically, the last stage of processing in subsystem 202 applies a frequency domain to time domain transform to the decoded audio data in the frequency domain such that the output of the subsystem is the decoded audio data in the time domain. The stage 203 is configured to apply the SBR tools (and eSBR tools) indicated by the SBR metadata (extracted by the format remover 215) and the eSBR metadata generated in the subsystem 401 to the decoded audio data (i.e., performing SBR and eSBR processing on the output of the subsystem 202 decoding using SBR and eSBR metadata) to generate fully decoded audio data that is output from decoder 400. Typically, decoder 400 includes a storage device (accessible to subsystem 202 and stage 203) that stores formatted audio data and metadata outputted from the format remover 215 (and optionally also from subsystem 401), and stage 203 is configured to access audio data and metadata as needed during SBR and eSBR processing. The SBR processing at step 203 can be considered post-processing on the output of the main decoding subsystem 202 . Optionally, the decoder 400 also includes a post-upmix subsystem (which can apply Parametric Stereo (“PS”) tools defined in the MPEG-4 AAC standard using the PS metadata extracted by the deformatter 215) coupled and configured to perform the upmix. with respect to the output of the stage 203 for generating the fully decoded and upmixed audio output from the APU 210.

Parametric stereo is an encoding tool that represents a stereo signal using a linear downmix of the left and right channels of the stereo signal and a set of spatial parameters that describe the stereo image. Three types of spatial parameters are typically used in parametric stereo: (1) inter-channel intensity differences (IIDs), which describe intensity differences between channels; (2) inter-channel phase differences (IPD) describing phase differences between channels; and (3) inter-channel coherence (ICC), describing the coherence (or similarity) between channels. Coherence can be measured as the maximum cross-correlation versus time or phase. These three parameters usually provide a high quality stereo image reconstruction. However, the IPD parameters only specify the relative phase differences between the channels of the stereo input signal and do not specify the distribution of these phase differences between the left and right channels. Therefore, in addition, you can use the parameters of the fourth type, describing the overall phase shift or overall phase difference (OPD). In the stereo reconstruction process, successive windowed segments of both the received downmix signal, s[n], and the decorrelated version of the received downmix, d[n], are processed together with the spatial parameters to generate the left (l _k (n)) and right (r _k (n)) reconstructed signals according to the following formula:

where H ₁₁ , H ₁₂ , H ₂₁ and H ₂₂ are determined by the stereo parameters. The signals l _k (n) and r _k (n) are finally converted back into the time domain by means of time-frequency conversion.

The control data generation subsystem 401 shown in FIG. 5 is connected and configured to detect at least one property of the encoded audio bitstream to be decoded and generate eSBR control data (which may be or include eSBR metadata of any of the types included in the encoded audio bitstreams according to other embodiments of the present of the invention) in response to at least one result of the discovery step. The eSBR control data is passed to step 203 to trigger the application of individual eSBR tools or combinations of eSBR tools upon detection of a particular property (or combination of properties) of the bitstream and/or control the application of such eSBR tools. For example, to control the execution of eSBR processing using harmonic transposition, some embodiments of the control data generation subsystem 401 would include: a music detector (e.g., a simplified version of a traditional music detector) to set the sbrPatchingMode[ch] parameter (and pass the set parameter to stage 203) in a response to detecting that the bitstream characterizes or does not characterize the music; a transient detector for setting a parameter sbrOversamplingFlag[ch] (and passing the set parameter to stage 203) in response to detecting the presence or absence of transient signals in the audio content indicated by the bitstream; and/or a pitch detector for setting parameters sbrPitchInBinsFlag[ch] and sbrPitchInBins[ch] (and passing the set parameters to stage 203) in response to detecting the pitch of the audio content indicated by the bitstream. Other aspects of the present invention are methods for decoding an audio bitstream performed using any embodiment of the decoder according to the present invention described in this paragraph and the previous paragraph.

Aspects of the present invention include an encoding or decoding method of the type that any embodiment of an APU, system, or device of the present invention is capable of performing (eg, programmed). Other aspects of the present invention include a system or device capable of (e.g., programmed to) perform any embodiment of the method of the invention, and a computer-readable medium (e.g., a disk) that stores (e.g., non-volatile) code for implementing any embodiment. method of the invention or its steps. For example, the system of the invention may be or include a general purpose programmable processor, a digital signal processor, or a microprocessor programmed using software or firmware and/or otherwise capable of performing any of a variety of operations on data, including an embodiment of a method. invention or its stages. Such a general purpose processor may be or include a computer system comprising an input device, a storage device, and processing circuitry programmed (and/or otherwise configured) to execute an embodiment of the method of the invention (or steps thereof) in response to transmitted her data.

Embodiments of the present invention may be implemented in hardware, firmware, software, or a combination thereof (eg, as a programmable logic array). Unless otherwise indicated, the algorithms or processes included as part of the present invention are not inherently specific to any particular computer or other device. In particular, various general purpose machines may be used in conjunction with programs written in accordance with the ideas set forth herein, or it may be more convenient to construct more specialized apparatus (eg, integrated circuits) to perform the required method steps. Thus, the present invention may be implemented in one or more computer programs executing on one or more programmable computer systems (for example, an implementation of any of the elements shown in Fig. 1 or encoder 100 shown in Fig. 2 (or its element) or decoder 200 shown in Fig. 3 (or element thereof), or decoder 210 shown in Fig. 4 (or element thereof), or decoder 400 shown in Fig. 5 (or element thereof) ), each of which contains at least one processor, at least one data storage system (including volatile and non-volatile memory devices and / or storage elements), at least one input device or port, and at least one output device or port . Program code is applied to input data to perform the functions described in this document and generate output information. The output information is applied to one or more output devices in a known manner.

Each such program may be implemented in any desired computer language (including machine, assembly, or high-level procedural, logical, or object-oriented programming languages) to communicate with a computer system. In either case, the language may be a compiled language or an interpreted language.

For example, when implemented by computer software instruction sequences, the various functions and steps of the embodiments of the present invention may be implemented by multi-threaded software instruction sequences running on suitable digital signal processing hardware, in which case the various apparatuses, steps, and functions of the embodiments may match parts of the software commands.

Each such computer program is preferably stored on or loaded onto storage media or a device (e.g., solid-state memory or media, or magnetic or optical media) readable by a general purpose or special purpose programmable computer to customize the configuration and operation of the computer when the storage media or device is read. computer system to perform the procedures in this document. The system of the invention may also be implemented as a computer-readable storage medium equipped with (i.e., storing) a computer program, where the storage medium so equipped causes the computer system to operate in a specific and predetermined manner to perform the functions described herein.

A number of embodiments of the present invention have been described. However, it should be understood that various modifications can be made without departing from the scope of the claims. In light of the above ideas, numerous modifications and variations of the present invention are possible. For example, to facilitate efficient implementations, phase shifts can be used in combination with analysis using analysis and synthesis complex QMF filters. The analysis filter bank is responsible for filtering the low band signal in the time domain generated by the main decoder into a plurality of subbands (eg, QMF subbands). The synthesis filter bank is responsible for combining the reconstructed high band obtained with the selected HFR technique (as indicated by the received sbrPatchingMode parameter) with the decoded low band to produce a wideband audio output signal. A given filterbank implementation operating in a particular sampling rate mode, such as conventional two-speed operation or downsampling SBR mode, should not, however, have bitstream dependent phase shifts. The QMF blocks used in SBR are a complex exponential extension of the theory of cosine modulated filter banks. It can be shown that when expanding a cosine-modulated filterbank with complex-exponential modulation, the restrictions associated with the exclusion of spurious signals become irrelevant. So, for QMF SBR blocks, both the analysis filters, h _k (n), and the synthesis filters, f _k (n), can be defined as:

where p ₀ (n) is a real-valued symmetric or asymmetric prototype filter (typically a low-pass prototype filter), M denotes the number of channels, and N is the order of the prototype filter. The number of channels used in the analysis filter bank may differ from the number of channels used in the synthesis filter bank. For example, an analysis filter bank may have 32 channels, and a synthesis filter bank may have 64 channels. When the synthesis filter bank is operating in downsampling mode, the synthesis filter bank can only have 32 channels. Because the subband samples from the filterbank are complex-valued, an additional, possibly channel dependent, phase shifting step can be applied to the analysis filterbank. These additional phase shifts must be compensated before the synthesis filter bank. Although the phase shift components can in principle be arbitrary values without degrading the QMF analysis/synthesis chain, they can also be limited to certain values for consistency checking. The SBR signal will be affected by the choice of phase factors, and the low pass signal coming from the main decoder will not be affected. The sound quality of the output signal is not adversely affected.

The prototype filter coefficients, p ₀ (n), can be defined with length L, 640, as shown in Table 4 below.

Table 4

The prototype filter, p ₀ (n), can also be obtained from Table 4 using one or more mathematical operations such as rounding, downsampling, interpolation, and decimation.

Although the setting of the SBR related control information is generally independent of the details of the transposition (as discussed previously), in some embodiments, some control data elements may be transmitted concurrently in the eSBR extension container (bs_extension_id ==EXTENSION_ID_ESBR) in order to elevate the quality of the recovered signal. Some of the simulcast elements may include noise floor data (e.g., noise floor scaling factors and a parameter indicating the direction, in either frequency or time direction, of the difference coding for each noise floor), inverse filtering data (e.g., a parameter indicating the inverse mode filtering selected from no inverse filtering, low inverse filtering, intermediate inverse filtering, and strong inverse filtering) and omitted harmonic data (for example, a parameter indicating whether a sine wave should be added to a particular high band to be restored). All of these elements rely on the encoder's synthetic emulation of the decoder's transposer, and therefore, if properly tuned for the selected transposer, can improve the quality of the reconstructed signal.

In particular, in some embodiments, the skipped harmonics data and the inverse filtering control data are transmitted in the eSBR extension container (along with other bitstream parameters from Table 3) and tuned for the eSBR harmonic transposer. The additional bit rate required to carry these two classes of metadata for the eSBR harmonic transposer is relatively low. Therefore, sending tuned harmonic skip data and/or reverse filtering control data in an eSBR extension container will improve the quality of the transposer audio with only a minimal impact on the bit rate. For backward compatibility with legacy decoders, the parameters configured for the SBR spectral transfer operation may also be sent in the bitstream as part of the SBR control data using implicit or explicit signaling.

The complexity of a decoder with SBR enhancements as described in this application should be limited so as not to significantly increase the overall computational complexity of the implementation. Preferably, the PCU (MOP) for the SBR object type, when using the eSBR tool, is less than or equal to 4.5, and the RCU for the SBR object type, when using the eSBR tool, is less than or equal to 3. Approximate processing power is given in units of Processor Complexity. Unit, PCU) expressed in integer MOPS. Approximate RAM usage is given in RAM Complexity Units (RCUs), expressed as integer kilowords (1000 words). The RCU numbers do not include work buffers that may be shared between different entities and/or channels. Also, PCU is proportional to the sample rate. PCU values are given in MOPS (millions of operations per second) per channel, while RCU values are given in kilowords per channel.

Compressed data such as HE-AAC encoded audio that can be decoded by other decoder configurations requires special attention. In this case, decoding can be performed in backwards compatible (AAC only) as well as in enhanced mode (AAC+SBR). If the compressed data allows both backward compatible and extended decoding, and if the decoder operates in enhanced mode such that it uses a post processor that introduces some additional delay (for example, the SBR post processor in HE-AAC), then it is necessary to ensure that this additional time delay , introduced with respect to the backward compatibility mode, as described by the corresponding value of n, was taken into account when presenting the alignment block. To ensure correct processing of align timestamps (so that audio stays in sync with other media), the additional delay introduced by post-processing, given in number of samples (per audio channel) at the output sample rate, is 3010 if the decoder mode of operation includes SBR enhancements (in including eSBR) as described in this application. Therefore, for an audio combiner, the alignment time is applicable to the 3011th audio sample in the combiner if the decoder mode of operation includes SBR enhancements as described in this application.

To improve the subjective quality of audio content with harmonic frequency structure and strong tonal characteristics, particularly at low bit rates, SBR enhancements should be enabled. The values of the corresponding bitstream element (ie esbr_data()) driving these tools can be determined in the encoder by applying a signal dependent classification mechanism.

Generally, using the harmonic patching mode (sbrPatchingMode == 0) is preferred for encoding music signals with very low bit rates, where the underlying codec can be severely limited in audio bandwidth. This is especially true if these signals have a pronounced harmonic structure. In contrast, the use of the conventional SBR insertion method is preferred for speech and mixed signals because it provides better preservation of speech temporal structure.

To improve the performance of the MPEG-4 SBR transposer, you can activate a pre-processing step (bs_sbr_preprocessing == 1) that eliminates the introduction of violations of the homogeneity of the spectrum of the signal going to the subsequent envelope corrector. This tool is useful for signal types in which the rough envelope of the low range signal spectrum used for high frequency reconstruction exhibits large level variations.

To improve the transient response of the SBR harmonic insert (sbrPatchingMode == 0), you can apply signal-adaptive frequency domain oversampling (sbrOversamplingFlag == 1). Because signal-adaptive frequency domain resampling increases the computational complexity of the transposer but only benefits frames containing transient signals, the use of this tool is controlled by a bitstream element that is transmitted once per frame and on an independent SBR channel.

The usual recommended bit rate settings for HE-AACv2 with SBR enhancements (i.e. including eSBR instrument harmonic transposer) are 20-32 kbps for stereo audio content at 44.1 kHz or 48 kHz sample rates. The relative subjective gain as SBR improvements increases towards the lower bit rate bound, and a properly designed decoder allows this range to be extended to even lower bit rates. The above bit rates are only guidelines and can be adapted to specific operational requirements.

A decoder operating in the proposed enhanced SBR mode typically needs to be able to switch between legacy and enhanced SBR insertion. Therefore, a delay may be introduced, which may be as long as the duration of one basic frame of audio, depending on the settings of the decoder. Typically, the delay will be the same for both traditional and enhanced SBR inserts.

It is to be understood that within the scope of the appended claims, the present invention may be practiced otherwise than as specifically described herein. Any references contained in the following claims are for illustration only and should not be used in any way to interpret or limit the claims.

Various aspects of the present invention can be understood from the following numbered exemplary embodiments (NEPs).

APVO 1. A method for reconstructing the high frequencies of an audio signal, including:

receiving an encoded audio bitstream, the encoded audio bitstream contains audio data representing a low-band portion of an audio signal and high frequency reconstruction metadata;

decoding the audio data to generate a decoded low band audio signal;

extraction of high frequency reconstruction metadata from the encoded audio bitstream, this high frequency reconstruction metadata contains operating parameters for the high frequency reconstruction process, the operating parameters include an insertion mode parameter located in a backwards compatible encoded audio bitstream extension container, wherein the first value of the insertion mode parameter is indicates spectral transfer, and the second value of the insert mode parameter indicates harmonic transposition using frequency band stretching by the phase vocoder;

filtering the decoded low band audio signal to generate a filtered low band audio signal;

restoring the high-band portion of the audio signal using the filtered low-band audio signal and high-frequency reconstruction metadata, whereby the restoration includes spectral transfer if the insert mode parameter has a first value, and the restoration includes harmonic transposition using frequency band stretching by the phase vocoder if the insert mode parameter has a second value; And

combining the filtered low band audio signal with the reconstructed high band portion to form a wideband audio signal,

wherein the filtering, reconstruction, and combining are performed as a post-processing operation with a delay of 3010 samples per audio channel or less.

TFAP 2. The method according to TFRP 1, wherein the encoded audio bitstream further includes a padding element with an identifier indicating the start of the padding element and padding data after the identifier, where the padding data contains a backwards compatible extension container.

TFRP 3. The method according to TFRP 2, wherein the identifier is a three-bit unsigned integer of which the most significant bit having a value of 0×6 is first transmitted.

TPOP 4. The method according to TPOP 2 or TPOP 3, wherein the padding data contains extension payload data, the extension payload contains spectral band copy extension data, wherein the extension payload data is identified by a four-bit unsigned integer of which the most significant is transmitted first. the significant bit having the value "1101" or "1110", and optionally

wherein the spectral band copy extension data contains:

optional spectral band copy header,

spectral band copy data after header and

a spectral band copy extension element after the spectral band copy data, and wherein a flag is included in the spectral band copy extension element.

TPTP 5. The method according to any of TPTP 1-4, wherein the high frequency reconstruction metadata contains envelope scale factors, noise threshold scale factors, time/frequency grid information, or a parameter indicating a crossover frequency.

6. The method according to any one of TFEPs 1 to 5, wherein the backward compatible extension container further comprises a flag indicating whether additional pre-processing is used to avoid discontinuities in the form of a high-band spectrum envelope when the insert mode parameter is equal to a first value. , where the first flag value enables additional preprocessing, and the second flag value disables additional preprocessing.

TFEP 7. The method according to TFEP 6 wherein the additional pre-processing includes calculating a pre-gain curve using a linear prediction filter coefficient.

8. The method according to any one of TFEPs 1 to 5, wherein the backward compatible extension container further comprises a flag indicating whether to apply signal-adaptive frequency domain resampling when the insert mode parameter is equal to a second value, wherein the first value of the flag includes adaptive signal-adaptive frequency domain resampling, and the second flag value disables signal-adaptive frequency domain resampling.

TFRP 9: A method according to TFRP 8 in which signal-adaptive frequency domain resampling is applied only to frames containing a transient signal.

TFEP 10. The method according to any of the previous TFEP, wherein harmonic transposition using frequency band stretching by a phase vocoder is performed with an estimated complexity of 4.5 million operations per second or less and 3 or less memory words.

FPVO 11. A non-volatile computer-readable medium containing instructions that, when executed by a processor, perform a method according to any of VPVO 1-10.

CPOP 12. A computer program product containing instructions that, when executed by a computing device or system, cause said computing device or system to execute a method according to any of CPOPs 1-10.

PPVO 13. Sound processing unit for reconstructing high frequencies of an audio signal, comprising:

an input interface for receiving an encoded audio bitstream, this encoded audio bitstream contains audio data representing a low-band portion of an audio signal and high frequency reconstruction metadata;

a main audio decoder for decoding audio data to generate a low band decoded audio signal;

a deformatter for extracting high frequency reconstruction metadata from the encoded audio bitstream, this high frequency reconstruction metadata contains operating parameters for the high frequency reconstruction process, the operating parameters include an insert mode parameter located in a backward compatible extension container of the encoded audio bitstream, wherein the first the value of the insert mode parameter indicates spectral transfer, and the second value of the insert mode parameter indicates harmonic transposition using phase vocoder stretching;

an analysis filter unit for filtering the decoded low band audio signal to generate a filtered low band audio signal;

a high-frequency restoration device for reconstructing the high-band portion of an audio signal using a filtered low-band audio signal and high-frequency reconstruction metadata, wherein the reconstruction includes spectral transfer if the insert mode parameter is set to a first value, and the reconstruction includes harmonic transposition using range stretching frequencies by the phase vocoder if the insert mode parameter has a second value; And

a synthesis filter unit for combining the filtered low-band audio signal with the restored high-band portion to form a broadband audio signal,

wherein the analysis filter bank, the high frequency restorer, and the synthesis filter bank are executed in the post-processor with a delay of 3010 samples per audio channel or less.

FPVO 14. An audio processing unit according to FPVO 13 in which harmonic transposition using frequency band stretching by a phase vocoder is performed with an estimated complexity of 4.5 million operations per second or less and 3 kwords of memory or less.

Claims (36)

1. A method for performing the reconstruction of high frequencies of an audio signal, including: receiving an encoded audio bitstream, the encoded audio bitstream contains audio data representing a low-band portion of an audio signal and high frequency reconstruction metadata; decoding the audio data to generate a decoded low band audio signal; extraction of high frequency reconstruction metadata from the encoded audio bitstream, this high frequency reconstruction metadata contains operating parameters for the high frequency reconstruction process, the operating parameters include an insertion mode parameter located in a backwards compatible encoded audio bitstream extension container, wherein the first value of the insertion mode parameter is indicates spectral transfer, and the second value of the insert mode parameter indicates harmonic transposition using frequency band stretching by the phase vocoder; filtering the decoded low band audio signal to generate a filtered low band audio signal; restoring the high-band portion of the audio signal using the filtered low-band audio signal and high-frequency reconstruction metadata, whereby the restoration includes spectral transfer if the insert mode parameter has a first value, and the restoration includes harmonic transposition using frequency band stretching by the phase vocoder if the insert mode parameter has a second value; And combining the filtered low band audio signal with the reconstructed high band portion to form a wideband audio signal, while filtering, restoring and combining is performed as a post-processing operation with a delay of 3010 samples per audio channel. 2. The method of claim 1, wherein the encoded audio bitstream further includes a padding element with an identifier indicating the start of the padding element and padding data after the identifier, where the padding data contains a backward compatible extension container. 3. The method according to claim 2, characterized in that the identifier is a three-bit unsigned integer, in which the most significant bit having a value of 0x6 is transmitted first. 4. The method according to claim 2 or 3, wherein the padding data contains extension payload data, wherein the extension payload contains spectral band copy extension data, and the extension payload data is identified with a four-bit unsigned integer of which is first transmitted the most significant bit having the value "1101" or "1110", and optionally wherein the spectral band copy extension data contains: optional spectral band copy header, spectral band copy data after header and a spectral band copy extension element after the spectral band copy data, and wherein a flag is included in the spectral band copy extension element. 5. The method according to any one of paragraphs. 1-4, characterized in that the high-frequency reconstruction metadata contains envelope scale factors, noise threshold scale factors, time/frequency grid information, or a parameter indicating a crossover frequency. 6. The method according to any one of paragraphs. 1-5, characterized in that the backward compatible extension container further comprises a flag indicating whether additional pre-processing is used to avoid discontinuities in the form of a spectrum envelope related to the upper range portion when the insert mode parameter is equal to a first value, wherein the first value flag enables additional preprocessing, and the second value of the flag disables additional preprocessing. 7. The method of claim 6, wherein the additional pre-processing comprises calculating a pre-gain curve using a linear prediction filter coefficient. 8. The method according to any one of paragraphs. 1-5, characterized in that the backward compatible extension container further comprises a flag indicating whether it is necessary to apply signal-adaptive frequency domain resampling when the insert mode parameter is equal to the second value, wherein the first value of the flag enables signal-adaptive frequency domain resampling. , and the second flag value disables signal-adaptive resampling in the frequency domain. 9. The method of claim 8, wherein signal-adaptive frequency domain resampling is applied only to frames containing a transient signal. 10. A method according to any one of the preceding claims, wherein the harmonic transposition using frequency band stretching by a phase vocoder is performed with an estimated complexity of 4.5 million operations per second or less. 11. The method according to any of the previous paragraphs, characterized in that filtering the lower band decoded audio signal to generate the lower band filtered audio signal, comprising filtering the lower band decoded audio signal into a plurality of subbands using a QMF complex analysis filter bank; And combining the filtered low-band audio signal with the reconstructed high-band portion to form a wideband audio signal involves using a complex QMF synthesis filter bank. 12. The method according to claim 11, characterized in that the analyzing filters h _k (n) of the block of complex analyzing QMF filters and the synthesis filters f _k (n) of the block of complex synthesizing QMF filters are determined by the formula:

where p ₀ (n) is a real-valued prototype filter, M denotes the number of channels, and N is the order of the prototype filter. 13. Sound processing unit for reconstructing the high frequencies of an audio signal, comprising: an input interface for receiving an encoded audio bitstream, the encoded audio bitstream comprising audio data representing a low-band portion of an audio signal and high frequency reconstruction metadata; a main audio decoder for decoding audio data to generate a low band decoded audio signal; a deformatter for extracting high frequency reconstruction metadata from the encoded audio bitstream, this high frequency reconstruction metadata contains operating parameters for the high frequency reconstruction process, the operating parameters include an insert mode parameter located in a backward compatible extension container of the encoded audio bitstream, wherein the first the value of the insert mode parameter indicates spectral transfer, and the second value of the insert mode parameter indicates harmonic transposition using phase vocoder stretching; an analysis filter unit for filtering the decoded low band audio signal to generate a filtered low band audio signal; a high-frequency restoration device for reconstructing the high-band portion of an audio signal using a filtered low-band audio signal and high-frequency reconstruction metadata, wherein the reconstruction includes spectral transfer if the insert mode parameter is set to a first value, and the reconstruction includes harmonic transposition using range stretching frequencies by the phase vocoder if the insert mode parameter has a second value; And a synthesis filter unit for combining the filtered low-band audio signal with the restored high-band portion to form a broadband audio signal, wherein the analysis filter bank, the high frequency restorer, and the synthesis filter bank are executed in the post-processor with a delay of 3010 samples per audio channel or less. 14. The sound processing unit according to claim 13, characterized in that the harmonic transposition using frequency band stretching by the phase vocoder is performed with an estimated complexity of 4.5 million or less operations per second.

Images