JP2022543083A - Encoding and Decoding IVAS Bitstreams - Google Patents

Abstract

encoding/decoding an immersive voice audio service (IVAS) bitstream encoding/decoding an encoding mode indicator in a common header (CH) section of the IVAS bitstream; encoding/decoding a mode header or tool header in a header (TH) section, where the TH section follows a CH section; and a metadata payload in the Metadata Payload (MDP) section of the bitstream. with the MDP section following the CH section and encoding/decoding the enhanced voice service (EVS) payload in the EVS payload (EP) section of the bitstream. where the EP section follows the CH section; on the encoder side, storing or streaming the encoded bitstream; and on the decoder side, encoding mode, tool header, EVS payload, and metadata. controlling audio decoders or storing representations thereof based on the payloads.
[Selection drawing] Fig. 2

Description

[Cross reference to related applications]
This application is based on U.S. Provisional Application No. 62/881,541, filed Aug. 1, 2019; U.S. Provisional Application No. 62/927,894, filed Oct. 30, 2019; No. 63/037,721 filed on July 11, 2020 and US Provisional Application No. 63/057,666 filed July 28, 2020. The entire disclosures of these US provisional applications are hereby incorporated by reference.

This disclosure relates generally to encoding and decoding audio bitstreams.

Standards development for audio and video encoder/decoders (“codecs”) has recently focused on developing codecs for immersive voice and audio services (IVAS). IVAS is expected to support various audio service capabilities. These audio service capabilities include, but are not limited to, mono-to-stereo upmixing and fully immersive audio encoding, decoding and rendering. IVAS is intended to be supported by a wide range of devices, endpoints and network nodes. These wide range of devices, endpoints and networks include mobile phones and smart phones, electronic tablets, personal computers, conference phones, conference rooms, virtual reality (VR) devices and augmented reality (AR) devices. , home theater devices, and other suitable devices. These devices, endpoints and network nodes can have various acoustic interfaces for sound capture and rendering.

Embodiments are disclosed for encoding and decoding IVAS bitstreams.

In some embodiments, a method of generating a bitstream of an audio signal is using an immersive audio audio service (IVAS) encoder to determine an encoding mode indicator or an encoding tool indicator, The encoding mode indicator or the encoding tool indicator indicates an encoding mode or encoding tool of the audio signal, and the IVAS encoder is used to convert the encoding mode indicator or the encoding tool indicator into an IVAS bitstream. using the IVAS encoder to obtain a mode header or a tool header; and using the IVAS encoder to convert the mode header or the tool header to the encoding within a Tool Header (TH) section of an IVAS bitstream, the TH section following the CH section; and using the IVAS encoder to encode a metadata payload including spatial metadata. and encoding the metadata payload into a metadata payload (MDP) section of the IVAS bitstream using the IVAS encoder, the MDP section following the CH section. and using the IVAS encoder to determine enhanced voice service (EVS) payloads, the EVS payloads including EVS coded bits for each channel or downmix channel of the audio signal. and encoding the EVS payload into an EVS payload (EP) section of the IVAS bitstream using the IVAS encoder, the EP section following the CH section.

In some embodiments, the IVAS bitstream is stored on non-transitory computer-readable media. In another embodiment, the IVAS bitstream is streamed to a downstream device, and the encoding mode or encoding tool indicator, the mode header or tool header, the metadata payload and the EVS payload are streamed to the downstream device or The CH, TH, MDP and EP sections of the IVAS bitstream are respectively extracted and decoded for use in reconstructing the audio signal in another device.

In some implementations, a method of decoding a bitstream of an audio signal includes using an immersive voice audio service (IVAS) decoder to detect a coding mode indicator or extracting and decoding a coding tool indicator, wherein the coding mode indicator or the coding tool indicator indicates the coding mode or coding tool of the audio signal; and using the IVAS decoder. to extract and decode a mode header or tool header in the Tool Header (TH) section of the IVAS bitstream, the TH section following the CH section; extracting and decoding a metadata payload from a metadata payload (MDP) section of the IVAS bitstream using including metadata; and using the IVAS decoder to extract and decode an enhanced voice service (EVS) payload from an EVS payload (EP) section of the IVAS bitstream, the EP section follows the CH section, the EVS payload containing EVS coded bits for each channel or downmix channel of the audio signal.

In some embodiments, a downstream device or an audio decoder of a downstream device for use in reconstructing the audio signal in another device detects the coding mode indicator or the coding tool indicator, the mode header or the Controlled based on the tool header, the EVS payload, and the metadata payload. In another embodiment, a representation of said encoding mode indicator or said encoding tool indicator, said mode header or said tool header, said EVS payload, and said metadata payload is stored on a non-transitory computer readable medium. do.

In some implementations, the bitrate of each EVS-encoded channel or each downmixed channel is determined by EVS total available bits, a SPAR bitrate distribution control table and a bitrate distribution algorithm.

In some embodiments, the CH is a multi-bit data structure, one value of the multi-bit data structure corresponds to a Spatial Reconstruction (SPAR) coding mode, and the other value of the data structure is Values correspond to other encoding modes.

In some embodiments, the method comprises storing an index offset for calculating a row index of a Spatial Reconstruction (SPAR) bitrate distribution control table in the TH section of the IVAS bitstream, respectively. further comprising reading from

In some implementations, the above method converts a quantization strategy indicator, a bitstream encoding strategy indicator, and quantized and encoded real and imaginary parts of a set of coefficients into the IVAS bitstream, respectively. Further comprising storing in or reading from the MDP section.

In some embodiments, the set of coefficients includes prediction coefficients, direct coefficients, diagonal real coefficients and lower triangular complex coefficients.

In some embodiments, the prediction coefficients are variable bit length based entropy coding, and the direct coefficients, the diagonal real coefficients and the lower triangular complex coefficients are variable bit length based downmix construction and entropy coding. is long.

In some embodiments, said quantization strategy indicator is a multi-bit data structure that indicates a quantization strategy.

In some embodiments, the bitstream coding strategy indicator is multi-bit indicating the number of spatial metadata bands and non-differential entropy coding scheme or time-differential entropy coding scheme. A data structure.

In some embodiments, said quantization of said coefficients follows an EVS bitrate distribution control strategy comprising metadata quantization and EVS bitrate distribution.

In some embodiments, the above method stores EVS payloads of EVS instances in the EP section of the IVAS bitstream, respectively, according to 3rd Generation Partnership Project (3GPP) Technical Specification (TS) 26.445. or reading from the EP section of the IVAS bitstream.

In some embodiments, the method comprises: determining a bitrate from the IVAS bitstream; reading an index offset from a Spatial Reconstruction (SPAR) Tool Header (TH) section of the IVAS bitstream; determining a table row index of the SPAR bitrate distribution control table using the offset; reading quantization strategy bits and encoding strategy bits from a metadata payload (MDP) section in the IVAS bitstream; dequantizing SPAR spatial metadata in said MDP section of said IVAS bitstream based on quantization strategy bits and said encoding strategy bits; total available EVS bits; and a SPAR bitrate distribution control table. and reading EVS coded bits from the EP section of the IVAS bitstream based on the EVS bitrate, using decoding the EVS bits; decoding the spatial metadata; and using the decoded EVS bits and the decoded spatial metadata, a first order Ambisonics (FoA ) generating an output.

Other embodiments disclosed herein relate to systems, apparatus and computer-readable media. The details of the disclosed embodiments are set forth in the accompanying drawings and the description below. Other features, objects and advantages are apparent from the following description, drawings and claims.

Certain implementations disclosed herein provide one or more of the following advantages. The disclosed IVAS bitstream format is an efficient and robust bitstream format that supports various audio service capabilities. These audio service capabilities include, but are not limited to, mono-to-stereo upmix and fully immersive audio encoding, decoding and rendering. In some implementations, the IVAS bitstream format supports complex advance coupling (CACPL) for analyzing and downmixing stereo audio signals. In another embodiment, the IVAS bitstream format supports spatial reconstruction (SPAR) for analyzing and downmixing first order Ambisonics (FoA) audio signals.

In the drawings, a specific arrangement or ordering of graphical elements, such as elements representing devices, units, instruction blocks, and data elements, is shown for ease of explanation. However, the specific ordering or arrangement of these graphical elements in the drawings suggests that some specific order or sequence of processing is required, or that a separation of processes is required. It should be understood by those skilled in the art that it is not. Furthermore, the inclusion of a drawing element in a drawing does not imply that such element is required in all embodiments nor does it imply that the feature represented by such element It is not meant to imply that a part cannot be included in or combined with other elements in some implementations.

Further, when the drawings use connecting elements, such as solid or dashed lines or arrows, to indicate a connection, relationship, or association between two or more other diagrammatic elements, such connecting elements are Its absence does not imply that a connection, relationship or association cannot exist. In other words, some connections, relationships or associations between elements are not shown in the drawings so as not to obscure the present disclosure. Additionally, for ease of illustration, single connecting elements are used to represent multiple connections, relationships or associations between elements. For example, where connecting elements represent communication of signals, data, or instructions, such elements represent one or more signal paths for carrying out the communication, as appropriate. It should be.

1 illustrates an IVAS system according to one embodiment; FIG.

1 is a block diagram of a system for encoding and decoding an IVAS bitstream, according to one embodiment; FIG.

1 is a block diagram of a FoA coder/decoder (“codec”) that encodes and decodes an IVAS bitstream in FoA format, according to one embodiment; FIG.

FIG. 4 is a flow diagram of an IVAS encoding process according to one embodiment;

FIG. 4 is a flow diagram of an IVAS encoding process using an alternative IVAS format, according to one embodiment;

FIG. 4 is a flow diagram of an IVAS decoding process according to one embodiment;

FIG. 4 is a flow diagram of an IVAS decoding process using an alternative IVAS format, according to one embodiment;

FIG. 4 is a flow diagram of an IVAS SPAR encoding process according to one embodiment;

FIG. 4 is a flow diagram of an IVAS SPAR decoding process according to one embodiment;

1 is a block diagram of an exemplary device architecture according to one embodiment; FIG.

The same reference numbers used in different drawings indicate similar elements.

In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various described embodiments. It will be apparent to those skilled in the art that the various described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments. Several features are described below that can each be used independently of each other or in any combination with other features.

Nomenclature As used herein, the terms "include"/"include" and variations thereof mean "including, but not limited to" non-limiting ( open-ended) term. The terms "or"/"or" should be interpreted as "and/or" unless the context clearly indicates otherwise. The term "based on" should be interpreted as "based at least in part on." The terms "one exemplary embodiment" and "one exemplary embodiment" should be interpreted as "at least one exemplary embodiment". The term "another embodiment" should be interpreted as "at least one alternative embodiment". The terms "determined,""determine,""determine,""determine,""obtain,""receive,""calculate,""calculate,""estimate,"" should be construed as 'predicting' or 'deriving'. Additionally, in the following description and claims, unless otherwise defined, all technical and scientific terms used herein are commonly understood by one of ordinary skill in the art to which this disclosure pertains. have the same meaning as

IVAS System Overview FIG. 1 illustrates an IVAS system 100 according to one or more embodiments. In some implementations, the various devices are, for example, public switched telephone network (PSTN) devices or public land mobile network (PLMN) devices denoted by PSTN/other PLMN 104. Communicate through a call server 102 configured to receive audio signals. The IVAS system 100 supports legacy devices 106 that render and capture audio in mono only. This legacy device supports enhanced voice service (EVS), multi-rate wideband (AMR-WB) and adaptive multi-rate narrowband (AMR-NB). Including but not limited to devices. The IVAS system 100 also supports user equipment (UE) 108, 114 that captures and renders stereo audio signals, or UE 110 that captures monophonic signals and binaurally renders them into multi-channel signals. The IVAS system 100 also supports immersive and stereo signals captured and rendered by the video conferencing room systems 116, 118, respectively. The IVAS system 100 provides stereo capture and immersive rendering of stereo audio signals for home theater systems, and mono capture and immersive rendering of audio signals for virtual reality (VR) gear 122 and immersive content ingest 124. It also supports rendering.

Exemplary IVAS Encoding/Decoding System FIG. 2 is a block diagram of a system 200 for encoding and decoding IVAS bitstreams, according to one or more implementations. For encoding, the IVAS encoder includes a spatial analysis downmix unit 202 that receives audio data 201 . This audio data includes mono signals, stereo signals, binaural signals, spatial audio signals (e.g., multi-channel spatial audio objects), FoA, higher order Ambisonics (HoA) and any other audio data. , but not limited to these. In some implementations, spatial analysis downmix unit 202 implements CACPL to analyze/downmix stereo audio signals and/or SPAR to analyze/downmix FoA audio signals. In other embodiments, spatial analysis downmix unit 202 implements other formats.

The output of spatial analysis downmix unit 202 includes spatial metadata and 1-4 channels of audio. The spatial metadata is input to quantization entropy encoding unit 203 which quantizes and entropy encodes the spatial data. In some implementations, quantization can include fine quantization strategies, medium quantization strategies, coarse quantization strategies, and very coarse quantization strategies, and entropy coding can be Huffman coding or arithmetic Encoding can be included. Enhanced voice service (EVS) encoding unit 206 encodes 1-4 channels of audio into one or more EVS bitstreams.

In some implementations, EVS encoding unit 206 complies with 3GPP TS26.445 and supports a wide range of functions, such as narrowband (EVS-NB) voice service and wideband (EVS-WB) voice service quality enhancement and Coding efficiency, improved quality using ultra-wideband (EVS-SWB) audio, improved mixed content and music quality in conversational applications, robustness against packet loss and delay jitter, and backwards to AMR-WB codecs Provide compatibility, etc. In some implementations, EVS encoding unit 206 is either a speech coder that encodes the speech signal at a bitrate specified based on mode/bitrate control 207, or a perceptual coder that encodes the audio signal. a preprocessing mode selection unit for selecting the In some implementations, the speech encoder performs enhanced algebraic code-excited linear prediction (ACELP) with LP-based modes specialized for different speech classes. It is a modified form. In some implementations, the audio encoder is a modified discrete cosine transform (MDCT) encoder with enhanced efficiency at low latency/low bitrates, and a seamless transition between speech and audio encoders. Designed for reliable switching.

In some implementations, the IVAS decoder comprises a quantized entropy decoding unit 204 configured to recover spatial metadata and an EVS decoder (singular or plural) and The recovered spatial metadata and audio signal are input to spatial synthesis/rendering unit 209 , which uses the spatial metadata to render the audio signal for playback on various audio systems 210 . composite/render.

Exemplary IVAS/SPAR Codec FIG. 3 is a block diagram of a FoA codec 300 for encoding and decoding FoA in SPAR format, according to some implementations. FoA codec 300 includes SPAR FoA encoder 301 , EVS encoder 305 , SPAR FoA decoder 306 and EVS decoder 307 . FoA codec 300 transforms the FoA input signal into a set of downmix channels and parameters that are used to regenerate the input signal at decoders 306,307. The downmix signal can vary between 1-channel to 4-channel and the parameters include prediction coefficient (PR), cross-prediction coefficient (C) and decorrelation coefficient (P). SPAR is a process used to reconstruct an audio signal from a downmix of the audio signal using the PR, C and P parameters, as described in more detail below. Please note.

ここで、ｆは、Ｘチャネル、Ｙチャネル、ＺチャネルのうちのいくつかをＷチャネルにミックスすることを可能にする定数（例えば０．５）である。ｐｒ_ｙ、ｐｒ_ｘおよびｐｒ_ｚは、予測（ＰＲ）係数である。パッシブＷでは、ｆ＝０であり、そのため、Ｘチャネル、Ｙチャネル、ＺチャネルのＷチャネルへのミックスは行われない。 Note that the exemplary implementation shown in FIG. 3 assumes a passive W channel and depicts a nominal two-channel downmix in which the W channel is sent unchanged to decoder 306 along with a single prediction channel Y′. Please note. In other implementations, W may be the active channel. The active W channel allows some mixing of the X, Y and Z channels into the W channel as follows.

where f is a constant (eg 0.5) that allows some of the X, Y and Z channels to be mixed into the W channel. pr _y , pr _x and pr _z are the prediction (PR) coefficients. In passive W, f=0, so there is no mixing of the X, Y, Z channels into the W channel.

As described in more detail below, the C coefficients allow some of the X and Z channels to be reconstructed from Y′, while the remaining channels are , is reconstructed by decorrelating the W channel.

In some implementations, SPAR FoA encoder 301 includes passive/active predictor unit 302 , remix unit 303 and extraction/downmix selection unit 304 . The passive/active predictor receives the FoA channel in 4-channel B format (W, Y, Z, X) and calculates the predicted channel (W or W', Y', Z', X'). The W channel is an omnidirectional polar pattern containing all sounds in a sphere coming from all directions at equal gain and phase, and the X is a forward-pointing figure-8 bidirectional polar pattern. , Y is a figure 8 bidirectional polar pattern pointing left, and Z is a figure 8 bidirectional polar pattern pointing up.

Extraction/downmix selection unit 304 extracts SPAR FoA metadata from the metadata payload section of the IVAS bitstream, as described in more detail below. A passive/active predictor unit 302 and a remix unit 303 use the SPAR FoA metadata to generate remixed FoA channels (W or W', A', B', C'), and these FoA channels is input to EVS encoder 305 and encoded into an EVS bitstream, which is encapsulated within an IVAS bitstream that is sent to decoder 306 . Note that in this example the Ambisonic B format channels are arranged in AmbiX format. However, other formats such as the Furse-Malham (FuMa) format (W, X, Y, Z) can be used as well.

Referring to SPAR FoA decoder 306, the EVS bitstream is decoded by EVS decoder 307, resulting in N (eg, N=4) downmix channels. In some implementations, SPAR FoA decoder 306 performs the inverse of the operations performed by SPAR encoder 301 . For example, remixed FoA channels (W or W', A', B', C') are recovered from N downmix channels using SPAR FoA spatial metadata. The remixed SPAR FoA channel is input to an inverse mixer 311 to recover the predicted SPAR FoA channel (W or W', Y', Z', X'). The predicted SPAR FoA channels are then input to an inverse predictor 312 to recover the original unmixed SPAR FoA channels (W, Y, Z, X). In this two-channel example, decorrelator blocks 309a(dec ₁ ) . . . Note that 309n(dec _D ) is used to generate a decorrelated version of the W channel using either the time domain decorrelator or the frequency domain decorrelator. The decorrelated channels are used in combination with SPAR FoA metadata to completely or parametrically reconstruct the X and Z channels.

In some implementations, depending on the number of downmix channels, one of the FoA inputs (W channel) is sent intact to the SPAR FoA decoder 306 while the other channels (Y, Z, and X) are transmitted to the SPAR FoA decoder 306 as residuals or completely parametrically. PR coefficients that remain the same regardless of the number N of downmix channels are used to minimize the predictable energy in the residual downmix channels. The C coefficients are used to further help regenerate the fully parameterized channel from the residual. Therefore, C coefficients are not needed in the one and four channel downmix cases where there is no residual channel or parameterization channel to predict. The P coefficients are used to fill in the remaining energy not accounted for by the PR and C coefficients. The number of P coefficients depends on the number N of downmix channels in each band. In some implementations, the SPAR PR coefficients (passive W only) are calculated as follows.

［１］
ここで、一例として、予測されるチャネルＹ’の予測パラメーターは、式［２］を使用して算出される。

［２］
ここで、

は、信号ＡおよびＢに対応する入力された共分散行列の要素である。同様に、Ｚ’残差チャネルおよびＸ’残差チャネルは、対応する予測パラメーターｐｒ_ｚおよびｐｒ_ｘを有する。ＰＲは、予測係数のベクトル

である。 Step one. Predict all side signals (Y, Z, X) from the main W signal using equation [1].

[1]
Here, as an example, the prediction parameters for the predicted channel Y' are calculated using equation [2].

[2]
here,

are the elements of the input covariance matrix corresponding to signals A and B; Similarly, the Z' and X' residual channels have corresponding prediction parameters pr _z and pr _x . PR is the vector of prediction coefficients

is.

［３］ Step two. Remix the W signal with the predicted (Y',Z',X') signal (most acoustically relevant to least acoustically relevant in that order). Here, "remix" means a permuted or recombined signal based on some methodology.

[3]

One implementation of remixing is the W,Y conversion of the input signal assuming that the audio cues from the left and right are more acoustically related than the front and back, and the front and back cues are more acoustically related than the top and bottom cues. ', X', Z' permutation.

［４］

［５］
ここで、ｄは、Ｗを越える余分のダウンミックスチャネル（すなわち２番目のチャネルからＮｄｍｘ番目までのチャネル）を表し、ｕは完全に再生成する必要があるチャネル（すなわち（Ｎｄｍｘ＋１）番目のチャネルから４番目までのチャネル）を表す。 Step three. Compute the covariance of the 4-channel posterior prediction and remix and downmix as shown in equations [4] and [5].

[4]

[5]
where d represents the extra downmix channels over W (i.e. the 2nd to Ndmxth channels) and u is the channel that needs to be completely regenerated (i.e. from the (Ndmx+1)th channel to up to 4th channel).

As an example of a WABC downmix with 1-4 channels, d and u represent the following channels shown in Table I.

［６］ Of primary interest in the calculation of SPAR _FoA metadata are the Rdd amount, the _Rud amount and the _Ruu amount. From the R _dd , R _ud and R _uu quantities, whether the system is able to co-predict the remainder of the fully parametric channel from the residual channel sent to the decoder to judge. In some implementations, the required extra C factor is given by the following equation.

[6]

Therefore, the C parameter has the shape (1×2) for a 3-channel downmix and the shape (2×1) for a 2-channel downmix.

［７］

［８］

［９］ Step 4. Calculate the remaining energy in the parameterized channel that must be reconstructed by the decorrelator. The residual energy Res _uu in the upmix channel is the difference between the actual energy R _uu (posterior prediction) and the regenerated co-prediction energy Reg _uu .

[7]

[8]

[9]

P is also a covariance matrix and is therefore Hermitian symmetric, so only parameters from the upper or lower triangle need to be sent to decoder 306 . The diagonal entries may be real, while the off-diagonal elements may be complex.

Exemplary Encoding/Decoding of IVAS Bitstreams As described with reference to FIGS. 2 and 3, IVAS bitstream(s) are encoded and decoded by an IVAS codec. In some implementations, the IVAS encoder determines the encoding tool indicator and the sampling rate indicator and encodes them in the common header (CH) section of the IVAS bitstream. In some implementations, the encoding tool indicator comprises a value corresponding to the encoding tool and the sampling rate indicator comprises a value indicative of the sampling rate. The IVAS encoder takes the EVS payload and encodes it into the EVS payload (EP) section of the bitstream. The EP section follows the CH section. The IVAS encoder takes the metadata payload and encodes it into the metadata payload (MDP) section of the bitstream. In some implementations, the MDP section follows the CH section. In other implementations, the MDP section follows the EP section of the bitstream, or the EP section follows the MDP section of the bitstream. In some implementations, the IVAS encoder stores the bitstream on a non-transitory computer-readable medium or streams the bitstream to downstream devices. In another embodiment, the IVAS encoder has the device architecture shown in FIG.

In some implementations, the IVAS decoder receives the IVAS bitstream and extracts and decodes the audio data encoded in the IVAS format by the IVAS encoder. The IVAS decoder extracts and decodes the encoding tool indicator and sampling rate indicator in the CH section of the IVAS bitstream. The IVAS decoder extracts and decodes the EVS payload within the EP section of the bitstream. The EP section follows the CH section. The IVAS decoder extracts and decodes the metadata payload within the MDP section of the bitstream. The MDP section follows the CH section. In other implementations, the MDP section follows the EP section of the bitstream, or the EP section follows the MDP section of the bitstream. In some implementations, the IVAS system controls the audio decoder based on the encoding tool, sampling rate, EVS payload, and metadata payload. In other embodiments, the IVAS system stores representations of the encoding tool, sampling rate, EVS payload, and metadata payload on non-transitory computer-readable media. In some implementations, the IVAS decoder has the device architecture shown in FIG.

In some implementations, the IVAS Encoding Tool Indicator is a multi-bit data structure. In another embodiment, the IVAS encoding tool indicator is a 3-bit data structure, a first value of the 3-bit data structure corresponds to a multi-mono encoding tool, and a second value of the 3-bit data structure is The value corresponds to a CACPL encoding tool and the third value of the 3-bit data structure corresponds to another encoding tool. In other embodiments, the IVAS encoding tool indicator is a 2-bit data structure that indicates 1-4 IVAS encoding tools or a 1-bit data structure that indicates 1 or 2 IVAS encoding tools. In other embodiments, the IVAS encoding tool indicator includes 3 or more bits to indicate different IVAS encoding tools.

In some implementations, the input sampling rate indicator is a multi-bit data structure that indicates various input sampling rates. In some implementations, the input sampling rate indicator is a 2-bit data structure, a first value of the 2-bit data structure indicating an 8 kHz sampling rate, and a second value of the 2-bit data structure indicating a 16 kHz sampling rate. Indicating the sampling rate, the third value of the 2-bit data structure indicates a 32 kHz sampling rate and the fourth value of the 2-bit data structure indicates a 48 kHz sampling rate. In other implementations, the input sampling rate indicator is a 1-bit data structure that indicates one or two sampling rates. In other embodiments, the input sampling rate indicator includes 3 or more bits that indicate various sampling rates.

In some embodiments, the system uses the number of EVS channels as described in the 3rd generation partnership project ( ^3GPP ) technical specification (TS) 26.445, in that order. bitrate (BR) extraction mode indicator; EVS BR data; and EVS payloads for all channels are stored in or read from the EP section of the bitstream.

In other embodiments, the system stores the EVS channel number indicator in the EP section of the bitstream or reads it from the EP section of the bitstream.

In other implementations, the system stores or reads the bitrate (BR) extraction mode indicator in the EP section of the bitstream.

In other embodiments, the system stores EVS BR data in or reads from the EP section of the bitstream.

In another embodiment, the system stores the EVS payloads of all channels in the EP section of the bitstream, as described in 3rd Generation Partnership Project (3GPP) Technical Specification (TS) 26.445, in that order. or read from the EP section of the bitstream.

In some embodiments, the IVAS system includes: a coding technique indicator; a number of bands indicator; an indicator indicating the delay configuration of the filterbank; and one or more coefficients stored in or read from the MDP section of the data stream.

In other implementations, the IVAS system stores the coding technique indicator in or reads the MDP section of the data stream.

In other embodiments, the IVAS system stores the band number indicator in or reads from the MDP section of the data stream.

In another embodiment, the IVAS system stores or reads from the MDP section of the datastream an indicator of the delay configuration of the filterbank.

In other embodiments, the IVAS system stores the quantization strategy indicator in or reads from the MDP section of the data stream.

In other implementations, the IVAS system stores or reads the entropy coder indicator in the MDP section of the data stream.

In other embodiments, the IVAS system stores the probabilistic model type indicator in or reads from the MDP section of the data stream.

In other implementations, the IVAS system stores the real coefficients in or reads the MDP section of the data stream. In other implementations, the IVAS system stores the coefficient imaginary part in or reads from the MDP section of the data stream.

In other implementations, the IVAS system stores one or more coefficients in or reads from the MDP section of the data stream.

Some examples of IVAS bitstream formats are given below.

Exemplary IVAS Bitstream Format—3 Subdivision Format In some implementations, the IVAS bitstream format includes three subdivisions as follows.

In some implementations, the parameters within each field in each subdivision and their respective bit assignments are shown below.

An advantage of the IVAS bitstream format embodiment described above is that it efficiently and compactly encodes data that supports a variety of audio service capabilities. These audio service capabilities include, but are not limited to, mono-to-stereo upmix and fully immersive audio encoding, decoding and rendering. This embodiment is also supported by a wide range of devices, endpoints and network nodes. These wide range of devices and others include mobile phones and smart phones, electronic tablets, personal computers, conference phones, conference rooms, virtual reality (VR) and augmented reality (AR) devices, home theater devices, and other suitable devices. but not limited to, each of which can have different acoustic interfaces for capturing and rendering sound. The IVAS bitstream format is extensible so that it can evolve easily with IVAS standards and technologies.

Exemplary IVAS Bitstream Format-4 Subdivision Format The following description of a further embodiment will focus on the differences between this further embodiment and the previously described embodiments. Accordingly, features common to both embodiments may be omitted from the following description and, if omitted, features of the previously described embodiments may be implemented or at least implemented in this further embodiment. It should be assumed (unless the following description requires otherwise) that it can. In addition, when a feature is taken from an embodiment disclosed below and added to a claim, that feature may not be related or closely related to other features of that embodiment.

In another embodiment, the IVAS bitstream contains four subdivisions as follows.

In some implementations, the IVAS encoder determines and encodes the encoding tool indicator into the common header (CH) section of the IVAS bitstream. The encoding tool indicator comprises a value corresponding to the encoding tool. The IVAS encoder finds the row index into the IVAS bitrate distribution control table and encodes it in the common spatial coding tool header (CTH) section of the IVAS bitstream. The CTH section follows the CH section. The IVAS encoder takes the EVS payload and encodes it into the EVS payload (EP) section of the IVAS bitstream. The EP section follows the CH section. The IVAS encoder takes the metadata payload and encodes it into the metadata payload (MDP) section of the IVAS bitstream. The MDP section follows the CH section.

In some embodiments, the EP section precedes or follows the MDP section depending on one or more parameters. In some embodiments, the one or more parameters include a backwards compatible mode of mono downmixing of multi-channel input with nominal bitrate mode, as described in 3GPP TS 26.445.

In some implementations, the IVAS system stores the IVAS bitstream on a non-transitory computer-readable medium. In other implementations, the IVAS system streams the bitstream to downstream devices. In some implementations, the IVAS encoder has the device architecture shown in FIG.

In some implementations, the IVAS decoder receives the IVAS bitstream and extracts and decodes the audio data encoded in the IVAS format by the IVAS encoder. The IVAS decoder extracts and decodes the encoding tool indicator in the CH section of the IVAS bitstream. The IVAS decoder extracts and decodes the index into the IVAS bitrate distribution control table. The IVAS decoder extracts and decodes the EVS payload within the EP section of the IVAS bitstream. The EP section follows the CH section. The IVAS decoder extracts and decodes the metadata payload in the MDP section of the IVAS bitstream. The MDP section follows the CH section.

In some implementations, the EP section precedes or follows the MDP section, depending on one or more parameters. In some embodiments, the one or more parameters include a backwards compatible mode of mono downmixing of multi-channel input with nominal bitrate mode, as described in 3GPP TS 26.445.

In some implementations, the IVAS system controls the audio decoder based on the encoding tool, the index into the IVAS bitrate distribution control table, the EVS payload, and the metadata payload. In another embodiment, the IVAS system stores representations of the encoding tool, the index into the IVAS bitrate distribution control table, the EVS payload, and the metadata payload on a non-transitory computer-readable medium. In some implementations, the IVAS decoder has the device architecture shown in FIG.

Metadata Payload (MDP):
An advantage of the IVAS bitrate distribution control table is that it records information about the spatial coding mode such that the information about the spatial coding mode need not be included in the MDP section.

によって与えることができる。ここで、Ｎ（例えば、Ｎ＝４）は、符号化するのに必要とされるオーディオダウンミックスチャネルの数であり、ＥＶＳ_{ＢＲ（ｉ）}は、ｉ番目のオーディオダウンミックスチャネルの算出されたＥＶＳビットレートであり、ｓｔｒｉｄｅ_ｓｅｃｓは、秒を単位とする入力ストライド長である。 EVS Payload (EP):
This section of the payload contains EVS coded bits for one or more audio downmix channels. In some implementations, the total number of bits in this section is

can be given by where N (eg, N=4) is the number of audio downmix channels needed to encode and EVS _BR(i) is the calculated EVS of the i-th audio downmix channel is the bitrate and stride _secs is the input stride length in seconds.

In some implementations, each table entry in the IVAS bitrate distribution control table has sufficient information to extract the bitrate of each EVS instance from all bits allocated for EVS. This structure provides the advantage that no additional header information is required within the EVS payload to extract the bits for each EVS instance.

In some embodiments, the parameters in the IVAS bitrate distribution control table have the following values.

An exemplary IVAS bitrate distribution control table is as follows.

Exemplary Decoding of an IVAS Bitstream In one embodiment, the steps for decoding an IVAS bitstream are as follows.

Step 1: Calculate the IVAS operating bitrate based on the bitstream length and stride _secs .

Step 2: Read the fixed length CH section indicating the spatial encoding tool.

Step 3: Based on the IVAS operating bitrate, determine the length of the CTH field by looking up the number of entries for the IVAS operating bitrate (calculated in step 1) in the IVAS bitrate distribution control table.

Step 3: Once the length of the CTH field is known, read the index offset within the CTH field.

Step 5: Use the index offset and the IVAS operating bitrate to find the actual IVAS bitrate distribution control table index.

Step 6: Read all information about EVS bitrate distribution and mono downmix backwards compatibility from the indexed table entry.

Step 7: If the mono downmix backward compatibility mode is ON, first pass the remaining IVAS bits to the EVS decoder, calculate the bit length of each EVS instance based on its EVS bitrate distribution, Read the EVS bits and decode the EVS bits using the corresponding EVS decoder to decode the spatial metadata in the MDP section.

Step 8: If the mono downmix backward compatibility mode is OFF, decode the spatial metadata in the MDP section, calculate the bit length of each EVS instance based on its EVS bitrate distribution, and Read and decode the EVS bits for each EVS instance from the EP section.

Step 9: Use the decoded EVS output and spatial metadata to construct the input audio format, such as stereo (CACPL) or FoA (SPAR).

An advantage of the IVAS bitstream format embodiment described above is that it efficiently and compactly encodes data that supports a variety of audio service capabilities. These audio service capabilities include, but are not limited to, mono-to-stereo upmix and fully immersive audio encoding, decoding and rendering. This embodiment is also supported by a wide range of devices, endpoints and network nodes. These wide range of devices and others include mobile phones and smart phones, electronic tablets, personal computers, conference phones, conference rooms, virtual reality (VR) and augmented reality (AR) devices, home theater devices, and other suitable devices. but not limited to, each of which can have different acoustic interfaces for capturing and rendering sound. The IVAS bitstream format is extensible so that it can evolve easily with IVAS standards and technologies.

Exemplary IVAS SPAR Encoding/Decoding The following description of a further embodiment focuses on the differences between this further embodiment and the previously described embodiments. Accordingly, features common to both embodiments may be omitted from the following description and, if omitted, features of the previously described embodiments may be implemented or at least implemented in this further embodiment. It should be assumed (unless the following description requires otherwise) that it can. In addition, when a feature is taken from an embodiment disclosed below and added to a claim, that feature may not be related or closely related to other features of that embodiment.

In some implementations, the IVAS SPAR encoder determines the encoding mode/tool indicator and encodes it in the common header (CH) section of the IVAS bitstream. The encoding mode/tool indicator has a value corresponding to the encoding mode/tool. The IVAS bitstream seeks a Mode Header/Tool Header and encodes it in the Tool Header (TH) section of the IVAS bitstream. Here, the TH section follows the CH section. The IVAS SPAR encoder takes a metadata payload and encodes it into the metadata payload (MDP) section of the IVAS bitstream. Here, the MDP section follows the CH section. The IVAS SPAR encoder takes an enhanced voice service (EVS) payload and encodes it into the EVS payload (EP) section of the IVAS bitstream. Here, the EP section follows the CH section. In some implementations, the IVAS system stores the bitstream on a non-transitory computer-readable medium. In other implementations, the IVAS system streams the bitstream to downstream devices. In some implementations, the IVAS SPAR encoder has the device architecture described with reference to FIG.

In some implementations, the EP section follows the MDP section. Efficient bit packing is ensured by having the EP section follow the MDP section of the IVAS bitstream, and by allowing the number of MDP and EP bits to vary (according to the bitrate distribution algorithm), Note that utilization of all available bits in the IVAS bitrate budget is ensured.

In some implementations, an IVAS SPAR decoder extracts and decodes an IVAS bitstream encoded in the IVAS SPAR format. The IVAS SPAR decoder extracts and decodes the coding mode/tool indicator in the CH section of the bitstream. The encoding mode/tool indicator has a value corresponding to the encoding mode/tool. The IVAS SPAR decoder extracts and decodes the mode header/tool header in the tool header (TH) section of the bitstream. A TH section follows a CH section. The IVAS SPAR decoder extracts and decodes the metadata payload in the MDP section of the bitstream. The MDP section follows the CH section. The IVAS SPAR decoder decodes the EVS payload within the EP section of the bitstream. The EP section follows the CH section.

In some implementations, the IVAS system controls the audio decoder based on the encoding mode, tool header, EVS payload, and metadata payload. In other implementations, the IVAS system stores representations of the encoding mode, tool header, EVS payload, and metadata payload on non-transitory computer-readable media. In some implementations, the IVAS SPAR decoder has the device architecture described with reference to FIG.

In some implementations, CH contains a 3-bit data structure, one of the values of the 3-bit data structure corresponds to a SPAR coding mode and the rest of the values correspond to other coding modes. . A 3-bit data structure is advantageous because it allows a compact code that can indicate up to 8 encoding modes. In other embodiments, CH contains less than 3 bits. In other embodiments, CH contains more than 3 bits.

In some implementations, the IVAS system stores or reads from the TH section of the IVAS bitstream a row index pointing to a row in the SPAR bitrate distribution control table. For example, the row index can be calculated based on the number of rows corresponding to the IVAS operating bitrate as follows: x=ceil(log ₂ (the number of rows corresponding to the IVAS bitrate)) . Therefore, the length of the TH section is variable.

In some implementations, the system stores the quantization strategy indicator; the encoding strategy indicator; and the quantized and encoded real and imaginary parts of one or more coefficients in the MDP section of the IVAS bitstream. or read from the MDP section of the IVAS bitstream.

In other embodiments, the system stores the quantization strategy indicator in or reads the MDP section of the IVAS bitstream.

In other implementations, the system stores the encoding strategy indicator in the MDP section of the IVAS bitstream or reads it from the MDP section of the IVAS bitstream.

In other implementations, the system stores or reads the quantized and encoded real and imaginary parts of one or more coefficients in the MDP section of the IVAS bitstream.

In some embodiments, the one or more coefficients include prediction coefficients, co-prediction coefficients (or direct coefficients), real (diagonal) decorrelator coefficients and complex (off-diagonal) decorrelator coefficients, including It is not limited.

In some implementations, more or fewer coefficients are stored in the MDP section of the IVAS bitstream and read out from the MDP section of the IVAS bitstream.

In some implementations, the IVAS system stores or reads the EVS payloads of all channels in the EP section of the IVAS bitstream according to 3GPP TS26.445.

An example IVAS bitstream using SPAR formatting is shown below. The IVAS bitstream contains four subdivisions as follows.

Common Header (CH):
In some implementations, the IVAS Common Header (CH) is formatted as follows.

Tool Header (TH):
In some implementations, the SPAR Tool Header (TH) is an index offset into the SPAR Bitrate Distribution Control Table.

An exemplary implementation of a SPAR bitrate distribution control table is shown below. Each IVAS bitrate has Bandwidth (BW), downmix configuration (dmx channel, dmx string), active W, complex flags, transition mode value, EVS bitrate setting, metadata quantization level setting and decorrelator ducking. One or more values of the (ducking) flag can be supported. In this exemplary implementation, the number of bits in the SPAR TH section is 0 since there is only one entry per bitrate. The acronyms used in the table below are defined as follows.
PR: prediction coefficient;
C: co-prediction coefficient (or direct coefficient),
P _r : real (diagonal) decorrelator coefficients,
P _c : complex (off-diagonal) decorrelator coefficients.

An example SPAR bitrate distribution control table is as follows.

Metadata Payload (MDP):
An example metadata payload (MDP) is as follows:

EVS Payload (EP):
In some implementations, the quantization and calculation of the actual EVS bitrate metadata for each downmix channel is performed using an EVS bitrate distribution control strategy. An exemplary implementation of the EVS bitrate distribution control strategy is described below.

Exemplary EVS Bitrate Distribution Control Strategy In some implementations, the EVS bitrate distribution control strategy includes two sections: metadata quantization and EVS bitrate distribution.

Metadata quantization. There are two defined thresholds in this section: Target Parameter Bitrate Threshold (MDtar) and Maximum Target Bitrate Threshold (MDmax).

Step 1: For each frame, the parameters are quantized with a non-temporal difference method and encoded with an entropy coder. In some implementations, an arithmetic coder is used. In another embodiment, a Huffman encoder is used. If the parameter bitrate estimate is less than MDtar, any extra available bits are provided to the audio encoder to increase the bitrate of the audio essence.

Step 2: If step 1 fails, then a subset of the parameter values in the frame are quantized and subtracted from the quantized parameter values in the previous frame, and the differentially quantized parameter values are used by the entropy coder. is encoded using If the parameter bitrate estimate is less than MDtar, any extra available bits are provided to the audio encoder to increase the bitrate of the audio essence.

Step 3: If step 2 fails, the bitrate of the quantized parameter is calculated without entropy.

Step 4: The results of steps 1, 2 and 3 are compared with MDmax. If the minimum of steps 1, 2 and 3 is within MDmax, the remaining bits are encoded and provided to the audio coder.

Step 5: If step 4 fails, the parameters are quantized more coarsely and the above steps are repeated as the first fallback strategy (fallback 1).

Step 6: If step 5 fails, the parameters are quantized using a quantization scheme guaranteed to be within MDmax as a second fallback strategy (fallback 2). After all the iterations described above, the metadata bitrate is guaranteed to be within MDmax, and the encoder produces the actual metadata bits, Metadata_actual_bits (MDact).

EVS bitrate distribution (EVSbd). For this section, the following definitions apply.
EVStar: EVS target bit, desired bit for each EVS instance.
EVSact: EVS Actual Bits, sum of actual bits available for all EVS instances.
EVSmin: EVS minimum bit, minimum bit for each EVS instance. The EVS bitrate must not fall below the value indicated by these bits.
EVSmax: EVS maximum bit, maximum bit for each EVS instance. The EVS bitrate must not exceed the value indicated by these bits.
EVS W: EVS instance that encodes the W channel.
EVS Y: EVS instance that encodes the Y channel.
EVS X: An EVS instance that encodes the X channel.
EVS Z: EVS instance that encodes the Z channel.
EVSact=IVAS_bits-header_bits-MDact

If EVSact is less than the sum of EVStar of all EVS instances, bits are taken from the EVS instances in the following order (Z, X, Y, W). Maximum bits that can be taken from any channel=EVStar(ch)-EVSmin(ch).

If EVSact is greater than the sum of EVStar of all EVS instances, then all additional bits are allocated to downmix channels in the following order (W, Y, X, Z). The maximum extra bits that can be added to any channel = EVSmax(ch) - EVStar(ch).

The EVSbd scheme described above calculates the actual EVS bitrates for all channels, ie EWa, EYa, EXa, EZa for W, Y, X and Z channels respectively. After each channel is encoded by a separate EVS instance using EWa, EYa, EXa and EZa bit rates, all EVS bits are concatenated and packed together. The advantage of this configuration is that no additional header is required to indicate the EVS bitrate for any channel.

In some embodiments, the EP section is as follows.

Exemplary SPAR Decoder Bitstream Unpacking In some embodiments, the steps for SPAR decoder bitstream unpacking are described as follows.

Step 1: Obtain the IVAS bit rate from the length of the received bit buffer.

Step 2: Parse the SPAR TH section based on the number of IVAS bitrate entries in the SPAR bitrate distribution control table and extract the index offset. Here, this index offset is determined by the IVAS operating bitrate.

Step 3: Use the index offset to obtain the actual table row index of the SPAR bitrate distribution control table, and read all the columns of the SPAR bitrate distribution control table row pointed to by this actual table row index.

Step 4: Read the quantization strategy bits and coding strategy bits from the MDP section of the IVAS bitstream and unquantize the SPAR spatial metadata in the MPD section based on the indicated quantization strategy and coding strategy. )do.

Step 5: Based on the total EVS bitrate (remaining bits read from the IVAS bitstream), find the actual EVS bitrate for each channel for each EVS bitrate distribution (EVSbd) described above.

Step 6: Read the encoded EVS bits from the EP section of the IVAS bitstream based on the actual EVS bitrate, and use the respective EVS instance to decode each channel of the FoA audio signal.

Step 7: Construct a FoA (SPAR) audio signal using the decoded EVS output and spatial metadata.

An advantage of the IVAS bitstream format embodiment described above is that it efficiently and compactly encodes data that supports a variety of audio service capabilities. These audio service capabilities include, but are not limited to, mono-to-stereo upmix and fully immersive audio encoding, decoding and rendering (eg FoA encoding). This embodiment is also supported by a wide range of devices, endpoints and network nodes. These wide range of devices and others include mobile phones and smart phones, electronic tablets, personal computers, conference phones, conference rooms, virtual reality (VR) and augmented reality (AR) devices, home theater devices, and other suitable devices. but not limited to, each of which can have different acoustic interfaces for capturing and rendering sound. The IVAS bitstream format is extensible so that it can evolve easily with IVAS standards and technologies.

Exemplary Process—IVAS Bitstream in CACPL Format FIG. 4A is a flow diagram of an IVAS encoding process 400 according to one embodiment. Process 400 may be implemented using a device architecture such as that described with reference to FIG.

The process 400 includes determining an encoding tool indicator and a sampling rate indicator using an IVAS encoder and encoding (401) the encoding tool indicator and the sampling rate indicator into a common header (CH) section of the IVAS bitstream. include. In some implementations, the tool indicator has a value corresponding to the encoding tool and the sampling rate indicator has a value that indicates the sampling rate.

The process 400 includes determining an enhanced voice service (EVS) payload using an IVAS encoder and encoding (402) the enhanced voice service (EVS) payload into an EVS payload (EP) section of an IVAS bitstream. Including further. In some embodiments, the EP section follows the CH section.

Process 400 further includes determining a metadata payload in the metadata payload using an IVAS encoder and encoding (403) the metadata payload into a metadata payload (MDP) section of the IVAS bitstream. In some implementations, the MDP section follows the CH section. In some implementations, the EP section follows the MDP section of the bitstream.

Process 400 further includes storing the IVAS bitstream on a non-transitory computer-readable medium or streaming the IVAS bitstream to a downstream device (404).

FIG. 4B is a flow diagram of an IVAS encoding process 405 using an alternative IVAS format, according to one embodiment. Process 405 can include device architecture as described with reference to FIG.

Process 405 includes determining an encoding tool indicator using an IVAS encoder and encoding 406 the encoding tool indicator into a common header (CH) section of the IVAS bitstream. In some implementations, the tool indicator has a value corresponding to the encoding tool.

Process 405 further includes encoding (407) a representation of the IVAS bitrate distribution control table into a Common Spatial Encoding Tool Header (CTH) section of the IVAS bitstream using an IVAS encoder.

Process 405 further includes determining a metadata payload using an IVAS encoder and encoding (408) the metadata payload into a metadata payload (MDP) section of the IVAS bitstream. In some implementations, the MDP section follows the CH section of the IVAS bitstream.

Process 405 determines an enhanced voice service (EVS) payload using an IVAS encoder and encodes (409) the enhanced voice service (EVS) payload into an EVS payload (EP) section of the IVAS bitstream. Including further. In some implementations, the EP section follows the CH section of the IVAS bitstream. In some implementations, the MDP section follows the EP section of the IVAS bitstream.

Process 405 further includes storing the IVAS bitstream on a storage device or streaming 410 the IVAS bitstream to a downstream device.

FIG. 5A is a flow diagram of an IVAS decoding process 500 according to one embodiment. Process 500 may be implemented using a device architecture such as that described with reference to FIG.

Process 500 includes extracting and decoding (501) an encoding tool indicator and a sampling rate indicator from a common header (CH) section of an IVAS bitstream using an IVAS decoder. In some implementations, the tool indicator has a value corresponding to the encoding tool and the sampling rate indicator has a value that indicates the sampling rate.

Process 500 further includes extracting and decoding Enhanced Voice Service (EVS) payload from the EVS payload (EP) section of the IVAS bitstream using an IVAS decoder (502). In some implementations, the EP section follows the CH section of the IVAS bitstream.

The process 500 further includes extracting and decoding (503) the metadata payload from the metadata payload (MDP) section of the bitstream using the IVAS decoder. In some implementations, the MDP section follows the CH section of the IVAS bitstream. In some implementations, the EP section follows the MDP section of the IVAS bitstream.

Process 500 controls an audio decoder based on the encoding tool, sampling rate, EVS payload, and metadata payload, or converts the representation of the encoding tool, sampling rate, EVS payload, and metadata payload into a non-temporal Further including storing (504) on a computer readable medium.

FIG. 5B is a flow diagram of an IVAS decoding process 505 using an alternate format, according to one embodiment. Process 505 may be implemented using a device architecture such as that described with reference to FIG.

Process 505 includes extracting and decoding (506) the coding tool indicator in the common header (CH) section of the IVAS bitstream using an IVAS decoder. In some implementations, the tool indicator has a value corresponding to the encoding tool.

Process 505 further includes using an IVAS decoder to extract and decode (507) the representation of the IVAS bitrate distribution control table in the Common Spatial Encoding Tool Header (CTH) section of the IVAS bitstream.

Process 505 further includes decoding 508 the metadata payload in the metadata payload (MDP) section of the IVAS bitstream using the IVAS decoder. In some implementations, the MDP section follows the CH section of the IVAS bitstream.

Process 505 further includes decoding 509 the enhanced voice service (EVS) payload within the EVS payload (EP) section of the IVAS bitstream using the IVAS decoder. In some implementations, the EP section follows the CH section of the IVAS bitstream. In some implementations, the MDP section follows the EP section of the IVAS bitstream.

Process 505 controls an audio decoder based on the representation of the encoding tool indicator, the IVAS bitrate distribution control table, the metadata payload, and the EVS payload, or the representation of the encoding tool indicator, the IVAS bitrate distribution control table. , the metadata payload, and the EVS payload representation on a storage device (510).

An Exemplary Process—IVAS Bitstream in SPAR Format FIG. 6 is a flow diagram of an IVAS SPAR encoding process 600 according to one embodiment. Process 600 may be implemented using a device architecture such as that described with reference to FIG.

The process 600 includes decoding the encoding mode/encoding tool indicator using an IVAS encoder and encoding the encoding mode/encoding tool indicator into a common header (CH) section of the IVAS bitstream (601 )including.

The process 600 further includes using an IVAS encoder to obtain a representation of a SPAR bitrate distribution control table and encoding (602) into a mode header/tool header in a tool header (TH) section of the IVAS bitstream. . Here, the TH section follows the CH section.

Process 600 further includes determining a metadata payload using an IVAS encoder and encoding 603 the metadata payload into a metadata payload (MDP) section of the IVAS bitstream. In some implementations, the MDP section follows the CH section of the IVAS bitstream.

In some implementations, the MDP section includes a quantization strategy indicator; a coding strategy indicator; and quantized and coded real and imaginary parts of one or more coefficients. In some embodiments, the one or more coefficients include prediction coefficients, co-prediction coefficients (or direct coefficients), real (diagonal) decorrelator coefficients and complex (off-diagonal) decorrelator coefficients, including It is not limited. In some implementations, more or fewer coefficients are stored in the MDP section of the IVAS bitstream and read out from the MDP section of the IVAS bitstream.

Process 600 further includes determining an enhanced voice service (EVS) payload using an IVAS encoder and encoding 604 the EVS payload into an EVS payload (EP) section of the IVAS bitstream. In some implementations, the EP section of the IVAS bitstream contains the EVS payloads of all channels according to 3GPP TS26.445. In some implementations, the EP section follows the CH section of the IVAS bitstream. In some implementations, the EP section follows the MDP section. Efficient bit packing is ensured by having the EP section follow the MDP section of the IVAS bitstream, and by allowing the number of MDP and EP bits to vary (according to the bitrate distribution algorithm), Note that utilization of all available bits in the IVAS bitrate budget is ensured.

Process 600 further includes storing the bitstream on a non-transitory computer-readable medium or streaming the bitstream to a downstream device (605).

FIG. 7 is a flow diagram of an IVAS SPAR decoding process 700 according to one embodiment. Process 700 may be implemented using a device architecture such as that described with reference to FIG.

Process 700 includes extracting and decoding (701) a coding mode indicator in a common header (CH) section of an IVAS bitstream using an IVAS decoder.

The process 700 includes extracting and decoding 702 a representation of the SPAR bitrate distribution control table in the mode header/tool header in the tool header (TH) section of the IVAS bitstream using an IVAS decoder. . In some embodiments, a TH section follows a CH section.

The process 700 further includes extracting and decoding 703 the metadata payload from the metadata payload (MDP) section of the IVAS bitstream using the IVAS decoder. In some implementations, the MDP section follows the CH section of the IVAS bitstream.

Process 700 further includes extracting and decoding Enhanced Voice Service (EVS) payload from the EVS payload (EP) section of the IVAS bitstream using an IVAS decoder (704). In some embodiments, the EP section follows the CH section. In some implementations, the EP section follows the MDP section. Efficient bit packing is ensured by having the EP section follow the MDP section of the IVAS bitstream, and by allowing the number of MDP and EP bits to vary (according to the bitrate distribution algorithm), Note that utilization of all available bits in the IVAS bitrate budget is ensured.

Process 700 controls an audio decoder based on a representation of a coding mode indicator, a SPAR bitrate distribution control table, an EVS payload, and a metadata payload, or a representation of a coding mode indicator, a SPAR bitrate distribution control table. , the EVS payload, and the metadata payload on a non-transitory computer-readable medium (705).

Exemplary System Architecture FIG. 8 depicts a block diagram of an exemplary system 800 suitable for implementing exemplary embodiments of the present disclosure. System 800 includes one or more server computers or any client device. These server computers or client devices can be any of the devices shown in FIG. Including, but not limited to, immersive content ingest 124 and the like. System 800 includes any consumer device including, but not limited to, smart phones, tablet computers, wearable computers, vehicle computers, game consoles, surround systems, kiosks.

As shown, the system 800 can be configured with programs stored, for example, in read only memory (ROM) 802 or loaded from, for example, storage unit 808 into random access memory (RAM) 803 . It includes a central processing unit (CPU) 801 capable of executing various processes according to programs written therein. Data required when the CPU 801 executes various processes is also stored in the RAM 803 as necessary. CPU 801 , ROM 802 and RAM 803 are interconnected via bus 804 . Input/output (I/O) interface 805 is also connected to bus 804 .

The following components: an input unit 806, which can include a keyboard, mouse, etc.; an output unit 807, which can include a display such as a liquid crystal display (LCD) and one or more speakers; a hard disk or other suitable storage. A storage unit 808 including devices; and a communication unit 809 including a network interface card such as a network card (eg, wired or wireless) are connected to the I/O interface 805 .

In some implementations, input unit 806 enables capture of audio signals in various formats (eg, mono, stereo, spatial, immersive, and other suitable formats) (depending on the host device). Contains one or more microphones at different positions.

In some implementations, the output unit 807 includes a system with varying numbers of speakers. As shown in FIG. 1, output unit 807 is capable of rendering audio signals in various formats (eg, mono, stereo, immersive, binaural, and other suitable formats) (depending on host device capabilities). can be done.

Communication unit 809 is configured to communicate with other devices (eg, over a network). Drives 810 are also connected to I/O interface 805 as needed. A removable medium 811 , such as a magnetic disk, optical disk, magneto-optical disk, flash drive, or other suitable removable medium, is provided so that a computer program read therefrom may be installed within storage unit 808 as desired. , is attached to drive 810 . Those skilled in the art will appreciate that although system 800 is described as including the components described above, in actual application some of these components may be added, removed, and/or replaced. It will be understood that all such changes or modifications are possible and are all included within the scope of the present disclosure.

Other Embodiments In one embodiment, a method of generating a bitstream of an audio signal uses an IVAS encoder to determine an encoding tool indicator and a sampling rate indicator, wherein the encoding tool indicator is the encoding tool and the sampling rate indicator has a value indicating the sampling rate; and using the IVAS encoder to set the encoding tool indicator and the sampling rate indicator in the common header (CH) section of the IVAS bitstream. Encoding; using an IVAS encoder to determine an enhanced voice service (EVS) payload; using the IVAS encoder to encode the EVS payload into an EVS payload (EP) section of an IVAS bitstream. using the IVAS encoder to determine the metadata payload; and using the IVAS encoder to convert the metadata payload to the metadata payload of the IVAS bitstream. (MDP) section, the MDP section following the CH section; and storing the IVAS bitstream on a non-transitory computer-readable medium or streaming the IVAS bitstream to a downstream device. Including things.

In one embodiment, a method of decoding a bitstream of an audio signal is extracting and decoding an encoding tool indicator and a sampling rate indicator from the CH section of the IVAS bitstream using an IVAS decoder. , the tool indicator has a value corresponding to the encoding tool, and the sampling rate indicator has a value indicating the sampling rate; and using an IVAS decoder to extract the EVS payload from the EP section of the bitstream. decoding the EP section following the CH section; decoding the metadata payload from the MDP section of the bitstream using an IVAS decoder, the MDP section following the CH section; and; controlling an audio decoder based on the encoding tool, sampling rate, EVS payload, and metadata payload; and storing on a readable computer-readable medium.

In one embodiment, the MDP section follows the EP section of the bitstream, or the EP section follows the MDP section of the bitstream.

In one embodiment, the IVAS encoding tool indicator is a 3-bit data structure, a first value of the 3-bit data structure corresponds to a multi-mono encoding tool, and a second value of the 3-bit data structure is corresponds to a Complex Advanced Coupling (CACPL) encoding tool, and the third value of the 3-bit data structure corresponds to another encoding tool.

In one embodiment, the input sampling rate indicator is a 2-bit data structure, a first value of the 2-bit data structure indicating an 8 kHz sampling rate, and a second value of the 2-bit data structure indicating a 16 kHz sampling rate. , the third value in the 2-bit data structure indicates a 32 kHz sampling rate, and the fourth value in the 2-bit data structure indicates a 48 kHz sampling rate.

In one embodiment, the method stores and reads from or stores an EVS channel number indicator, a bitrate (BR) extraction mode indicator, EVS BR data, and an EVS payload in the EP section of the bitstream, respectively. including.

In one embodiment, the method includes a coding technique indicator, a number of bands indicator, an indicator of the delay configuration of the filterbank, an indicator of quantization strategy, an entropy coder indicator, a probabilistic model type indicator, a coefficient real part, a coefficient imaginary part, and respectively storing or reading from the MDP section of the data stream one or more coefficients.

In one embodiment, a method of generating a bitstream of an audio signal is using an IVAS encoder to determine an encoding tool indicator, the tool indicator having a value corresponding to the encoding tool. using the IVAS encoder to encode the encoding tool indicator into the common header (CH) section of the IVAS bitstream; and using the IVAS encoder to obtain a representation of the index of the IVAS bitrate distribution control table. and; using an IVAS encoder to encode a representation of the index of the IVAS bitrate distribution control table into the Common Spatial Encoding Tool Header (CTH) section of the IVAS bitstream, where the CTH section is the CH section. determining the metadata payload using an IVAS encoder; and encoding the metadata payload into a metadata payload (MDP) section of the IVAS bitstream using the IVAS encoder. the MDP section follows the CTH section; using the IVAS encoder to determine the enhanced voice service (EVS) payload; using the IVAS encoder to convert the EVS payload to the EVS payload of the IVAS bitstream encoding into an (EP) section, the EP section following the CTH section; storing the bitstream on a non-transitory computer-readable medium or streaming the bitstream to a downstream device. including.

In one embodiment, a method of decoding a bitstream of an audio signal comprises: receiving the bitstream by an IVAS decoder; calculating an IVAS operating bitrate based on the length and stride of the bitstream; reading a coding tool indicator from the common header (CH) section of the bitstream; and determining the length of the common spatial coding tool header (CTH) section of the bitstream based on the IVAS operating bitrate, comprising: The determining includes examining the number of entries corresponding to the IVAS operating bitrate in an IVAS bitrate distribution control table within the CTH section; determining the length of the CTH section and determining the index of the IVAS bitrate distribution control table. and reading the values in the CTH section; and reading the information about the enhanced voice service (EVS) bitrate distribution from the IVAS bitrate distribution control table entry corresponding to the IVAS bitrate distribution control table index. and providing information about the EVS bitrate distribution to the EVS decoder.

In one embodiment, any of the above methods include reading an indicator of mono downmix backward compatibility with 3GPP TS26.445 from an entry in the IVAS bitrate distribution control table.

In one embodiment, the method includes: determining that the mono downmix backward compatibility indicator is in ON mode; providing the remainder of the bitstream to the EVS decoder in response to the ON mode; calculating the respective bit length of each EVS instance from the rest of the bitstream based on the EVS bitrate distribution; reading the EVS bits of each EVS instance based on the corresponding bit length; providing the bits as a first portion to an EVS decoder; and providing remaining portions of the bitstream to an MDP decoder to decode the spatial metadata.

In one embodiment, the method includes determining that the mono downmix backward compatibility indicator is in an OFF mode; and then calculating the respective bit length of each EVS instance from the rest of the bitstream based on the EVS bitrate distribution; and the EVS of each EVS instance based on the corresponding bit length. reading the bits; and providing the EVS bits to the EVS decoder as a first portion.

In one embodiment, a system stores instructions that, when executed by one or more computer processors, cause the one or more processors to perform the actions of any one of the above method claims a non-transitory computer-readable medium for

In one embodiment, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform the actions of any one of the above method claims.

According to exemplary embodiments of the present disclosure, the processes described above may be implemented as a computer software program or on a computer-readable storage medium. For example, embodiments of the present disclosure include computer program products including computer programs tangibly embodied on machine-readable media, which computer programs include program code for performing methods. In such an embodiment, the computer program can be downloaded and implemented from a network via communication unit 809 and/or installed from removable media 811, as shown in FIG.

In general, various exemplary embodiments of the present disclosure can be implemented in hardware or dedicated circuitry (eg, control circuitry), software, logic, or any combination thereof. For example, the units described above may be executed by control circuitry (eg, a CPU in combination with other components of FIG. 8), which thus performs the operations described in this disclosure. be able to. Some aspects can be implemented in hardware, while other aspects can be implemented in firmware or software, which can be executed by a controller, microprocessor or other computing device (eg, control circuitry). Although various aspects of the exemplary embodiments of the present disclosure are illustrated and described using block diagrams, flowcharts, or some other graphical representation, the blocks, devices, that any system, technique, or method may be implemented, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof; will be understood.

In addition, the various blocks illustrated in the flowcharts represent multiple blocks configured to perform method steps and/or actions resulting from operation of the computer program code and/or associated function(s). It can be viewed as a combined logic circuit element. For example, an embodiment of the present disclosure includes a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program having program code configured to perform a method as described above. including.

In the context of this disclosure, a machine-readable medium may be any tangible medium capable of containing or storing a program for use by or in connection with an instruction execution system, apparatus, or device. can. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may be non-transitory and may include electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof; It is not limited to these. More specific examples of machine-readable storage media are electrical connections having one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM). or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

Computer program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These computer program code are represented by general purpose computers, such that when the program code is executed by a processor of a computer or other programmable data processing apparatus, it causes the functions/acts specified in the flowchart illustrations and/or block diagrams to be performed. , dedicated computer, or other programmable data processing apparatus having control circuitry. The program code may run wholly on a computer, partially on a computer, as a stand-alone software package, partly on a computer and partly on a remote computer, or in whole. It can run on remote computers or servers, and can be distributed across one or more remote computers and/or remote servers.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of what may be claimed, but rather of features that may be inherent in particular embodiments. should be interpreted as an explanation. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features are described above as operating in certain combinations, and may even be originally claimed as such, one or more of the claimed combinations may be used in combination. The features of may in some cases be omitted from the combination, and the claimed combination may cover sub-combinations or variations of sub-combinations. The logic flow depicted in the figures does not require the particular order shown or the sequential order shown to achieve desirable results. Additionally, other steps may be provided or deleted from the described flows, and other components may be added or deleted from the described systems. Accordingly, other implementations are within the scope of the following claims.

Claims (18)

A method for generating a bitstream of an audio signal, comprising:
Determining a coding mode indicator or coding tool indicator using an immersive audio audio service (IVAS) encoder, wherein the coding mode indicator or the coding tool indicator is the coding mode of the audio signal. or indicating an encoding tool;
encoding the encoding mode indicator or the encoding tool indicator into a common header (CH) section of an IVAS bitstream using the IVAS encoder;
determining a mode header or tool header using the IVAS encoder;
encoding the mode header or the tool header into a tool header (TH) section of the IVAS bitstream using the IVAS encoder, the TH section following the CH section;
determining a metadata payload containing spatial metadata using the IVAS encoder;
encoding the metadata payload into a metadata payload (MDP) section of the IVAS bitstream using the IVAS encoder, the MDP section following the CH section;
determining an enhanced voice service (EVS) payload using the IVAS encoder, the EVS payload including EVS coded bits for each channel or downmix channel of the audio signal;
encoding the EVS payload into an EVS payload (EP) section of the IVAS bitstream using the IVAS encoder, the EP section following the CH section;
A method, including further comprising storing the IVAS bitstream on a non-transitory computer-readable medium or streaming the IVAS bitstream to a downstream device;
The encoding mode or encoding tool indicator, the mode header or the tool header, the metadata payload and the EVS payload are for use in reconstructing the audio signal in the downstream device or another device, the 2. The method of claim 1, extracted and decoded respectively from the CH section, the TH section, the MDP section and the EP section of an IVAS bitstream. A method of decoding a bitstream of an audio signal, comprising:
extracting and decoding a coding mode indicator or coding tool indicator in a common header (CH) section of an IVAS bitstream using an immersive voice audio service (IVAS) decoder, said encoding wherein the mode indicator or coding tool indicator indicates a coding mode or coding tool for the audio signal;
extracting and decoding, using the IVAS decoder, a mode header or tool header in the Tool Header (TH) section of the IVAS bitstream, the TH section following the CH section; When,
extracting and decoding, using the IVAS decoder, a metadata payload from a metadata payload (MDP) section of the IVAS bitstream, the MDP section following the CH section; the payload includes spatial metadata;
extracting and decoding, using the IVAS decoder, an enhanced voice service (EVS) payload from an EVS payload (EP) section of the IVAS bitstream, the EP section following the CH section; , the EVS payload includes EVS coded bits for each channel or each downmix channel of the audio signal;
A method, including the encoding mode indicator or the encoding tool indicator, the mode header or the tool header, the EVS payload, an audio decoder of the downstream device for use in reconstructing the audio signal in a downstream device or another device; and controlling based on said metadata payload or representation of said encoding mode indicator or said encoding tool indicator, said mode header or said tool header, said EVS payload, and said metadata payload to a non-transitory computer 4. The method of claim 3, further comprising storing on a readable medium. The CH is a multi-bit data structure, one value of the multi-bit data structure corresponds to a spatial reconstruction (SPAR) coding mode, and the other value of the data structure corresponds to another coding. 5. A method according to any one of claims 1 to 4, corresponding to a mode. 6. Storing in or reading from the TH section of the IVAS bitstream respectively an index offset for calculating a row index of a Spatial Reconstruction (SPAR) bitrate distribution control table. A method according to any one of a quantization strategy indicator;
a bitstream encoding strategy indicator;
quantized and encoded real and imaginary parts of the set of coefficients;
in or reading from the MDP section of the IVAS bitstream, respectively. The EP section follows the MDP section to ensure efficient bit packing, and the number of bits in the MDP section of the IVAS bitstream and the number of bits in the EP section of the IVAS bitstream are equal to IVAS bits. 8. A method according to any one of claims 1 to 7, varying according to the SPAR bitrate distribution control table and bitrate distribution algorithm to ensure utilization of all available bits in the rate budget. 9. A bitrate according to any one of claims 1 to 8, wherein the bitrate of each EVS encoded channel or each downmix channel is determined by EVS Total Available Bits, Bitrate Distribution Control Table and Bitrate Distribution Algorithm. Method. 8. The method of claim 7, wherein the set of coefficients includes prediction coefficients, direct coefficients, diagonal real coefficients and lower triangular complex coefficients. 11. The prediction coefficients are variable bit lengths based on entropy coding, and the direct coefficients, the diagonal real coefficients and the lower triangular complex coefficients are variable bit lengths based on downmix construction and entropy coding. The method described in . 8. The method of claim 7, wherein said quantization strategy indicator is a multi-bit data structure that indicates a quantization strategy. 8. The method of claim 7, wherein the bitstream encoding strategy indicator is a multi-bit data structure that indicates the number of spatial metadata bands and non-differential or temporal differential entropy encoding scheme. 8. The method of claim 7, wherein said quantization of said coefficients follows an EVS bitrate distribution control strategy comprising metadata quantization and EVS bitrate distribution. Storing or reading from the EP section of the bitstream, respectively, the EVS payload of the EVS instances in accordance with the 3rd Generation Partnership Project (3GPP) Technical Specification (TS) 26.445. 15. The method of any one of claims 1-14, comprising determining a bitrate from the IVAS bitstream;
reading an index offset from a Spatial Reconstruction (SPAR) Tool Header (TH) section of the IVAS bitstream;
determining a table row index of the SPAR bitrate distribution control table using the index offset;
reading quantization strategy bits and encoding strategy bits from a metadata payload (MDP) section in the IVAS bitstream;
dequantizing SPAR spatial metadata in the MDP section of the IVAS bitstream based on the quantization strategy bits and the encoding strategy bits;
determining an enhanced voice service (EVS) bitrate for each channel in the IVAS bitstream using all available EVS bits, a SPAR bitrate distribution control table and a bitrate distribution algorithm;
reading EVS encoded bits from the EP section of the IVAS bitstream based on the EVS bitrate;
decoding the EVS bits;
decoding the spatial metadata;
generating a first order Ambisonics (FoA) output using the decoded EVS bits and the decoded spatial metadata;
16. The method of any one of claims 3-15, further comprising: one or more processors;
a non-transitory computer-readable medium storing instructions that, when executed by said one or more processors, cause said one or more processors to perform the operations of the method of any one of claims 1 to 16;
A system comprising: A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the operations of the method of any one of claims 1 to 16.

Images