RU2821284C1 - Distribution of bit rates in immersive voice and audio services - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to computer engineering for processing bit streams of audio data. Technical result is achieved by determining, using one or more processors, a combination of one or more bit rates for downmix channels; determining, using one or more processors, a metadata quantization level from a set of metadata quantization levels; quantizing and encoding, using one or more processors, spatial metadata using a metadata quantization layer; generating, using one or more processors and a combination of one or more bit rates, a downmix bit stream for one or more downmix channels; combining, using one or more processors, a downmix bit stream, quantized and encoded spatial metadata and a set of quantization levels into an IVAS bit stream.

EFFECT: reducing spatial metadata, mono codec overhead and minimizing bit loss to zero.

12 cl, 11 dwg, 4 tbl

Description

Cross reference to related applications

[0001] This application claims priority to U.S. Provisional Patent Application No. 62/927,772, filed October 30, 2019; and U.S. Provisional Patent Application No. 63/092,830, filed October 16, 2020, which are incorporated herein by reference.

Field of technology to which the invention relates

[0002] This invention relates generally to encoding and decoding audio bitstreams.

State of the art

[0003] Standards development for voice and audio encoders/decoders (“codecs”) has recently focused on codec development for immersive voice and audio services (IVAS). IVAS is expected to support a range of audio delivery service characteristics including, but not limited to, mono to stereo upmixing and encoding, decoding and rendering of fully immersive audio. IVAS intends to be supported through a wide range of devices, endpoints and network nodes, including but not limited to: mobile phones and smartphones, electronic tablets, personal computers, conference phones, meeting rooms, virtual reality (VR) style devices and augmented reality (AR), home theater devices and other suitable devices. These devices, endpoints, and network nodes may have different acoustic interfaces for capturing and rendering audio.

Disclosure of the Invention

[0004] Implementations for distributing bit rates in immersive voice and audio services are disclosed.

[0005] In an embodiment, a method for encoding a bitstream in support of immersive voice and audio services (IVAS), the method comprising: receiving, using one or more processors, an input audio signal; downmixing, using one or more processors, the input audio signal into one or more downmix channels and spatial metadata associated with the one or more input audio signal channels; reading, using one or more processors, a set of one or more bit rates for the downmix channels and a set of quantization levels for spatial metadata from the bit rate distribution control table; determining, using one or more processors, a combination of one or more bit rates for the downmix channels; determining, using one or more processors, a metadata quantization level from a set of metadata quantization levels using a bit rate distribution process; quantizing and encoding, using one or more processors, spatial metadata using a metadata quantization layer; generating, using one or more processors and a combination of one or more bit rates, a downmix bit stream for one or more downmix channels; combining, using one or more processors, the downmix bitstream, the quantized and encoded spatial metadata, and a set of quantization levels into an IVAS bitstream; and streaming or storing the IVAS bitstream for playback on an IVAS-enabled device.

[0006] In an embodiment, the input audio signal is a four-channel first-order ambiophony (FoA) audio signal, a three-channel flat FoA signal, or a two-channel stereo audio signal.

[0007] In an embodiment, the one or more bit rates represent the bit rates of one or more mono audio encoder/decoder (codec) bit rate channels.

[0008] In an embodiment, the mono audio codec is an enhanced voice services (EVS) codec, and the downmix bitstream is an EVS bitstream.

[0009] In an embodiment, obtaining, using one or more processors, one or more bit rates for the downmix channels and spatial metadata using the bit rate distribution control table further comprises: identifying a row in the bit rate distribution control table with using a table index that includes the input audio signal format, the input audio signal bandwidth, the enabled spatial encoding tool, the transient mode, and the backward compatible mode for mono downmixing; retrieving, from an identified bit rate allocation control table row, a target bit rate, a bit rate ratio, a minimum bit rate, and bit rate deviation steps, wherein the bit rate ratio indicates the ratio with which the total bit rate should be allocated between transmission channels of downmix audio signals, the minimum bit rate is a value below which the full bit rate is not permitted to be reduced, and the bit rate deviation steps are steps to reduce the target bit rate when the first priority for downmix signals is greater than or equal to or below the second priority of spatial metadata; and determining one or more bit rates for the downmix channels and spatial metadata based on the target bit rate, bit rate ratio, minimum bit rate, and bit rate deviation steps.

[0010] In an embodiment, quantization of spatial metadata for one or more channels of an input audio signal using a set of quantization levels is performed in a quantization loop that applies increasingly coarse quantization strategies based on the difference between the target metadata bit rate and the actual bit rate in bits of metadata.

[0011] In an embodiment, quantization is determined according to mono codec priority and spatial metadata priority based on properties extracted from the input audio signal and covariance values in the channel bandwidth.

[0012] In an embodiment, the input audio signal is a stereo signal, and the downmix signals include an average signal representation, residuals from the stereo signal, and spatial metadata.

[0013] In an embodiment, the spatial metadata includes prediction coefficients (PR), cross-prediction coefficients (C) and decorrelation coefficients (P) for the format for spatial reconstruction module (SPAR) and prediction coefficients (P) and decorrelation coefficients ( PR) for Advanced Complex Communications (CACPL) format.

[0014] In an embodiment, a method for encoding a bitstream in support of immersive voice and audio services (IVAS), the method comprising: receiving, using one or more processors, an input audio signal; extracting, using one or more processors, properties of the input audio signal; calculating, using one or more processors, spatial metadata for the input audio signal channels; reading, using one or more processors, a set of one or more bit rates for the downmix channels and a set of quantization levels for spatial metadata from the bit rate distribution control table; determining, using one or more processors, a combination of one or more bit rates for the downmix channels; determining, using one or more processors, a metadata quantization level from a set of metadata quantization levels using a bit rate distribution process; quantizing and encoding, using one or more processors, spatial metadata using a metadata quantization layer; generating, using one or more processors and a combination of one or more bit rates, a downmix bit stream for one or more downmix channels using one or more bit rates; combining, using one or more processors, the downmix bitstream, the quantized and encoded spatial metadata, and a set of quantization levels into an IVAS bitstream; and streaming or storing the IVAS bitstream for playback on an IVAS-enabled device.

[0015] In an embodiment, the properties of the input audio signal include one or more of bandwidth, speech/music classification data, and voice activity detection (VAD) data.

[0016] In an embodiment, the number of downmix channels to be encoded into the IVAS bitstream is selected based on the residual level indicator in the spatial metadata.

[0017] In an embodiment, a method for encoding a bitstream supporting immersive voice and audio services (IVAS) further comprises: receiving, using one or more processors, a first-order ambiophonic audio (FoA) input signal; extracting, using one or more processors and the IVAS bit rate, properties of the input FoA audio signal, wherein one of the properties is the bandwidth of the input FoA audio signal; generating, using one or more processors, spatial metadata for the input FoA audio signal using properties of the FoA signals; selecting, using one or more processors, the number of residual channels to be sent based on the residual level indicator and decorrelation coefficients in the spatial metadata; obtaining, using one or more processors, a bit rate distribution control table index based on the IVAS bit rate, bandwidth, and number of downmix channels; reading, using one or more processors, a spatial recovery module (SPAR) configuration from a row in the bit rate distribution control table pointed to by the bit rate distribution control table index; determining, using one or more processors, a target metadata bit rate from the IVAS bit rate, a sum of the target EVS bit rates, and an IVAS header length; determining, using one or more processors, a maximum metadata bit rate from the IVAS bit rate, a sum of the minimum EVS bit rates, and an IVAS header length; quantizing, using one or more processors and a quantization loop, the spatial metadata in a time-nondifferential manner according to a first quantization strategy; entropy encoding, using one or more processors, of quantized spatial metadata; calculating, using one or more processors, a first actual transmission rate in metadata bits; determining, using one or more processors, whether the first actual metadata bit rate is less than or equal to the target metadata bit rate; and according to the first actual metadata bit rate less than or equal to the target metadata bit rate: leaving the quantization loop.

[0018] In an embodiment, the method further comprises: determining, using one or more processors, a first overall actual EVS bit rate by adding a first number of bits equal to the difference between the target metadata bit rate and the first actual metadata bit rate , with the full EVS target bit rate; generating, using one or more processors, an EVS bit stream using the first full actual EVS bit rate; generating, using one or more processors, an IVAS bitstream including an EVS bitstream, a bit rate distribution control table index, and quantized and entropy encoded spatial metadata; according to the first actual metadata bit rate greater than the target metadata bit rate: quantizing, using one or more processors, the spatial metadata in a time differential manner according to the first quantization strategy; entropy encoding, using one or more processors, of quantized spatial metadata; calculating, using one or more processors, a second actual transmission rate in metadata bits; determining, using one or more processors, whether the second actual metadata bit rate is less than or equal to the target metadata bit rate; and according to a second actual metadata bit rate less than or equal to the target metadata bit rate: leaving the quantization loop.

[0019] In an embodiment, the method further comprises: determining, using one or more processors, a second overall actual EVS bit rate by adding a second number of bits equal to the difference between the target metadata bit rate and the second actual metadata bit rate , with the full EVS target bit rate; generating, using one or more processors, an EVS bitstream using the second full actual EVS bitrate; generating, using one or more processors, an IVAS bitstream including an EVS bitstream, a bit rate distribution control table index, and quantized and entropy encoded spatial metadata; according to a second actual metadata bit rate greater than the target metadata bit rate: quantizing, using one or more processors, the spatial metadata in a time-nondifferential manner according to a first quantization strategy; encoding, using one or more processors and a radix 2 encoder, the quantized spatial metadata; calculating, using one or more processors, a third actual bit rate of metadata; and according to a third actual metadata bit rate less than or equal to the target metadata bit rate: leaving the quantization loop.

[0020] In an embodiment, the method further comprises: determining, using one or more processors, a third overall actual EVS bit rate by summing a third number of bits equal to the difference between the target metadata bit rate and the third actual metadata bit rate , with the full EVS target bit rate; generating, using one or more processors, an EVS bitstream using a third full actual EVS bit rate; generating, using one or more processors, an IVAS bitstream including an EVS bitstream, a bit rate distribution control table index, and quantized and entropy encoded spatial metadata; according to a third actual metadata bit rate greater than the target metadata bit rate: setting, using one or more processors, a fourth actual metadata bit rate as the minimum of the first, second and third actual metadata bit rates ; determining, using one or more processors, whether the fourth actual metadata bit rate is less than or equal to the maximum metadata bit rate; according to a fourth actual metadata bit rate less than or equal to a maximum metadata bit rate: determining, using one or more processors, whether the fourth actual metadata bit rate is less than or equal to a target metadata bit rate ; and according to a fourth actual metadata bit rate less than or equal to the target metadata bit rate: leaving the quantization loop.

[0021] In an embodiment, the method further comprises: determining, using one or more processors, a fourth overall actual EVS bit rate by adding a fourth number of bits equal to the difference between the target metadata bit rate and the fourth actual metadata bit rate , with the full EVS target bit rate; generating, using one or more processors, an EVS bitstream using a fourth of the full actual EVS bitrate; generating, using one or more processors, an IVAS bitstream including an EVS bitstream, a bit rate distribution control table index, and quantized and entropy encoded spatial metadata; and according to the fourth actual metadata bit rate, greater than the target metadata bit rate and less than or equal to the maximum metadata bit rate: leaving the quantization loop.

[0022] In an embodiment, the method further comprises: determining, using one or more processors, a fifth overall actual EVS bit rate by subtracting a number of bits equal to the difference between the fourth actual metadata bit rate and the target metadata bit rate, from the full EVS target bit rate; generating, using one or more processors, an EVS bitstream using a fifth of the actual EVS bitrate; generating, using one or more processors, an IVAS bitstream including an EVS bitstream, a bit rate distribution control table index, and quantized and entropy encoded spatial metadata; according to a fourth actual metadata bit rate greater than the maximum metadata bit rate: changing the first quantization strategy to a second quantization strategy and entering the quantization loop again using the second quantization strategy, the second quantization strategy being more approximate than the first strategy quantization. In an embodiment, a third quantization strategy may be used that is guaranteed to provide an actual MD bit rate that is less than the maximum MD bit rate.

[0023] In an embodiment, the SPAR configuration is specified by a downmix string, an active W flag, a complex spatial metadata flag, spatial metadata quantization strategies, minimum, maximum, and target bit rates for one or more enabled monocoder/decoder (codec) instances. enhanced voice services (EVS) and a time domain decorrelator muting flag.

[0024] In an embodiment, the total actual number of EVS bits is equal to the number of IVAS bits minus the number of header bits minus the actual metadata bit rate, and wherein if the total number of actual EVS bits is less than the total number of EVS target bits, then the bits are extracted from the EVS channels in the following order: Z, X, Y and W, and the maximum number of bits that can be extracted from any channel is equal to the number of target EVS bits for the channel minus the minimum number of EVS bits for the channel, and if the number of actual EVS bits is greater number of target EVS bits, then all additional bits are assigned to the downmix channels in the following order: W, Y, X and Z, and the maximum number of additional bits that can be added to any channel is equal to the maximum number of EVS bits minus the number of target EVS bits.

[0025] In an embodiment, a method for decoding a bitstream supporting immersive voice and audio services (IVAS) comprises: receiving, using one or more processors, an IVAS bitstream; obtaining, using one or more processors, the IVAS bit rate from the bit length of the IVAS bit stream; obtaining, using one or more processors, a bit rate distribution control table index from the IVAS bit stream; parsing, using one or more processors, a strategy for quantizing metadata from the IVAS bitstream header; parsing and dequantizing, using one or more processors, bits of quantized spatial metadata based on a metadata quantization strategy; setting, using one or more processors, an actual number of enhanced voice services (EVS) enabled bits equal to the remaining bit length of the IVAS bit stream; reading, using one or more processors and the bit rate distribution control table index, table entries for the bit rate distribution control table that contain a target and minimum EVS bit rate and a maximum EVS bit rate for one or more EVS instances; obtaining, using one or more processors, the actual EVS bit rate for each downmix channel; and decoding, using one or more processors, each EVS channel using the actual EVS bit rate for the channel; and upmixing, using one or more processors, the EVS channels into first order ambiophonic (FoA) channels.

[0026] In an embodiment, the system includes: one or more processors; and a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause one or more processors to perform operations in any of the methods described above.

[0027] In an embodiment, a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause one or more processors to perform operations in any of the methods described above.

[0028] Other implementations disclosed herein relate to a system, device, and computer-readable medium. Details of the disclosed implementations are set forth in the accompanying drawings and in the description below. Other features, objects and advantages will become apparent from the description, drawings and claims.

[0029] Specific implementations disclosed herein provide one or more of the following advantages. The IVAS codec bit rate is shared between the mono codec and spatial metadata (MD) and among multiple mono codec instances. For a given audio frame, the IVAS codec determines the spatial audio coding mode (parametric or residual coding). The IVAS bitstream is optimized to reduce spatial MD, reduce mono codec overhead, and minimize bit loss to zero.

Brief description of drawings

[0030] In the drawings, specific arrangements or arrangements of schematic elements, for example, elements that represent devices, modules, instruction processing units, and data elements, are shown for ease of description. However, those skilled in the art will understand that the particular ordering or arrangement of the schematic elements in the drawings is not intended to imply that a particular order or sequence of processing or division of processes is required. Moreover, the inclusion of a schematic element in the drawing is not intended to imply that such element is required in all embodiments, or that in some implementations the features represented by such element may not be included or may be combined with other elements.

[0031] Additionally, in drawings in which connecting elements, such as solid or dotted lines or arrows, are used to illustrate a connection, relationship, or association between two or more other schematic elements, the absence of such connecting elements is not intended to imply that connection, relationship or association cannot exist. In other words, certain connections, relationships or associations between elements are not shown in the drawings so as not to obscure the invention. In addition, for ease of illustration, a single connecting element is used to represent multiple connections, relationships, or associations between elements. For example, if a connecting element represents the exchange of signals, data, or instructions, those skilled in the art will understand that such element represents one or more signal paths, depending on the need for communication.

[0032] FIG. 1 illustrates use cases for the IVAS codec, according to an embodiment.

[0033] FIG. 2 is a block diagram of a system for encoding and decoding IVAS bitstreams according to an embodiment.

[0034] FIG. 3 is a block diagram of a first-order ambiophony (FoA) encoder/decoder (“codec”) for a spatial reconstruction module (SPAR) for encoding and decoding IVAS bitstreams in FoA format according to an embodiment.

[0035] FIG. 4A is a block diagram of an IVAS signal chain for FoA input signals and stereo signals, according to an embodiment.

[0036] FIG. 4B is a block diagram of an alternative IVAS signal chain for FoA input signals and stereo signals, according to an embodiment.

[0037] FIG. 5A is a flowchart of a process for allocating bit rates for input stereo signals, flat FoA and FoA signals according to an embodiment.

[0038] FIG. 5B and 5C are a flow diagram of a process for allocating bit rates for FoA input signals for a spatial recovery module (SPAR), according to an embodiment.

[0039] FIG. 6 is a flow chart of a process for allocating bit rates for input stereo signals, flat FoA and FoA signals, according to an embodiment.

[0040] FIG. 7 is a flowchart for a bit rate allocation process for an input FoA SPAR signal, according to an embodiment.

[0041] FIG. 8 is a block diagram of an exemplary device architecture, according to an embodiment.

[0042] Like reference numerals used in different drawings indicate like elements.

Carrying out the invention

[0043] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. It will be apparent to those skilled in the art that the various implementations described may be practiced without these specific details. In other cases, well-known methods, procedures, components and circuits are not described in detail so as not to obscure aspects of the embodiments. A variety of features are described below that can be used independently of each other or with any combination of other features.

Terminology

[0044] When used herein, the term “includes” and its variations should be read as non-exclusive terms that mean “includes but is not limited to.” The term “or” should be read as “and/or” unless the context clearly indicates otherwise. The term “based on” should be interpreted as “at least partially based on.” The terms “one example implementation” and “example implementation” should be read as “at least one example implementation”. The term "other implementation" should be read as "at least one other implementation". The terms “determined,” “determines,” or “determining” should be read as receiving, receiving, calculating, calculating, estimating, predicting, or extracting. In addition, in the following description and claims, unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention relates.

Examples of IVAS use cases

[0045] FIG. 1 illustrates use cases 100 for IVAS codec 100 according to one or more implementations. In some implementations, various devices communicate through a call server 102, which is configured to receive audio signals from, for example, a public switched telephone network (PSTN) or a public land mobile network (PLMN), illustrated by PSTN/other PLMN 104. Use cases 100 support previously created devices 106 that render and capture only audio in mono, including, but not limited to: devices that support enhanced voice services (EVS), the adaptive multi-rate wideband (AMR-WB) standard, and the narrowband standard. adaptive multi-rate coding (AMR-NB). Use cases 100 also support a user equipment (UE) 108, 114 that captures and renders stereo audio signals, or a UE 110 that captures and binaurally renders mono signals to multi-channel signals. Use cases 100 also support immersive and stereo signals captured and rendered by video conference room systems 116, 118, respectively. Use cases 100 also support stereo capture and immersive rendering of stereo audio signals for home theater systems 120 and computer 112 and mono capture and immersive rendering of audio signals for virtual reality (VR) system headset 122 and immersive content absorption module 124 .

Example IVAS encoding/decoding systems

[0046] FIG. 2 is a block diagram of a system 200 for encoding and decoding IVAS bitstreams according to one or more implementations. For encoding, the IVAS encoder includes a spatial analysis and downmix module 202 that receives audio data 201 including, but not limited to: mono signals, stereo signals, binaural signals, spatial audio signals (eg, multi-channel spatial audio objects), FoA, higher order ambiophony (HoA) and any other audio data. In some implementations, spatial analysis and downmixing module 202 implements advanced complex communication (CACPL) for analyzing/downmixing stereo/FoA and/or SPAR audio signals to analyze/downmixing FoA audio signals. In other implementations, spatial analysis and downmixing module 202 implements other formats.

[0047] The output of spatial analysis and downmix module 202 includes spatial metadata and 1-N audio downmix channels, where N is the number of input channels. The spatial metadata is input to a quantization and entropy encoding module 203, which quantizes and entropy encodes the spatial data. In some implementations, quantization may include multiple levels of increasingly coarse quantization, such as, for example, fine, average, approximate, and supercoarse quantization strategies, and entropy encoding may include Huffman encoding or arithmetic encoding. Enhanced Voice Services (EVS) encoding module 206 encodes 1-N audio channels into one or more EVS bitstreams.

[0048] In some implementations, EVS encoding module 206 complies with 3GPP TS 26.445 and provides a wide range of functionality, for example, improved quality and coding efficiency for narrowband (EVS-NB) and wideband (EVS-WB) voice services, improved quality using ultra-wideband (EVS-SWB) speech, enhanced quality for mixed content and music in conversational applications, robust to packet loss and latency jitter, and backwards compatible with the AMR-WB codec. In some implementations, EVS encoding module 206 includes a preprocessing and mode selection module that selects between a speech encoder for encoding speech signals and a perceptual encoder for encoding audio signals at a specified bit rate based on mode/bit rate control 207. In some implementations, the speech encoder is an improved variant of algebraically code-excited linear prediction (ACELP), extended with specialized linear prediction (LP)-based modes for various speech classes. In some implementations, the audio encoder is a modified discrete cosine transform (MDCT) encoder with improved performance at low latency/low bit rates and is designed to perform transparent and reliable switching between speech and audio encoders.

[0049] In some implementations, the IVAS decoder includes a quantization and entropy decoding module 204, configured to recover spatial metadata, and EVS decoder(s) 208, configured to recover 1-N channel audio signals. The recovered spatial metadata and audio signals are input to a spatial synthesis/rendering module 209, which synthesizes/renders the audio signals using the spatial metadata for playback on various audio systems 210.

Sample codec IVAS/SPAR

[0050] FIG. 3 is a block diagram of a FoA codec 300 for encoding and decoding FoA in the SPAR format, according to some implementations. The FoA codec 300 includes a FoA SPAR encoder 301, an EVS encoder 305, a FoA SPAR decoder 306, and an EVS decoder 307. The FoA SPAR encoder 301 converts the FoA input signal into a set of downmix channels and parameters used to regenerate the input signal in the FoA SPAR decoder 306. Downmix signals can range from 1 to 4 channels, and parameters include prediction coefficients (PR), cross-prediction coefficients (C), and decorrelation coefficients (P). It should be noted that SPAR is a process used to reconstruct an audio signal from a downmixed version of an audio signal using the PR, C and P parameters, as described in more detail below.

[0051] It should be noted that the exemplary implementation shown in FIG. 3 illustrates a nominal two-channel downmix in which a W (passive predictive) or W' (active predictive) channel is sent with one predictive channel Y' to decoder 306. In some implementations, W may be an active channel. The active W channel allows some mixing of the X, Y, Z channels into the W channel as follows:

W'=W+f*pr _y* Y+f*pr _z* Z+f*pr _x* X,

where f is a constant (eg 0.5) that allows some channels X, Y, Z to be mixed into channel W, and pr _y , pr _x and pr _z are prediction coefficients (PR). In passive W, f=0, so there is no mixing of the X, Y, Z channels into the W channel.

[0052] Cross-prediction coefficients (C) provide the ability to recover some portion of the parametric channels from the residual channels in cases where at least one channel is sent as a residual and at least one is sent parametrically, i.e., for two or three-channel downmixes. For two-channel downmixes (as described in more detail below), the C coefficients allow some of the X and Z channels to be recovered from Y', and the remaining channels are recovered through decorrelated versions of the W channel, as described in more detail below. In the case of a three-channel downmix, Y' and X' are used to restore only Z.

[0053] In some implementations, FoA SPAR encoder 301 includes a passive/active prediction module 302, a remix module 303, and a downmix extract/select module 304. The passive/active prediction module receives FoA channels in 4-channel B format (W, Y, Z, X) and calculates the downmix channels (W, Y', Z', X' representation).

[0054] The downmix extract/select module 304 extracts the SPAR FoA metadata from the working data section of the IVAS bitstream metadata, as described in more detail below. Passive/active prediction module 302 and remix module 303 use SPAR FoA metadata to generate remixed FoA channels (W or W' and A'), which are input to EVS encoder 305 for encoding into an EVS bitstream that is encapsulated in an IVAS bitstream. sent to decoder 306. In this example, it should be noted that the ambiophonic channels in format B are arranged according to the AmbiX convention. However, other notations can also be used, for example the Furs-Malham (FuMa) notation (W, X, Y, Z).

[0055] With respect to the SPAR FoA decoder 306, the EVS bitstream is decoded by the EVS decoder 307, resulting in N_dmx (eg, N_dmx=2) downmix channels. In some implementations, the FoA SPAR decoder 306 performs in reverse order the operations performed by the SPAR encoder 301. For example, in the example of FIG. The 3 remixed FoA channels (W', A', B', C' representation) are reconstructed from the 2 downmix channels using FoA SPAR spatial metadata. The remixed FoA SPAR channels are input to the downmixer 311 to restore the downmix FoA SPAR channels (W', Y', Z', X' representation). The predicted FoA SPAR channels are then input to the inverse prediction module 312 to reconstruct the original unmixed FoA SPAR channels (W, Y, Z, X).

It should be noted that in this two-channel example, decorrelator blocks 309A (dec1) and 309B (dec2) are used to generate decorrelated versions of channel W using a time domain or frequency domain decorrelator. Downmix channels and decorrelated channels are used in combination with FoA SPAR metadata to reconstruct the X and Z channels either completely or parametrically. Block 308 C means multiplying the residual channel by a 2×1 coefficient matrix C, creating two cross-prediction signals that are summed into parametrically reconstructed channels, as shown in FIG. 3. P1 block 310A and P2 block 310B mean multiplying the decorrelator outputs by the columns of the coefficient matrix P 2×2, creating four outputs that are summed into parametrically reconstructed channels, as shown in FIG. 3.

[0056] In some implementations, depending on the number of downmix channels, one of the FoA inputs is sent to FoA SPAR decoder 306 intact (W channel), and one to three other channels (Y, Z, and X) are sent as residuals or completely parametrically to the 306 FoA SPAR decoder. PR coefficients, which remain the same regardless of the number N of downmix channels, are used to minimize the predicted energy in the residual downmix channels. The C coefficients are used to further assist in regenerating fully parametric channels from the residuals. Therefore, C coefficients are not required in single- and quad-channel downmix cases where there are no residual channels or parameterized channels to predict. The P coefficients are used to fill the remaining energy not accounted for by the PR and C coefficients. The number of P coefficients depends on the number N of downmix channels in each frequency band. In some implementations, PR SPAR coefficients (passive W only) are calculated as follows.

[0057] Step 1: Predict all side signals (Y, Z, X) from the main signal W using equation [1].

, [1]

where, as an example, the prediction parameter for the predicted channel Y' is calculated using equation [2].

, [2]

Where are the elements of the input covariance matrix corresponding to signals A and B, and can be calculated on a per-bandwidth basis. Likewise, the residual Z' and X' channels have corresponding prediction parameters, prz and prx. PR is the vector of coefficients forecasting.

[0058] Step 2: Remixing the W signal and the predicted signals (Y', Z', X') from most to least acoustically relevant, where "remixing" means reordering or recombining the signals based on some technology,

. [3]

[0059] One implementation of remixing is to reorder the input signals in W, Y', X', Z', with the assumption that audio cues left and right are more acoustically relevant than front to back, and audio cues from front to back are more acoustically relevant than top-down cues.

[0060] Step 3: Calculate the covariance of the four-channel post-prediction and remix downmix as shown in equations [4] and [5].

[4]

. [5]

where d represents the residual channels (ie the second - N_dmxth channels) and u represents the parametric channels that must be completely reconfigured (ie the (N_dmx+1)th - fourth channels).

[0061] For an example of WABC downmix with 1-4 channels, d and u represent the following channels shown in Table I:

Table I. Representations of d- and u-channels

N d-channels u-channels 1 -- A', B', C' 2 A' B', C' 3 A', B' C' 4 A', B', C' --

[0062] Of primary interest in computing FoA SPAR metadata are the values of R_dd, R_ud and R_uu. From the values of R_dd, R_ud and R_uu, codec 300 determines whether any remaining portion of the full parameter channels can be cross-predicted from the residual channels sent to the decoder. In some implementations, the required additional coefficients C are specified as follows:

[6]

[0063] Therefore, parameter C is of the form (1×2) for three-channel downmix and (2×1) for two-channel downmix.

[0064] Step 4: Calculate the remaining energy in the parameterized channels to be reconstructed by decorrelators 309A, 309B. The residual energy in the upmix channels Res_uu is the difference between the actual (post-prediction) energy R_uu and the regenerated cross-prediction energy Reg_uu.

, [7]

[8]

. [9]

In an embodiment, the square root of the matrix is taken after the normalized matrix Res _uu has off-diagonal elements set to zero. P is also a covariance matrix, thereby being Hermitian symmetric, and as such, only parameters from the upper or lower triangle need to be sent to decoder 306. Diagonal entries are real, while off-diagonal entries may be complex. In an embodiment, the coefficients P may be further divided into diagonal and off-diagonal elements P_d and P_o.

Example IVAS signal chain (FoA or stereo input)

[0065] FIG. 4A is a block diagram of an IVAS signal chain 400 for FoA and stereo audio input signals according to an embodiment. In this example configuration, the audio input to signal chain 400 may be a four-channel FoA audio signal or a two-channel stereo audio signal. The downmixer 401 generates downmix audio channels (dmx_ch) and spatial MDs. The downmix channels are input to a bit rate (BR) allocation module 402, which is configured to quantize spatial MDs and provide mono codec bit rates for the audio downmix channels using a BR allocation control table and the IVAS bit rate, as explained in detail below. The output of the BR distribution module 402 is input to the EVS module 403, which encodes the downmix audio channels into the EVS bitstream. The EVS bitstream and quantized and encoded spatial MDs are input to an IVAS bitstream packetizer 405 to form an IVAS bitstream that is transferred to an IVAS decoder and/or stored for subsequent processing or playback on one or more IVAS devices.

[0066] For stereo input signals, the downmixer 401 is configured to generate a representation of the average signal (M'), residuals (Re) from the stereo signal, and spatial MDs. Spatial MDs include the PR, C, and P coefficients for SPAR and the PR and P coefficients for CACPL, as described in more detail below. The M' signal, Re, spatial MD, and BR allocation control table are input to a BR (bit rate) allocation module 402, which is configured to quantize the spatial metadata and provide mono codec bit rates for the downmix channels using the signal characteristics of the M' and signal BR distribution control tables. The mono codec signal M', Re and BR are input to the EVS module 403, which encodes the M' and Re signal into the EVS bitstream. The EVS bitstream and quantized and encoded spatial MDs are input to an IVAS bitstream packetizer 405 to form an IVAS bitstream that is transferred to an IVAS decoder and/or stored for subsequent processing or playback on one or more IVAS devices.

[0067] For FoA input signals, the downmixer 401 is configured to generate 1-4 FoA downmix channels W', Y', X' and Z' and spatial MDs. Spatial MDs include the PR, C, and P coefficients for SPAR and the PR and P coefficients for CACPL, as described in more detail below. 1-4 channels (W', Y', X', Z') of the FoA downmix are input to a BR allocation module 402, which is configured to quantize spatial MDs and provide mono codec bit rates for the FoA downmix channel(s) using characteristics of the FoA downmix channel(s) signals and the BR distribution control table. The FoA downmix channel(s) are input to the EVS module 403, which encodes the FoA downmix channel(s) into the EVS bitstream. The EVS bitstream and quantized and encoded spatial MDs are input to an IVAS bitstream packetizer 405 to form an IVAS bitstream that is transferred to an IVAS decoder and/or stored for subsequent processing or playback on one or more IVAS devices. The IVAS decoder can reverse the operations performed by the IVAS encoder to recover input audio signals for playback on the IVAS device.

[0068] FIG. 4B is a block diagram of an alternative IVAS signal chain 405 for FoA and stereo audio input signals according to an embodiment. In this exemplary configuration, the audio input to signal chain 405 may be a four-channel FoA audio signal or a two-channel stereo audio signal. In this embodiment, preprocessor 406 extracts signal properties from the input audio signals, such as bandwidth (BW), speech/music classification data, voice activity detection (VAD) data, etc.

[0069] Spatial MD processing module 407 generates spatial MDs from the input audio signal using the extracted signal properties. The input audio signal, signal properties, and spatial MDs are input to a BR allocation module 408, which is configured to quantize the spatial MDs and provide mono codec bit rates for the audio downmix channels using the BR allocation control table and IVAS bit rates described in detail below.

[0070] The input audio signals, quantized spatial MDs, and number of downmix channels (d_dmx) output by the BR distribution unit 408 are input to the downmixer 409, which generates the downmix channel(s). For example, for FoA signals, the downmix channels may include W' and N_dmx-1 residues (Re).

[0071] The EVS bit rates output by the BR distribution module 408 and the downmix channel(s) are input to the EVS module 410, which encodes the downmix channel(s) into the EVS bitstream. The EVS bitstream and quantized encoded spatial MDs are input to an IVAS bitstream packetizer 411 to form an IVAS bitstream that is transferred to an IVAS decoder and/or stored for subsequent processing or playback on one or more IVAS devices. The IVAS decoder can reverse the operations performed by the IVAS encoder to recover input audio signals for playback on the IVAS device.

Exemplary Bit Rate Allocation Control Strategy

[0072] In an embodiment, the IVAS bit rate allocation control strategy includes two components. The first component is the BR allocation control table, which provides the initial conditions for the BR allocation control process. The index into the BR distribution control table is determined through the codec configuration parameters. Codec configuration parameters may include IVAS bit rate, input format such as stereo, FoA, flat FoA or any other format, audio bandwidth (BW), spatial encoding mode (or number N _re of residual channels), mono codec priority, and spatial MD. For stereo encoding, N _re =0 corresponds to the full parameter (FP) mode, and N _re =1 corresponds to the mean residual (MR) mode. In an embodiment, the allocation control table index BR indicates the target, minimum and maximum mono codec bit rates for each of the downmix channels and a plurality of quantization strategies (eg, fine, average, approximate) for encoding spatial MDs. In another embodiment, the allocation control table index BR indicates the overall target and minimum bit rate for all mono codec instances, the ratio with which the available bit rate should be divided among all downmix channels, and several quantization strategies to encode spatial MDs. The second component of the IVAS bitrate control strategy is a process that uses the outputs of the BR allocation control table and the properties of the input audio signals to determine the quantization levels and bitrates of the spatial metadata, as well as the bitrate of each downmix channel, as described in Ref. fig. 5A and 5B.

Bit rate distribution process: overview

[0073] The main processing components of the bit rate distribution processes disclosed herein include:

- audio bandwidth (BW) detection (e.g., narrow band (NB), wide band (WB), ultra wide band (SWB), full band (FB)). At this stage, the BW of the average or signal W is detected and the metadata is quantized accordingly. The EVS then treats the IVAS BW as an upper limit and encodes the downmix channels accordingly;

- extracting properties of input audio signals (for example, speech or music);

- selection of spatial coding mode (for example, full-parameter (FP), average-residual (MR)) or the number of residual channels, N_re, while for stereo coding the FP mode is selected if N_re=0, and the MR mode is selected if N_re=1;

- mono codec and spatial MD priority decision: target bit rate, minimum and maximum bit rates for each downmix channel, or the ratios at which the total mono codec bit rate should be shared between downmix channels.

Detection of BW audio signal

[0074] This component detects the BW of the average or W signal. In an embodiment, the IVAS codec uses the EVS BW detector described in EVS TS 26.445.

Extracting properties of input signals

[0075] This component classifies each frame of the input audio signal as speech or music. In an embodiment, the IVAS codec uses the EVS speech/music classifier as described in EVS TS 26.445.

Decision on mono codec priority relative to spatial MDs

[0076] This component determines the priority of the mono codec over spatial MDs based on the properties of the downmix signals. Examples of downmix signal properties include speech or music as determined by speech/music classifier data, and midband (M-S) band covariance estimates for stereo and W-Y, W-X, W-Z channel band covariance estimates for FoA. The speech/music classifier data can be used to give higher priority to the mono codec when the input audio signal is music, and covariance estimates can be used to give higher priority to spatial MD when the input audio signal is hard panned.

[0077] In an embodiment, a priority decision is calculated for each frame of the input audio signal. For a given IVAS bit rate, BW average or W signal and input configuration, the bit rate distribution starts with the target or required bit rates for the downmix channels (for example, the mono codec bit rate is determined according to the subjective or objective assessment) present in the table BR distribution control, and the most accurate quantization strategies for metadata. If the initial condition is not within the given IVAS bit rate budget, then the mono codec bit rate or the spatial MD quantization level, or both, is reduced iteratively in the quantization loop based on their respective priorities until they are both within the budget by IVAS bit rates.

Distribution of bit rates between downmix channels

Full-parametric versus mean-residual

[0078] In FP mode, only the -channel M' or W' is encoded by a mono codec, and additional parameters are encoded in spatial MDs indicating the level of the residual channel or the level of decorrelation to be added by the decoder. For bit rates at which FP and MR are practical, the IVAS BR allocation process dynamically selects the number of residual channels to be encoded by a mono codec and transmitted/streamed to the decoder, based on spatial MDs on a frame-by-frame basis. If the level of any residual channel is above the threshold, then that residual channel is encoded using a mono codec; otherwise, the process runs in FP mode. Transient frame processing is performed to reset codec state buffers when the number of residual channels to be encoded by the mono codec changes.

Bit rate distribution for MR downmixing

[0079] Eavesdropping evaluation is performed with different input signals and bit rate distributions between the middle channel and the residual channel. Based on focused listening tests, the most efficient mid-to-residual bit rate ratio is 3:2. However, other relationships may be used based on the application case requirements. In an embodiment, the bit rate distribution uses a fixed ratio that is further adjusted during the setup phase. During the iterative process of selecting a quantization strategy and BR for downmix channels, the BR for each downmix channel is modified according to this ratio.

[0080] In an embodiment, instead of maintaining a fixed ratio between the bit rates of the downmix channel, the target bit rate and the minimum and maximum bit rates for each downmix channel are separately listed in the BR allocation control table. These bit rates are selected based on careful subjective and objective evaluations. During the iterative process of selecting the quantization strategy and BR for the downmix channels, bits are added to or extracted from the downmix channels based on the priority of all the downmix channels. The priority of downmix channels can be fixed or dynamic on a frame-by-frame basis. In an embodiment, the priority of the downmix channels is fixed.

Bit Rate Allocation Process: Process Flow

[0081] FIG. 5A is a flow diagram of a process 500 for allocating bit rates for input stereo and FoA signals, according to an embodiment. Inputs to process 500 are IVAS bitrate, constants (eg, bitrate distribution control table, IVAS bitrate), downmix channels, spatial MDs, input format (eg, stereo, FoA, flat FoA), and force parameters command line (eg maximum bandwidth, encoding mode, EVS backwards compatible mode for mono downmix). The outputs of process 500 are the EVS bit rate for each downmix channel, metadata quantization levels, and encoded metadata bits. The following steps are performed as part of process 500.

Extracting Audio Downmix Features

[0082] At step 501, the following signal properties are extracted from the input audio signal: bandwidth (e.g., narrowband, wideband, ultra-wideband, fullband) and speech/music classification data, voice activity detection (VAD) data. . Bandwidth (BW) is the user-specified minimum of the actual input audio bandwidth and the maximum command-line bandwidth. In an embodiment, the downmix audio signal may be in a pulse code modulation (PCM) format.

Defining a table index

[0083] At step 502, process 500 retrieves the IVAS bit rate distribution control table indexes from the IVAS bit rate distribution control table using the IVAS bit rate. At step 503, process 500 determines the input format table indexes based on the signal parameters extracted at step 501 (i.e., BW and speech/music classification), the format of the input audio signals, the IVAS bit rate distribution control table indexes extracted at step 502 , and EVS backwards compatible mode for mono downmixing. At step 504, process 500 selects a spatial encoding mode (i.e., FP or MR) or number of residual channels (i.e., N_re=0-3) based on bitrate control table indices, transient audio encoding mode, and spatial M.D. At step 505, process 500 determines the final fine table index based on the six parameters described above. In an embodiment, the audio spatial encoding mode selection at step 504 is based on the residual channel level indicator in the spatial MD. The spatial audio encoding mode indicates either an MR encoding mode, in which the middle or W channel representation (M' or W') is accompanied by one or more residual channels in the downmixed audio signal, or an FP encoding mode, in which only the average or W channel representation (M' or W') is present in the downmixed audio signal. In an embodiment, the transient audio encoding mode is set to 1 if the spatial audio encoding mode in the previous frame includes residual channel encoding while the current frame requires only M' or W' channel encoding. Otherwise, the transient audio encoding mode is set to 0. If the number of residual channels to be encoded differs between the current frame and the previous frame, the transient audio encoding mode is set to 1.

Mono codec priority and spatial MD calculations

[0084] At step 506, process 500 determines mono codec/spatial MD priority based on properties of the input audio signals extracted in step 1 and covariance estimates in the midband or W-Y, W-X, W-Z channel, or W-Y frequency bands , W-X, W-Z channel. In an embodiment, there are four possible priority outcomes: high mono codec priority and low priority spatial MDs, low mono codec priority and high priority spatial MDs, high mono codec priority and high priority spatial MDs; and low priority mono codec and low priority spatial MD.

Retrieving Mono Codec Bit Rate Related Variables from a Table

[0085] At step 507, the following parameters are read from the table entry pointed to by the final table index calculated at step 505: target mono codec bit rate (EVS), bit rate ratio, minimum EVS bit rate, and bit rate deviation steps EVS bits. The actual mono codec bit rate (EVS) may be higher or lower than the target mono codec bit rate (EVS) indicated in the BR allocation control table, depending on the mono codec/spatial MD priority determined at step 506 and the spatial MD bit rate with different quantization levels. The bit rate ratio indicates the ratio with which the full EVS bit rate should be distributed among the input audio signal transmission channels. The minimum EVS bit rate is a value below which the full EVS bit rate is not permitted to be reduced. EVS bit rate deviation steps are steps to reduce the target EVS bit rate when the EVS priority is higher than or equal to or lower than the priority of the spatial MDs.

Calculate the best EVS bit rate and metadata quantization level based on input parameters

[0086] At step 508, the optimal EVS bit rate and metadata quantization strategy is calculated based on the input parameters obtained in steps 501-503, according to the following sub-steps. The high bit rate of the downmix channels and coarse quantization strategy can lead to spatial problems, while the fine quantization strategy and low bit rate of the downmix audio channel can lead to mono codec encoding artifacts. "Optimal" as used herein represents the most balanced allocation of IVAS bitrate between the EVS bitrate and the metadata quantization level while using all available bits in the IVAS bitrate budget or at least significantly reducing bit loss.

[0087] Step 508.1: Quantize the metadata to the finest quantization level and test condition 508.a (shown below). If condition 508.a is true, then execute step 508.b (shown below). Otherwise, proceed to step 508.2 or 508.3 or 508.4 based on the priorities calculated in step 503.

[0088] Step 508.2: If the EVS priority is high and the spatial MD priority is low, then reduce the spatial MD quantization level and check condition 508.a. If condition 508.a is true, then execute step 508.b. Otherwise, reducing the target EVS bit rate based on step 507 (EVS bit rate deviation steps) and checking condition 508.a. If condition 508.a is true, then execute step 508.b, otherwise repeat step 508.2.

[0089] Step 508.3: If the EVS priority is low and the spatial MD priority is high, then reduce the target EVS bitrate based on step 507 (EVS bitrate rejection steps) and check condition 508.a. If condition 508.a is true, then execute step 508.b. Otherwise, reduce the spatial MD quantization level and check condition 508.a. If condition 508.a is true, then execute step 508.b. Otherwise, repeat step 508.3.

[0090] Step 508.4: If the EVS priority is equal to the priority of the spatial MDs, then reduce the target EVS bitrate based on step 507 (EVS bitrate rejection steps) and check condition 508.a. If condition 508.a is true, then execute step 508.b. Otherwise, reduce the spatial metadata quantization level and check condition 508.a. If condition 508.a is true, then execute step 508.b, otherwise repeat step 5.4.

[0091] Condition 508.a mentioned above checks whether the sum of the metadata bit rate, the target EVS bit rate, and overhead bits is less than or equal to the IVAS bit rate.

[0092] Step 508.b mentioned above calculates the EVS bit rate as equal to the IVAS bit rate minus the metadata bit rate minus overhead bits. The EVS bit rate is then distributed among the downmix audio channels according to the bit rate ratio mentioned in step 507.

[0093] If the minimum target EVS bitrate and the most approximate quantization level do not fit into the IVAS bitrate budget, then the bitrate allocation process 500 is performed at a lower bandwidth.

[0094] In an embodiment, table index and metadata quantization level information are included in the overhead bits of the IVAS bitstream sent to the IVAS decoder. The IVAS decoder reads the table index and metadata quantization level from the overhead bits in the IVAS bitstream and decodes the spatial MD. This leaves the IVAS decoder to only process the EVS bits in the IVS bitstream. The EVS bits are divided among the input audio transmission channels according to the ratio indicated by the table index (step 508.b). Each instance of the EVS decoder is then called with the appropriate bits, resulting in the reconstruction of the downmix audio channels.

Example IVAS Bit Rate Allocation Control Table

[0095] The following is an example IVAS bit rate distribution control table (Table II). The following parameters shown in Table II have the meanings given below:

[0096] Input Format: Stereo - 1, Flat FoA - 2, FoA - 3

[0097] BW: NB - 0, WB - 1, SWB - 2, FB - 3

[0098] Enabled spatial coding tool: FP - 1, MR - 2

[0099] Transition mode: 1 → MR/FP transition, 0 → otherwise

[00100] Backward compatible mode for mono downmix: 1 → if the middle channel should be 3GPP EVS compliant, 0 → otherwise.

Table II. Sample IVAS bit rate distribution table

IVAS BR (Kbit/s) Input Format B.W. Audio Spatial Mode Transitional mode Backward compatible mode for mono downmixing Target EVS BR (bits/
With) BR ratio Minimum EVS BR (bits/
With) EVS BR deviation steps (bits/
With) 16.4 1 1 1 0 0 11400 (10) 9000 (200, 400, 800) 16.4 1 2 1 0 0 11400 (10) 9000 (200, 400, 800) 16.4 1 2 1 0 1 9600 (10) 9600 (0, 0, 0) 24.4 1 1 1 0 0 19200 (10) 16400 (200, 400, 800) 24.4 1 1 2 0 0 19200 (3, 2) 16400 (50, 100, 200) 24.4 1 1 1 1 0 19200 (3, 2) 16400 (50, 100, 200) 24.4 2 1 1 0 0 16400 (100) 13200 (200, 400, 800) 24.4 1 2 1 0 0 19200 (10) 16400 (200, 400, 800) 24.4 1 2 2 0 0 19200 (3, 2) 16400 (50, 100, 200) 24.4 1 2 1 1 0 19200 (3, 2) 16400 (50, 100, 200) 24.4 1 2 2 0 1 19200 (eleven) 19200 (0, 0, 0) 24.4 2 2 1 0 0 16400 (100) 13200 (200, 400, 800) 24.4 2 2 1 0 1 13200 (100) 13200 (0, 0, 0) 24.4 1 3 1 0 0 19200 (10) 16400 (200, 400, 800) 32 1 1 2 0 0 28000 (3, 2) 24400 (50, 100, 200) 32 2 1 1 0 0 23200 (100) 19200 (400, 800, 1200) 32 3 1 1 0 0 20800 (1, 0, 0, 0) 16400 (400, 800, 1200) 32 1 2 1 0 0 28000 (10) 24400 (400, 800, 1200) 32 1 2 2 0 0 28000 (3, 2) 24400 (50, 100, 200) 32 1 2 2 0 1 26000 (41, 24) 26000 (0, 0, 0) 32 1 2 1 1 0 28000 (3, 2) 24400 (50, 100, 200) 32 2 2 1 0 0 26600 (100) 25200 (400, 800, 1200) 32 2 2 2 0 0 26600 (3, 2, 2) 25200 (50, 100, 200) 32 2 2 1 0 1 16400 (100) 16400 (0, 0, 0) 32 2 2 1 1 0 26600 (3, 2, 2) 25200 (50, 100, 200) 32 3 2 1 0 0 20800 (1, 0, 0, 0) 16400 (400, 800, 1200) 32 1 3 1 0 0 26000 (10) 23200 (400, 800, 1200) 32 2 3 1 0 0 26400 (100) 23200 (400, 800, 1200) 48 1 1 2 0 0 44000 (3, 2) 40000 (100, 200, 400) 48 2 1 2 0 0 40000 (3, 2, 2) 36000 (100, 200, 400) 48 3 1 2 0 0 39600 (3, 2, 2, 2) 34200 (100, 200, 300) 48 1 2 2 0 0 44000 (3, 2) 40000 (100, 200, 400) 48 1 2 2 0 1 40800 (61, 41) 40800 (0, 0, 0) 48 2 2 2 0 0 40000 (3, 2, 2) 36000 (100, 200, 400) 48 2 2 2 0 1 35600 (41, 24, 24) 35600 (0, 0, 0) 48 3 2 1 0 0 34000 (1, 0, 0, 0) 30000 (600, 1000, 1600) 48 3 2 1 0 1 24400 (1, 0, 0, 0) 24400 (0, 0, 0) 48 1 3 1 0 0 44000 (10) 40000 (600, 1000, 1600) 48 1 3 2 0 0 44000 (3, 2) 40000 (100, 200, 400) 48 1 3 1 1 0 44000 (3, 2) 40000 (100, 200, 400) 48 2 3 1 0 0 39200 (100) 35200 (600, 1000, 1600) 48 3 3 1 0 0 34000 (1, 0, 0, 0) 30000 (600, 1000, 1600) 64 1 1 2 0 0 60000 (3, 2) 56000 (100, 200, 400) 64 2 1 2 0 0 57400 (3, 2, 2) 52500 (100, 200, 400) 64 3 1 2 0 0 52000 (3, 2, 2, 2) 45000 (100, 200, 300) 64 1 2 2 0 0 60000 (3, 2) 56000 (100, 200, 400) 64 1 2 2 0 1 48800 (eleven) 48800 (0, 0, 0) 64 2 2 2 0 0 57400 (3, 2, 2) 52200 (100, 200, 400) 64 2 2 2 0 1 50800 (61, 33, 33) 50800 (0, 0, 0) 64 3 2 2 0 0 52000 (3, 2, 2, 2) 45000 (100, 200, 300) 64 3 2 2 0 1 45200 (41, 24, 24, 24) 45200 (0, 0, 0) 64 1 3 2 0 0 60000 (3, 2) 56000 (100, 200, 400) 64 2 3 1 0 0 57400 (100) 52500 (800, 1200, 2000) 64 2 3 2 0 0 57400 (3, 2, 2) 52500 (100, 200, 400) 64 2 3 1 1 0 57400 (3, 2, 2) 52500 (100, 200, 400) 64 3 3 1 0 0 48000 (1, 0, 0, 0) 40000 (800, 1200, 2000) 96 1 1 2 0 0 90000 (3, 2) 86000 (200, 400, 600) 96 2 1 2 0 0 86000 (3, 2, 2) 78000 (200, 300, 400) 96 3 1 2 0 0 84000 (3, 2, 2, 2) 76000 (100, 200, 300) 96 1 2 2 0 0 90000 (3, 2) 86000 (200, 400, 600) 96 1 2 2 0 1 88000 (6, 5) 88000 (0, 0, 0) 96 2 2 2 0 0 86000 (3, 2, 2) 78000 (200, 300, 400) 96 2 2 2 0 1 80800 (80, 61, 61) 80800 (0, 0, 0) 96 3 2 2 0 0 84000 (3, 2, 2, 2) 76000 (100, 200, 300) 96 3 2 2 0 1 81200 (80, 41, 41, 41) 81200 (0, 0, 0) 96 1 3 2 0 0 90000 (3, 2) 86000 (200, 400, 600) 96 2 3 2 0 0 86000 (3, 2, 2) 78000 (200, 300, 400) 96 3 3 1 0 0 84000 (1, 0, 0, 0) 76000 (1000, 2000, 3000) 96 3 3 2 0 0 84000 (3, 2, 2, 2) 76000 (100, 200, 300) 96 3 3 1 1 0 84000 (3, 2, 2, 2) 76000 (100, 200, 300) 128 1 1 2 0 0 122000 (3, 2) 118000 (200, 400, 600) 128 2 1 2 0 0 118000 (3, 2, 2) 110000 (200, 300, 400) 128 3 1 2 0 0 116000 (3, 2, 2, 2) 108000 (100, 200, 300) 128 1 2 2 0 0 122000 (3, 2) 118000 (200, 400, 600) 128 2 2 2 0 0 118000 (3, 2, 2) 110000 (200, 300, 400) 128 3 2 2 0 0 116000 (3, 2, 2, 2) 108000 (100, 200, 300) 128 1 3 2 0 0 122000 (3, 2) 118000 (200, 400, 600) 128 2 3 2 0 0 118000 (3, 2, 2) 110000 (200, 300, 400) 128 3 3 2 0 0 116000 (3, 2, 2, 2) 108000 (100, 200, 300) 256 1 1 2 0 0 248000 (3, 2) 244000 (400, 800, 1000) 256 2 1 2 0 0 244000 (3, 2, 2) 236000 (300, 500, 800) 256 3 1 2 0 0 240000 (3, 2, 2, 2) 232000 (300, 400, 600) 256 1 2 2 0 0 248000 (3, 2) 244000 (400, 800, 1000) 256 2 2 2 0 0 244000 (3, 2, 2) 236000 (300, 500, 800) 256 3 2 2 0 0 240000 (3, 2, 2, 2) 232000 (300, 400, 600) 256 1 3 2 0 0 248000 (3, 2) 244000 (400, 800, 1000) 256 2 3 2 0 0 244000 (3, 2, 2) 236000 (300, 500, 800) 256 3 3 2 0 0 240000 (3, 2, 2, 2) 232000 (300, 400, 600)

[00101] Also in FIG. 5A shows the IVAS bitstream. In an embodiment, the IVAS bitstream includes a fixed length IVAS common header (CH) 509 and a variable length common tool header (CTH) 510. In an embodiment, the bit length of the CTH section is calculated based on the number of entries corresponding to a given IVAS bit rate in the IVAS bit rate allocation control table. The relative table index (the offset from the first index for that IVAS bit rate in the table) is stored in the CTH section. When operating in backward compatible mode for mono downmixing, CTH 510 is followed by EVS work data 511, followed by spatial MD work data 513. When operating in IVAS mode, CTH 510 is followed by spatial MD operating data 512, followed by EVS operating data 514. In other embodiments, the order may be different.

Sample Processes

[00102] An exemplary bit rate distribution process may be performed by an IVAS codec or an encoding/decoding system including one or more processors executing instructions stored on a non-transitory computer readable storage medium.

[00103] In an embodiment, the audio encoding system receives audio input and metadata. The system determines, based on the audio input, metadata, and parameters of the IVAS codec used in encoding the audio input, one or more bitrate distribution control table indexes, the parameters including an IVAS bitrate, an input format, and a backwards compatibility mono mode, wherein one or more The indices include the audio spatial encoding mode and the audio input bandwidth.

[00104] The system searches the bit rate distribution control table based on the IVAS bit rate, the input format, the audio spatial coding mode, and one or more indexes, the search identifying an entry in the bit rate distribution control table, the entry including target EVS bit rate, bit rate ratio, minimum EVS bit rate, and representation of EVS bit rate deviation steps.

[00105] The system passes the identified recording to a bit rate calculation process, which is programmed to determine the bit rates of audio inputs (eg, downmix channels), metadata bit rates, and metadata quantization levels. The system provides downmix channel bit rates and at least one of metadata bit rates or metadata quantization levels to the downstream IVAS device.

[00106] In some implementations, the system may extract properties from the audio input, where the properties include an indicator of whether the audio input is speech or music and the bandwidth of the audio input. The system determines, based on properties, the priority between the bit rate of the downmix channels and the metadata bit rate. The system provides priority to the bit rate calculation process.

[00107] In some implementations, the system extracts one or more parameters including the residual level (side channel prediction errors) from the spatial MDs. The system determines, based on the parameters, an audio spatial coding mode that indicates the need for one or more residual channels in the IVAS bitstream. The system provides a spatial audio encoding mode into the bit rate calculation process.

[00108] In some implementations, the bit rate distribution control table index is stored in the common tool header (CTH) of the IVAS bitstream.

[00109] The audio decoding system is configured to receive the IVAS bitstream. The system determines, based on the IVAS bit stream, the IVAS bit rate and the bit rate distribution control table indexes. The system searches the bit rate distribution control table based on the table indexes and retrieves the input format, spatial encoding mode, backward compatibility mono mode and one or more indexes, the target EVS bit rate and the bit rate ratio. The system extracts and decodes downmix audio bits per downmix channel and spatial MD bits. The system transmits the extracted downmix signal bits and spatial MD bits to the downstream IVAS device. The downstream IVAS device may be an audio processing device or a data storage device.

FoA SPAR Bit Rate Allocation Process

[00110] In an embodiment, the bit rate distribution process described above for stereo input signals can also be modified and applied to the FoA SPAR bit rate distribution using the FoA SPAR bit rate distribution control table shown below. Definitions for the terms included in the table are provided below to assist readers, in accordance with the FoA SPAR Bit Rate Allocation Control table.

- Target number of metadata bits (MDtar)=IVAS_bits-header_bits-evs_target_bits (EVStar)

- Maximum number of metadata bits (MDmax)=IVAS_bits-header_bits-evs_minimum_bits (EVSmin)

- The target number of metadata bits should always be less than "MDmax".

Table III. Example FoA SPAR Bit Rate Allocation Control Table

IVAS BR (Kbit/s) B.W. N_dmx Remix line Active W Complex flag transient mode with dmx switching (filler) (Target, Min, Max) EVS BR (Kbps) MD Quantization Levels
Target
Recovery 1 after a failure
Recovery 2 after a failure
(Notation: [PR, C, P_d, P_o]) Muting the TD Decorrelator (Target, maximum) MD BR (Kbps) MD BR (Kbit/s) worst case for recovery 2 with radix 2 encoding; encoding for real coefficients including 0.4 Kbps header 32 3 1 WYXZ 1 0 0 W':
(24, 20.45, 31.95) T: [21,1,5,1]
F1: [15,1,5,1]
F2: [15,1,3,1] 0 (8, 11.55) 11.2 64 3 2 WYXZ 0 0 0 W:(38, 34.05, 56)
Y': (16, 15.60, 20.40) T: [21,7,5,1]
F1: [15,7,5,1]
F2: [15,7,3,1] 1 (10, 14.35) 13.6 96 3 3 WYXZ 0 0 0 W: (47, 42.60, 56)
Y': (23, 22.6, 31.95;
X': (16, 15.60, 20.4) T: [21,9,9,1]
F1: [21,7,5,1]
F2: [21,7,5,1] 1 (10, 15.2) 14.8 160 3 3 WYXZ 0 0 0 W: (74, 70.9, 112)
Y': (41, 40.05, 56)
X': (35, 34.05, 56) T: [21,11,11,1]
F1: [21,9,9,1]
F2: [21,7,7,1] 1 (10, 15) 14.8 256 3 4 WYXZ 0 0 0 W: (90, 90, 112)
Y': (70, 70, 112)
X': (50, 50, 56)
Z': (36.6, 36.6, 56) T: [31,1,1,1]
F1: [31,1,1,1]
F2: [31,1,1,1] 1 (9.0, 9.4) 9.4

[00111] Some example calculations of maximum MD bit rates (actual coefficients) are shown in Table IV below.

Table IV. Calculations of maximum MD bit rates (real coefficients)

N_dmx Number of spatial parameters Quantization levels → bits Calculation: Number of parameters*bits*50 Maximum BR (bit/s) PR C P_d P_o 1 36 0 36 36 [15,1,3,1] → (4,0,2,0) (4*36+0+2*36+0)*50 10800 2 36 24 24 12 [15,7,3,1] → (4,3,2,0) (4*36+3*24+2*24+0)*50 13200 3 36 24 12 0 [21,7,7,1] → (5,3,3,0) (5*36+3*24+3*12+0)*50 14400 4 36 0 0 0 [31,1,1,1] → (5,0,0,0) 5*36*50 9000

Approximate metadata quantization outline

[00112] In an embodiment, the metadata quantization loop is implemented as described below. The metadata quantization loop includes two threshold values (defined above): MDtar and MDmax.

[00113] Step 1: For each frame of the input audio signal, the MD parameters are quantized in a non-differential time manner and encoded using an arithmetic encoder. The actual metadata bit rate (MDact) is calculated based on the encoded MD bits. If MDact is lower than MDtar, then this step is treated as a pass and the process exits the quantization loop and the MDact bits are integrated into the IVAS bitstream. Additional available bits (MDtar-MDact) are supplied to the encoder with mono codec support (EVS) to increase the significant bit rate of the downmix audio channels. The higher bit rate allows more information to be encoded through a mono codec, and the decoded audio output should have comparatively less loss.

[00114] Step 2: If Step 1 fails, then a subset of MD parameter values in the frame is quantized and then subtracted from the quantized MD parameter values in the previous frame, and the differential quantized parameter value is encoded using an arithmetic (i.e., time differential) encoder coding). MDact is calculated based on the encoded MD bits. If MDact is lower than MDtar, then this step is treated as a pass and the process exits the quantization loop and the MDact bits are integrated into the IVAS bitstream. Additional available bits (MDtar-MDact) are supplied to the encoder with mono codec support (EVS) to increase the significant bit rate of the downmix audio channels.

[00115] Step 3: If Step 2 fails, then the bit rate (MDact) of the quantized MD parameters is calculated without entropy.

[00116] Step 4: The MDact bit rate values calculated in steps 1-3 are compared with MDmax. If the minimum MDact bit rates calculated in stage 1, stage 2 and stage 3 are within MDmax, then this stage is treated as a pass and the process exits the quantization loop and the MD bitstream with the minimum MDact is integrated into the IVAS bitstream. If MDact is higher than MDtar, then the bits (MDact-MDtar) are extracted from the encoder with mono codec support (EVS).

[00117] Step 5: If step 4 fails, the parameters are quantized more approximately and the above steps are repeated as the first failover strategy (failover 1).

[00118] Step 6: If Step 5 fails, the parameters are quantized using a quantization scheme that is guaranteed to fit within the MDmax limits as a second failover strategy (failover 2).

[00119] After all the iterations mentioned above, it is guaranteed that the metadata bit rate must fit within the MDmax limits, and the encoder must generate the actual metadata bits or MDact.

EVS bit rate distribution for downmix channels (EVSbd)

[00120] In an embodiment, actual EVS bits (EVSact)=IVAS_bits-header_bits-MDact. If "EVSact" is less than "EVStar", then bits are extracted from the EVS channels in the following order: (Z, X, Y, W). The maximum number of bits that can be extracted from any channel is EVStar(ch) minus EVSmin(ch). If "EVSact" is greater than "EVStar", then all extra bits are assigned to the downmix channels in the following order: W, Y, X and Z. The maximum number of extra bits that can be added to any channel is EVSmax(ch)-EVStar(ch ).

Unpacking via SPAR decoder

[00121] In an embodiment, the SPAR decoder decompresses the IVAS bitstream as follows:

1. Obtain the IVAS bit rate from the bit length and obtain the table index from the tool header (CTH) in the IVAS bitstream.

2. Parse the header/metadata bits in the IVAS bitstream.

3. Parsing and dequantization of metadata bits.

4. Setting "EVSact"=remaining length in bits.

5. Read the table entries associated with the target, minimum and maximum EVS bit rates and repeat the "EVSbd" step in the decoder to obtain the actual EVS bit rate for each channel.

6. Decoding EVS channels and upmixing into FoA channels.

BR Allocation Process for FoA SPAR Audio Inputs

[00122] FIG. 5B and 5C are a flow diagram of a process 515 for allocating bit rates for FoA SPAR input signals, according to an embodiment. Process 515 begins by preprocessing 517 the FoA (W, Y, Z, X) input 516 to extract signal properties using the IVAS bit rate, such as BW, speech/music classification data, VAD data, etc. . Process 515 continues by generating spatial MDs 518 (eg, coefficients PR, C, P) and selecting the number of residual channels to be sent to the IVAS decoder based on the residual level indicator in the spatial MDs (520), and obtaining a control table index BR- distribution based on the IVAS bit rate, BW and the number of downmix channels (N_dmx) (521). In some embodiments, P coefficients in spatial MDs may serve as an indicator of the residual level. The BR allocation control table index is sent to the IVAS bit packetizer (see FIGS. 4A, 4B) for inclusion in the IVAS bitstream, which can be stored and/or sent to the IVAS decoder.

[00123] Process 515 continues by reading the SPAR configuration from the row in the BR allocation control table pointed to by the table index (521). As shown in Table III above, the SPAR configuration is specified by one or more features including, but not limited to: downmix line (remix), active W flag, complex spatial MD flag, spatial MD quantization strategies, min/target/max EVS bit rates and time domain decorrelator mute flag.

[00124] Process 515 continues by determining the bit rates MDmax, MDtar from the IVAS bit rate, EVSmin bit rate values, and EVStar bits (522) as described above, and entering a quantization loop that includes quantizing the spatial MDs non-differentially time in a manner using a quantization strategy, encoding the quantized spatial MDs using an entropy encoder (eg, an arithmetic encoder), and calculating MDact (523). In an embodiment, the first iteration of the quantization loop uses an exact quantization strategy.

[00125] Process 515 continues by checking whether MDact is less than or equal to MDtar (524). If MDact is less than or equal to MDtar, then the MD bits are sent to the IVAS bit packetizer for inclusion in the IVAS bitstream, and the (MDtar-MDact) bits are summed with the EVStar bit rates (532) in the following order: W, Y, X, Z; N_dmx EVS bitstreams (channels) are generated, and the EVS bits are sent to the IVAS bit packetizer for inclusion in the IVAS bitstream as described above. If MDact is not less than or equal to MDtar, then process 515 quantizes the spatial MDs in a time-differential manner using an exact quantization strategy, encodes the quantized spatial MDs using an entropy encoder, and computes the MDact again (525). If MDact is less than or equal to MDtar, then the MD bits are sent to the IVAS bit packetizer for inclusion in the IVAS bitstream, and the (MDtar-MDact) bits are summed with the EVStar bit rates (532) in the following order: W, Y, X, Z ; N_dmx EVS bitstreams (channels) are generated, and the EVS bits are sent to the IVAS bit packetizer for inclusion in the IVAS bitstream as described above. If MDact is greater than MDtar, the spatial MDs are quantized in a non-time-differential manner using an exact quantization strategy and are entropy encoded and base 2 encoded, and a new value for MDact is computed (527). It should be noted that the maximum number of bits that can be summed with any EVS instance is EVSmax-EVStar.

[00126] Process 515 again determines whether MDact is less than or equal to MDtar (528). If MDact is less than or equal to MDtar, then the MD bits are sent to the IVAS bit packetizer for inclusion in the IVAS bitstream, and the (MDtar-MDact) bits are summed with the EVStar bit rates (532) in the following order: W, Y, X, Z ; N_dmx EVS bitstreams (channels) are generated, and the EVS bits are sent to the IVAS bit packetizer for inclusion in the IVAS bitstream as described above. If MDact is greater in MDtar, then process 515 sets MDact as the minimum of the three MDact bit rates calculated at (523), (525), (527), and compares MDact with MDmax (529). If MDact is greater than MDmax (530), the quantization loop (steps 523-530) is repeated using the coarse quantization strategy as described above.

[00127] If MDact is less than or equal to MDmax, then the MD bits are sent to the IVAS bit packetizer for inclusion in the IVAS bitstream, and process 515 again determines whether MDact is less than or equal to MDtar (531). If MDact is less than or equal to MDtar, then the (MDtar-MDact) bits are summed with the EVStar bit rates (532) in the following order: W, Y, X, Z; N_dmx EVS bitstreams (channels) are generated, and the EVS bits are sent to the IVAS bit packetizer for inclusion in the IVAS bitstream as described above. If MDact is greater than MDtar, then (MDtar-MDact) bits are subtracted from the EVStar bit rates (532) in the following order: Z, X, Y, W; N_dmx EVS bitstreams (channels) are generated, and the EVS bits are sent to the IVAS bit packetizer for inclusion in the IVAS bitstream as described above. It should be noted that the maximum number of bits that can be subtracted from any EVS instance is EVStar-EVSmin.

Sample Processes

[00128] FIG. 6 is a flow diagram of an IVAS encoding process 600, according to an embodiment. Process 600 may be implemented using a device architecture as described with respect to FIG. 8.

[00129] Process 600 includes receiving an input audio signal (601), downmixing the input audio signal into one or more downmix channels, and spatial metadata associated with one or more channels of the input audio signal (602); reading a set of one or more bit rates for the downmix channels and a set of quantization levels for spatial metadata from the bit rate distribution control table (603); determining a combination of one or more bit rates for the downmix channels (604); determining a metadata quantization level from a set of metadata quantization levels using a bit rate distribution process (605); quantizing and encoding spatial metadata using a metadata quantization layer (606); generating, using a combination of one or more bit rates, a downmix bitstream for one or more downmix channels (607); combining the downmix bitstream, quantized and encoded spatial metadata, and a set of quantization levels into an IVAS bitstream (608); and streaming or storing the IVAS bitstream for playback on an IVAS-enabled device (609).

[00130] FIG. 7 is a flow diagram of an alternative IVAS encoding process 700, according to an embodiment. Process 700 may be implemented using a device architecture as described with respect to FIG. 8.

[00131] Process 700 includes receiving an input audio signal (701); extracting properties of the input audio signal (702); calculating spatial metadata for the input audio signal channels (703); reading a set of one or more bit rates for the downmix channels and a set of quantization levels for spatial metadata from the bit rate distribution control table (704); determining a combination of one or more bit rates for the downmix channels (705); determining a metadata quantization level from a set of metadata quantization levels using a bit rate distribution process (706); quantizing and encoding spatial metadata using a metadata quantization layer (707); generating, using a combination of one or more bit rates, a downmix bit stream for one or more downmix channels using one or more bit rates (708); combining the downmix bitstream, quantized and encoded spatial metadata, and a set of quantization levels into an IVAS bitstream (709); and streaming or storing the IVAS bitstream for playback on an IVAS-enabled device (710).

Approximate system architecture

[00132] FIG. 8 shows a block diagram of an example system 800 suitable for implementing example embodiments of the present invention. System 800 includes one or more server computers or any client device, including, but not limited to, any of the devices shown in FIG. 1, such as call server 102, previously created devices 106, subscriber device 108, 114, conference room systems 116, 118, home theater systems, VR headset 122, and immersive content absorption module 124. System 800 includes any consumer devices including, but not limited to: smartphones, tablet computers, wearable computers, in-vehicle computers, game consoles, surround sound systems, kiosks.

[00133] As shown, the system 800 includes a central processing unit (CPU) 801 that is capable of executing various processes in accordance with a program stored, for example, in a read-only memory (ROM) 802, or a program loaded, for example, from storage module 808 to random access memory (RAM) 803. The RAM 803 also stores data required when the CPU 801 executes various processes as needed. The CPU 801, ROM 802, and RAM 803 are connected to each other via bus 804. An I/O interface 805 is also connected to bus 804.

[00134] The following components connect to the I/O interface 805: an input module 806, which may include a keyboard, mouse, etc.; an output module 807, which may include a display such as a liquid crystal display (LCD) and one or more speakers; a storage module 808 including a hard disk or other suitable storage device; and a communication module 809 including a network interface card, such as a network card (eg, wired or wireless).

[00135] In some implementations, input module 806 includes one or more microphones in various positions (depending on the host device) to capture audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other suitable formats).

[00136] In some implementations, output module 807 includes systems with varying numbers of speakers. As illustrated in FIG. 1, output module 807 (depending on the characteristics of the host device) can render audio signals in various formats (eg, mono, stereo, immersive, binaural, and other suitable formats).

The communication module 809 is configured to exchange data with other devices (eg, over a network). Drive 810 also connects to I/O interface 805 as needed. Removable media 811, such as a magnetic disk, optical disk, magneto-optical disk, flash drive, or other suitable removable media, is mounted on the drive 810 such that a computer program read from it is installed on the storage module 808 as needed. Those skilled in the art will understand that although the system 800 is described as including the components described above, in actual applications, some of these components may be added, removed, and/or replaced, and all such modifications or changes fall within the scope of this inventions.

[00137] In accordance with exemplary embodiments of the present invention, the processes described above may be implemented as programs implemented in the form of computer software or on a computer-readable storage medium. For example, embodiments of the present invention include a computer program product including a computer program tangibly embodied on a computer readable medium, the computer program including program code for implementing the methods. In such embodiments, a computer program may be downloaded and mounted from a network via communications module 809 and/or installed from removable media 811, as shown in FIG. 8.

[00138] In general, various exemplary embodiments of the present invention may be implemented in hardware or specialized circuitry (eg, control circuitry), software, logic, or any combination thereof. For example, the modules explained above may be implemented by a control circuit (eg, a CPU in combination with other components of FIG. 8) such that the control circuit may perform the actions described herein. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device (eg, control circuitry). Although various aspects of exemplary embodiments of the present invention are illustrated and described in the form of block diagrams, method flow diagrams, or some other graphical representation, it should be appreciated that the blocks, devices, systems, technologies, or methods described herein may be embodied, by way of non-limiting examples, in hardware, software, firmware, application-specific circuits or logic, general purpose hardware or a controller, or other computing devices, or some combination thereof.

[00139] In addition, the various blocks shown in the block diagrams may be considered as method steps and/or as operations that result from an operation of computer program code, and/or as a plurality of interconnected logical circuit elements designed with the ability to perform an associated function. For example, embodiments of the present invention include a computer program product including a computer program tangibly embodied on a computer readable medium, wherein the computer program includes program codes configured to implement the methods described above.

[00140] In the context of the invention, a computer-readable medium can be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or apparatus. The computer-readable medium may be a computer-readable signaling medium or a computer-readable storage medium. A computer-readable medium may be non-transitory and may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any suitable combination thereof. More specific examples of a computer readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash). memory), fiber optic, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[00141] The computer program code for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer program codes may be transmitted to a general purpose computer processor, special purpose computer, or other programmable data processing device that has control circuitry such that the program codes, when executed by the computer processor or other programmable data processing device, cause the implementation of functions/ operations indicated on the method flowcharts and/or flowcharts. The software code may run entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server, or may be distributed across one or more remote computers and/or servers.

[00142] Although this document contains many specific implementation details, they should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of features that may be relevant to specific embodiments. Certain features that are explained in this detailed description in the context of individual embodiments may also be implemented combined in a single embodiment. Conversely, various features that are described in the context of one embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as operating in certain combinations and even initially defined as such in the claims, one or more features of a claimed combination may in some cases be excluded from the combination, and the claimed combination may refer to a sub-combination or variants of a sub-combination . The logical sequences of operations illustrated in the drawings do not require a particular order shown or sequential order to achieve the desired results. In addition, other steps may be provided or steps may be omitted from the processes described, and other components may be added or removed from the systems described. Accordingly, other implementations are within the scope of the appended claims.

Claims (25)

1. A method for encoding a bitstream with support for immersive voice and audio services (IVAS), the method comprising the steps of: - receive, using one or more processors, an input audio signal; - downmixing, using one or more processors, the input audio signal into one or more downmix channels and spatial metadata associated with the one or more input audio signal channels; - obtaining, using one or more processors, a set of one or more bit rates for the downmix channels and a set of quantization levels for spatial metadata from the bit rate distribution control table; - determining, using one or more processors, a combination of one or more bit rates for the downmix channels; - determining, using one or more processors, a metadata quantization level from a set of metadata quantization levels; - quantize and encode, using one or more processors, spatial metadata using a metadata quantization layer; - generating, using one or more processors and a combination of one or more bit rates, a downmix bit stream for one or more downmix channels; - combine, using one or more processors, the downmix bitstream, quantized and encoded spatial metadata, and a set of quantization levels into an IVAS bitstream. 2. The method of claim 1, wherein the input audio signal is a four-channel first-order ambiophony (FoA) audio signal, a three-channel flat FoA signal, or a two-channel stereo audio signal. 3. The method of claim 1 or 2, wherein the one or more bit rates are bit rates of one or more instances of a mono audio encoder/decoder (codec). 4. The method of claim 1 or 2, wherein the mono audio codec is an enhanced voice services (EVS) codec and the downmix bitstream is an EVS bitstream. 5. The method of claim 1 or 2, wherein obtaining, using one or more processors, one or more bit rates for the downmix channels and spatial metadata using the bit rate distribution control table further comprises the steps of: - identifying a row in the bit rate distribution control table using a table index that includes one or more of an input audio signal format, an input audio signal bandwidth, an enabled spatial encoding tool, a transition mode, and a backwards compatible mode for down-mono mixing; And - extracting from an identified row of the bit rate distribution control table one or more of a target bit rate, a bit rate ratio, a minimum bit rate, and bit rate deviation steps, wherein the bit rate ratio indicates the ratio with which the overall bit rate bits must be distributed among the transmission channels of the downmix audio signals, the minimum bit rate is the value below which the full bit rate is not allowed to be reduced, and the bit rate rejection steps are the steps to reduce the target bit rate when the first priority is for the downmix signals higher than or equal to or lower than the second spatial metadata priority; And wherein the determination of a combination of one or more bit rates for the downmix channels and spatial metadata is based on one or more of a target bit rate, a bit rate ratio, a minimum bit rate, and bit rate deviation steps. 6. The method of claim 1 or 2, wherein quantization and encoding of spatial metadata for one or more channels of the input audio signal using a set of metadata quantization levels is performed in a quantization loop that applies increasingly coarse quantization strategies based on the difference between the target bit rate in metadata bits and the actual transmission rate in metadata bits. 7. The method of claim 1 or 2, wherein the quantization is determined according to mono codec priority and spatial metadata priority based on properties extracted from the input audio signal and covariance values in the channel bandwidth. 8. The method of claim 1 or 2, wherein the input audio signal is a stereo signal and the downmix signals include an average signal representation, residuals from the stereo signal, and spatial metadata. 9. The method of claim 1 or 2, wherein the spatial metadata includes prediction coefficients (PR), cross-prediction coefficients (C) and decorrelation coefficients (P) for a format for spatial reconstruction module (SPAR) and coefficients (P) prediction and decorrelation coefficients (PR) for the advanced complex linkage (CACPL) format. 10. The method of claim 1 or 2, further comprising storing, streaming, or outputting the IVAS bitstream. 11. A system for bitstream encoding supporting immersive voice and audio services (IVAS), comprising: - one or more processors; And - a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, direct one or more processors to perform the operations of any one of claims. 1-10 per method. 12. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, direct one or more processors to perform the operations of any one of claims. 1-10 per method.

Images