JP2021036341A - Layered coding for compressed sound or sound field representations - Google Patents

Abstract

To provide a method of layered encoding of a compressed sound representation of a sound or sound field.SOLUTION: The compressed sound representation comprises a basic compressed sound representation comprising a plurality of components, basic side information for decoding the basic compressed sound representation to a basic reconstructed sound representation of the sound or sound field, and enhancement side information including parameters for improving the basic reconstructed sound representation. The method comprises sub-dividing the plurality of components into a plurality of groups of components and assigning each group of the plurality of groups to an individual among a plurality of respective hierarchical layers. The number of groups corresponds to the number of layers, and the plurality of layers includes a base layer and one or more hierarchical enhancement layers. The method further includes: adding the basic side information to the base layer; determining a plurality of portions of enhancement side information from the enhancement side information; and assigning the plurality of portions of enhancement side information to the plurality of respective layers.SELECTED DRAWING: Figure 2

Description

Mutual reference to related applications This application applies to European Patent Application No. 15306589.1 filed on October 15, 2015, and US Patent Application No. 62 / 361,461 and No. 15306653.5 filed on October 15, 2015. It claims the priority of No. 62 / 361,416. The contents of these applications are incorporated herein by reference in their entirety.

Technical Field This paper relates to methods and devices for layered audio coding. In particular, this article presents methods and devices for layered audio coding of compressed sound (or sound field) representations, such as Higher-Order Ambisonics (HOA) sound (or sound field) representations. Regarding.

For streaming sound (or sound field) representations through transmission channels with time-varying conditions, layered coding adapts the quality of the received sound representation to the transmission conditions and avoids particularly unwanted signal loss. It is a means to do.

For layered coding, the sound (or sound field) representation is typically divided into a relatively small size high priority base layer and an additional enhancement layer with decrementing priority and any size. Subdivided. Each improvement layer is typically assumed to contain incremental information to supplement the information of all lower layers in order to improve the quality of sound (or sound field) representation. The amount of error protection for the transmission of individual layers is controlled based on their priority. In particular, the base layer is provided with high error protection, which is reasonable and acceptable due to its small size.

However, there is a need for a layered coding scheme for (an extended version of) a special type of compressed representation of a sound or sound field, such as a compressed HOA sound or sound field representation. ..

ISO / IEC JTC1 / SC29 / WG11 23008-3: 2015 (E), Information technology --High efficiency coding and media delivery in heterogeneous environments --Part 3: 3D audio, February 2015 ISO / IEC JTC1 / SC29 / WG11 23008-3: 2015 / PDAM3, Information technology --High efficiency coding and media delivery in heterogeneous environments --Part 3: 3D audio, AMENDMENT 3: MPEG-H 3D Audio Phase 2, July 2015

This paper addresses the above issues. In particular, methods and encoders / decoders for layered coding of compressed sound or sound field representations are described.

According to one aspect, a method of encoding a layered structure of a compressed sound representation of a sound or sound field is described. The compressed sound representation may include a basic compressed sound representation that includes a plurality of components. The plurality of components may be complementary components. The compressed sound representation may further include basic side information for decoding the basic compressed sound representation into a basic reconstructed sound representation of the sound or sound field. The compressed sound representation may further include improvement side information including parameters for improving (eg, improving) the basic reconstructed sound representation. The method may include subdividing (eg, grouping) the plurality of components into a plurality of component groups. The method may further include assigning (eg, adding) each group of the plurality of groups to an individual of the plurality of hierarchical layers. Allocations may indicate the correspondence between individual groups and tiers. The components assigned to each layer may be said to be contained in that layer. The number of groups may correspond to the number of layers (eg, they may be equal). The plurality of layers may include a basic layer and one or more hierarchical improvement layers. Even if the plurality of hierarchical layers are ordered from the basic layer, through the first improvement layer, the second improvement layer, etc., to the overall highest improvement layer (overall top layer). Good. The method further includes adding basic side information to the basic layer (eg, including basic side information in the basic layer for transmission or storage purposes, or assigning basic side information to the basic layer). You may be. The method may further include discriminating a plurality of parts of the improved side information from the improved side information. The method may further include assigning (eg, adding) each of the plurality of portions of the improved side information to each layer of the plurality of layers. Each piece of improved side information is reconstructed (eg, decompressed) obtained from the data contained (eg, assigned or added) in that layer and any layers below that layer. It may include parameters to improve the sound expression. Layered encoding may be performed for transmission over a transmission channel or for storage on a suitable storage medium such as a CD, DVD or Blu-ray Disc ™.

Constructed as described above, the proposed method provides layered coding to multiple components as well as basic and improved side information having the above properties (eg, independent basic and improved side information). Allows for efficient application to compressed sound representations, including. In particular, the proposed method includes suitable side information for each layer to reconstruct a reconstructed sonic representation from components contained in any layer up to the layer in question. Here, the layer up to the problem layer is understood to include, for example, a basic layer up to the problem layer, a first improvement layer, a second improvement layer, and the like. Thus, the actual highest available layer (eg, the layer below the lowest layer that has not yet been effectively received; all layers below the highest available layer and the highest available layer itself are valid. Regardless of (received in), the decoder may improve or improve the reconstructed sound representation, even if the reconstructed sound representation differs from the complete (eg, full) sound representation. Is made possible. In particular, improve or improve the reconstructed sonic representation that can be obtained based on all the components contained in the layers up to the actual best available layer, regardless of the actual best available layer. It is sufficient for the decoder to decode the payload of the improved side information for only a single layer (ie, for the highest available layer). That is, for each time interval (eg, frame), only a single payload of improved side information may need to be decoded. On the other hand, the proposed method allows to take full advantage of the reduced bandwidth requirements that can be achieved when applying layered coding.

In embodiments, the components of the basic compressed sound representation may correspond to a monaural signal (eg, a transport signal or a monaural transport signal). The monaural signal may represent either a predominant sound signal or a coefficient sequence of HOA representation. The monaural signal may be quantized.

In embodiments, the basic side information may include information that specifies one or more decodings (eg, decompression) of the plurality of components individually and independently of the other components. For example, the basic side information may represent side information related to individual monaural signals independently of other monaural signals. Therefore, the basic side information is sometimes referred to as independent basic side information.

In various embodiments, the improvement side information may represent improvement side information. The improved side information is for the basic compressed sound representation and the basic compressed sound representation for improving (eg improving) the basic reconstructed sound representation obtained from the basic side information. Predictive parameters may be included.

In embodiments, the method further transmits the data of the plurality of layers (eg, data assigned or added to each layer, or otherwise included in each layer). May include generating a transport stream for. The basic layer may have the highest priority of transmission, and the hierarchical improvement layers may have the decrementing priority of transmission. That is, the transmission priority may be reduced from the basic layer to the first improvement layer, from the first improvement layer to the second improvement layer, and so on. The amount of error protection for the transmission of the plurality of layers of data may be controlled according to the priority of each transmission. This reduces the overall required bandwidth by not applying excessive error protection to the upper layers, while ensuring that at least some lower layers are transmitted reliably.

In various embodiments, the method may further include generating, for each layer of the plurality of layers, a transport layer packet containing data for each layer. For example, for each time interval (eg, frame), each transport layer packet may be generated for each layer of the plurality of layers.

In embodiments, the compressed sound representation may further include additional basic side information for decoding the basic compressed sound representation into a basic reconstructed sound representation. .. The additional basic side information may include information that specifies the decoding of one or more of the plurality of components depending on the other components. The method may further include breaking down the additional basic side information into multiple parts of the additional basic side information. The method further adds those parts of the additional basic side information to the base layer (eg, for transmission or storage, includes those parts of the additional basic side information in the base layer, or is additional. Allocate those parts of the basic side information to the basic layer). Each piece of additional basic side information may correspond to each layer, and the decoding of one or more components assigned to each layer may be lower than that of each layer and any of the layers. It may contain information that is specified depending on each other component (only) assigned to the layers of. That is, each part of the additional basic side information is any other component assigned to the layer above each layer with the components in each layer to which that part of the additional basic side information corresponds. Specify without reference.

Constructed as such, the proposed method avoids additional fragmentation of basic side information by adding all parts to the base layer. In other words, all parts of the additional basic side information is included in the basic layer. The decomposition of the additional basic side information ensures that for each layer, a portion of the additional basic side information that does not require knowledge of the components of the higher layers is available. Thus, it is sufficient for the decoder to decode the additional basic side information contained in the layers up to the highest available layer, regardless of the actual best available layer.

In embodiments, the additional basic side information may include information that specifies one or more decodings (eg, decompression) of the plurality of components depending on the other components. For example, additional basic side information may represent side information related to individual monaural signals, depending on other monaural signals. Therefore, the additional basic side information is sometimes referred to as the dependent basic side information.

In embodiments, the compressed sound representation may be processed for a series of time intervals, eg, time intervals of equal size. The series of time intervals may be frames. Thus, the method can operate on a frame basis. That is, the compressed sound representation can be encoded frame by frame. The compressed sound representation may be available for each successive time interval (eg, for each time frame). That is, the compression operation in which the compressed sound expression is obtained may be performed on a frame basis.

In embodiments, the method may further include, for each layer, generating configuration setting information indicating the components of the basic compressed sound representation assigned to that layer. In this way, the decoder has easy access to the information needed for decoding without unnecessary parsing through the received data payload.

Another aspect describes how to encode the layered structure of a compressed sound representation of a sound or sound field. The compressed sound representation may include a basic compressed sound representation that includes a plurality of components. The plurality of components may be complementary components. The compressed sound representation further provides basic side information (eg, independent basic side information) for decoding the basic compressed sound representation into a basic reconstructed sound representation of the sound or sound field. And may include a third piece of information (eg, subordinate basic side information). The basic side information may include information that individually specifies the decoding of one or more of the plurality of components and is independent of the other components. The additional basic side information may include information that specifies the decoding of one or more of the plurality of components depending on each other component. The method may include subdividing (eg, grouping) the plurality of components into a plurality of component groups. The method may further include assigning (eg, adding) each group of the plurality of groups to an individual of the plurality of hierarchical layers. Allocations may indicate the correspondence between individual groups and tiers. The components assigned to each layer may be said to be contained in that layer. The number of groups may correspond to the number of layers (eg, they may be equal). The plurality of layers may include a basic layer and one or more hierarchical improvement layers. The method further includes adding basic side information to the basic layer (eg, including basic side information in the basic layer for transmission or storage purposes, or assigning basic side information to the basic layer). You may be. The method further decomposes the additional basic side information into multiple parts of the additional basic side information and adds those parts of the additional basic side information to the basic layer (eg, for transmission or storage). , Include those parts of additional basic side information in the base layer, or allocate those parts of additional basic side information to the base layer). Each piece of additional basic side information may correspond to each layer, and the decoding of one or more components assigned to each layer may be lower than that of each layer and any of the layers. It may contain the specified information depending on each other component assigned to the layers of.

Constructed as such, the proposed method contains for each layer the components contained in any layer up to that layer without the need for effective receipt or decoding (or generally knowledge) of the higher layers. Ensure that appropriate additional basic side information is available to decode. For compressed HOA representations, the proposed method ensures that in vector coding mode a suitable V vector is available for all components belonging to the layers up to the highest usable layer. To do. In particular, the proposed method eliminates cases where the elements of the V vector corresponding to the components in the higher layers are not explicitly signaled. Thus, the information contained in the layers up to the highest usable layer is sufficient to decode (eg, decompress) any component belonging to the layers up to the highest usable layer. This ensures proper decompression of each reconstructed HOA representation for the lower layers, even if the upper layers have not been effectively received by the decoder. On the other hand, the proposed method allows to take full advantage of the reduced bandwidth requirements that can be achieved when applying layered coding.

The embodiment of this aspect may be related to the embodiment of the above aspect.

Another aspect describes how to decode a compressed sound representation of a sound or sound field. The compressed sound representation may be encoded in a plurality of hierarchical layers. The plurality of hierarchical layers may include a basic layer and one or more hierarchical improvement layers. The plurality of layers may be assigned components of a basic compressed sound expression of sound or sound field. In other words, the plurality of layers may contain basic compressed side information components. Those components may be assigned to each layer in each component group. The plurality of components may be complementary components. The base layer may contain basic side information for decoding the basic compressed sound representation. Each layer contains some of the improved side information, including parameters for improving the basic reconstructed sonic representation obtained from the data contained in each layer and any layers below the respective layers. You may be. The method may include receiving a data payload corresponding to each of the plurality of hierarchical layers. The method is further among the plurality of layers, the highest to be used to decode the basic compressed sound representation into the basic reconstructed sound representation of the sound or sound field. It may include determining the first layer index indicating the available layers of. The method further uses the basic side information to represent the basic reconstructed sound from components assigned to the highest usable layer and any layers lower than the highest usable layer. May include obtaining. The method further determines which portion of the improved side information should be used to improve (eg, improve) the basic reconstructed sonic representation as a second layer index. It may be included. The method may further include obtaining the reconstructed sound representation of the sound or sound field from the basic reconstructed sound representation with reference to the second layer index.

As such, the proposed method makes the best use of available (eg, validly received) information to ensure that the reconstructed sonic representation has optimum quality.

In embodiments, the components of the basic compressed sound representation may correspond to a monaural signal (eg, a monaural transport signal). The monaural signal may represent either a predominant sound signal or a coefficient sequence of HOA representation. The monaural signal may be quantized.

In various embodiments, the basic side information may include information that individually designates decoding (for example, decompression) of one or more of the plurality of components independently of the other components. For example, the basic side information may represent side information related to individual monaural signals independently of other monaural signals. Therefore, the basic side information is sometimes referred to as independent basic side information.

In various embodiments, the improvement side information may represent improvement side information. The improved side information is for the basic compressed sound representation and the basic compressed sound representation for improving (eg improving) the basic reconstructed sound representation obtained from the basic side information. Predictive parameters may be included.

In embodiments, the method may further include, for each layer, determining whether or not each layer has been effectively received. The method may further include determining the first layer index as the layer index of the layer immediately below the lowest layer that has not been effectively received.

In embodiments, determining the second layer index is when determining that the second layer index is equal to the first layer index, or when obtaining the reconstructed sonic representation. An index value indicating that no improvement side information is not used may be involved in determining as the second layer index. In the latter case, the reconstructed sound representation may be equal to the basic reconstructed sound representation.

In embodiments, the data payload may be received and processed for a series of time intervals, eg, time intervals of equal size. The series of time intervals may be frames. Thus, the method can operate on a frame basis. The method further determines that the second layer index is equal to the first layer index if the compressed sound representations for those series of time intervals can be decoded independently of each other. You may.

In embodiments, the data payload may be received and processed for a series of time intervals, eg, time intervals of equal size. The series of time intervals may be frames. Thus, the method can operate on a frame basis. The method further applies to each layer for a given time interval within the set of time intervals, if the compressed sound representations for the set of time intervals cannot be decoded independently of each other. It may include determining whether each layer has been effectively received. The method further includes the first layer index for the given time interval, the first layer index for the time interval preceding the given time interval, and the lowest layer that was not effectively received. It may include determining as the smaller of the layer indexes of the layer immediately below.

In embodiments, the method relates to the given time interval if the compressed sonic representations for those sequence of time intervals cannot be decoded independently of each other. May include determining if the first layer index of is equal to the first layer index for the preceding time interval. The method further displays the second layer index for the given time interval if the first layer index for the given time interval is equal to the first layer index for the preceding time interval. It may include determining to be equal to the first layer index for the given time interval. The method further provides any improved side information when obtaining the reconstructed sonic representation if the first layer index for the given time interval is not equal to the first layer index for the preceding time interval. It may include determining an index value indicating that the second layer index is not used as the second layer index.

In embodiments, the base layer corresponds to each layer, and the decoding of one or more of the components assigned to each layer is optionally lower than that of each layer and of each layer. It may contain at least one portion of additional basic side information, including information specified depending on other components assigned to the layers of. The method further refers to the components of the additional basic side information for each portion of the additional basic side information by referring to the components assigned to each layer and any layers below the respective layers. It may include decoding the part. The method further refers to said portion of additional basic side information for the highest available layers and the components assigned to any layer between the highest available layers and the respective layers. By doing so, it may include correction. The basic reconstructed sound representation is from the components assigned to the highest available layers and any layers below the highest available layers, to the basic side information and the highest available layers. It may be obtained by using the corrected parts of the additional basic side information obtained from the parts of the additional basic side information corresponding to the layers of.

In embodiments, the additional basic side information may include information that specifies the decoding (eg, decompression) of one or more of the plurality of components depending on the other component. .. For example, additional basic side information may represent side information related to individual monaural signals depending on other monaural signals. Thus, the additional basic side information is sometimes referred to as the dependent basic side information.

Another aspect describes how to decode a compressed sound representation of a sound or sound field. The compressed sound representation may be encoded in a plurality of hierarchical layers. The plurality of hierarchical layers may include a basic layer and one or more hierarchical improvement layers. The plurality of layers may be assigned components of a basic compressed sound expression of sound or sound field. In other words, the plurality of layers may contain basic compressed side information components. Those components may be assigned to each layer in each component group. The plurality of components may be complementary components. The base layer may contain basic side information for decoding the basic compressed sound representation. The base layer further corresponds to each layer and decodes one or more of the components assigned to each layer into each layer and any layers lower than each layer. It may contain at least one portion of additional basic side information, including information specified depending on other assigned components. The method may further include receiving the data payload corresponding to each of the plurality of hierarchical layers. The method is further among the plurality of layers, the highest to be used to decode the basic compressed sound representation into the basic reconstructed sound representation of the sound or sound field. It may include determining the first layer index indicating the available layers of. The method further refers to the components of the additional basic side information for each portion of the additional basic side information by referring to the components assigned to each layer and any layers below the respective layers. It may include decoding the part. The method further provides, for each portion of the additional basic side information, the portion of the additional basic side information between the highest available layer and the highest available layer and the respective layers. It may include correction by referring to the components assigned to any of the layers. The basic reconstructed sound representation is from the components assigned to the highest available layers and any layers below the highest available layers, to the basic side information and the highest available layers. It may be obtained by using the corrected parts of the additional basic side information obtained from the parts of the additional basic side information corresponding to the layers of. The method may further include determining a second layer index that is equal to or indicates omission of improved side information during decoding.

Constructed as such, the proposed method ensures that the additional basic side information ultimately used to decode the basic compressed sound representation does not contain redundant elements. Makes the actual decoding of the basic compressed sound representation more efficient.

The embodiments of this aspect may relate to the embodiments of the above aspect.

According to another aspect, an encoder for encoding the layered structure of a compressed sound representation of a sound or sound field is described. The compressed sound representation may include a basic compressed sound representation that includes a plurality of components. The plurality of components may be complementary components. The compressed sound representation may further include basic side information for decoding the basic compressed sound representation into a basic reconstructed sound representation of the sound or sound field. .. The compressed sound representation may further include improvement side information including parameters for improving (eg, improving) the basic reconstructed sound representation. The encoder may include a processor configured to perform some or all of the method steps of the method based on the above aspects mentioned first and the above aspects mentioned second.

According to another aspect, a decoder is described for decoding a compressed sound representation of sound or sound field. The compressed sound representation may be encoded in a plurality of hierarchical layers. The plurality of hierarchical layers may include a basic layer and one or more hierarchical improvement layers. The plurality of layers may be assigned components of a basic compressed sound expression of sound or sound field. In other words, the plurality of layers may contain basic compressed side information components. Those components may be assigned to each layer in each component group. The plurality of components may be complementary components. The base layer may contain basic side information for decoding the basic compressed sound representation. Each layer contains improved side information, including parameters for improving (eg, improving) the basic reconstructed sound representation obtained from the data contained in the respective layer and any layers lower than the respective layer. May include a part of. The decoder may include a processor configured to perform some or all of the method steps of the method according to the third mentioned aspect and the fourth mentioned aspect of the method.

According to other aspects, methods, devices and systems are directed towards decoding compressed higher ambisonics (HOA) sound representations of sound or sound fields. The device has a receiver configured to receive a bitstream containing the compressed HOA representation corresponding to multiple hierarchical layers, including a basic layer and one or more hierarchical enhancement layers. Alternatively, the method may carry out the receipt. The plurality of layers are assigned components of a basic compressed sound expression of sound or sound field, and these components are assigned to each layer in each component group. The device is to decode the compressed HOA representation based on the basic side information associated with the base layer and based on the improvement side information associated with the one or more hierarchical improvement layers. It may have a configured decoder, or the method may perform the decoding. The basic side information may include basic independent side information related to the first individual monaural signals that are decoded independently of the other monaural signals. Each of the one or more hierarchical improvement layers is for improving the basic reconstructed sound representation obtained from the data contained in each layer and any layers lower than the respective layers. It may include a part of the improvement side information including parameters.

The basic independent side information may indicate that the first individual monaural signal represents a directional signal with an incident direction. The basic side information may further include basic dependent side information related to a second individual monaural signal that is decoded depending on the other monaural signal. The basic dependent side information may include vector-based signals that are directionally distributed within the sound field. Here, the directional distribution is specified by a vector. The vector component is set to 0 and is not part of the compressed vector representation.

The components of the basic compressed sound representation can correspond to a monaural signal representing either the dominant sound signal or the coefficient sequence of the HOA representation. The bitstream contains a data payload corresponding to each of the plurality of hierarchical layers. Improved side information: may include parameters related to at least one of spatial prediction, subband directional signal synthesis and parametric ambient sound replication. The improved side information may include information that allows the prediction of the sound or the missing part of the sound field from the directional signal. Further, for each layer, it may be determined whether or not each layer has been effectively received, and the layer index of the layer immediately below the lowest layer that has not been effectively received may be determined.

According to another aspect, software programs are written. The software program is adapted for execution on a processor and may be adapted to perform some or all of the method steps outlined in this article when executed on a computing device.

Yet another aspect describes the storage medium. This storage medium is a software program that is adapted for execution on a processor and for performing some or all of the method steps outlined in this article when executed on a computing device. It may be included.

As those skilled in the art will understand, statements made with respect to any of the above aspects or embodiments thereof also apply to other aspects or embodiments thereof. Repeating these statements for each aspect or embodiment is omitted for brevity.

Methods and devices that include the preferred embodiments outlined in this article may be used alone or in combination with other methods and systems disclosed herein. In addition, all aspects of the methods and devices outlined in this paper may be arbitrarily combined. In particular, the features of the claims may be combined with other features in any way.

Method steps and device features can be exchanged in many ways. In particular, as those skilled in the art will understand, the details of the disclosed method can be implemented as a device adapted to perform some or all of the steps of the method and vice versa.

The present invention will be described below in an exemplary manner with reference to the accompanying drawings.
It is a flowchart which shows the example of the layer structure encoding method based on the embodiment of this disclosure. It is a block diagram which shows typically the example of the encoder stage based on the embodiment of this disclosure. It is a flowchart which shows an example of the method of decoding the compressed sound expression of a sound or a sound field encoded into a plurality of hierarchical layers based on the embodiment of the present disclosure. It is a block diagram which shows typically the example of the decoder stage based on the embodiment of this disclosure. It is a block diagram which shows typically the example of the decoder stage based on the embodiment of this disclosure. It is a block diagram which shows schematic example of the hardware implementation of the encoder based on the embodiment of this disclosure. It is a block diagram which shows typically the example of the hardware implementation of the decoder based on the embodiment of this disclosure.

First, a method based on the present disclosure and a compressed sound (or sound field) representation to which an encoder / decoder can be applied (hereinafter referred to as a compressed sound representation for brevity) will be described. In general, a fully compressed sound (or sound field) representation (hereinafter referred to as a fully compressed sound representation for brevity) contains three components (eg, three components: ) Possible: Basic compressed sound (or sound field) representation (hereinafter referred to as basic compressed sound representation for brevity), basic side information and improved side information.

The basic compressed sound representation itself contains (eg, consists of) several components (eg, complementary components). The basic compressed sound representation can make up a prominent proportion of the fully compressed sound representation. The basic compressed sound representation may consist of a dominant sound signal or a monaural transport signal representing the coefficient sequence of the original HOA representation.

The basic side information is needed to decode the basic compressed sound representation and can be assumed to be much smaller in size than the basic compressed sound representation. This may further consist, in large part, of separate parts, each specifying the decompression of only one particular component of the basic compressed sound representation. The basic side information may consist of a first part, which may be known as independent basic side information, and a second part, which may be known as additional basic side information.

Both the first and second parts, namely independent basic side information and additional basic side information, can specify the decompression of certain components of the basic compressed sound representation. The second part is optional and may be omitted. In this case, the compressed sound representation is sometimes said to include a first part (eg, basic side information).

The first part (eg, basic side information) contains side information that describes the individual (complementary) components of the basic compressed sound representation independently of the other (complementary) components. You may be. In particular, the first part (eg, basic side information) may specify the decoding of one or more of the plurality of components individually and independently of the other components. Thus, the first part may be referred to as independent basic side information.

The second (arbitrary) part, also known as additional basic side information, transforms the individual (complementary) components of the basic compressed sound representation into other (complementary) components. It can be described as dependent. This second part may be referred to as subordinate basic side information. In particular, the dependency may have the following attributes:
Dependent basic side information about each individual (complementary) component of the basic compressed sound representation includes some other (complementary) component to the basic compressed sound representation. If not, achieve the maximum range.
• When some additional (complementary) components are added to the basic compressed sound representation, the dependent basic side information about the individual (complementary) components being considered is the source. It becomes a subset of the dependent basic side information of, which can reduce its size.

Improvement side information is also optional. It can be used to improve or improve (eg, parametrically improve or improve) the basic compressed sound representation. Its size is also assumed to be much smaller than the size of a basic compressed sound representation.

As described above, in various embodiments, the compressed sound expression is described by decoding (for example, decompressing) a basic compressed sound expression including a plurality of components and the basic compressed sound expression. Basic side information for a basic reconstructed sound representation of a sound or sound field, and parameters for improving or improving (eg, parametrically improving or improving) a basic compressed sound representation. May include improved side information including. The compressed sound representation may further contain additional basic side information for decoding (eg, decompressing) the basic compressed sound representation into the basic reconstructed sound representation. Often, this may include information that specifies the decoding of one or more of the plurality of components depending on each other component.

One example of such a type of fully compressed sound representation is the compressed compression specified by Chapter 12 and Annex C.5 of the Preliminary Version of the MPEG-H 3D Audio Standard (Non-Patent Document 1). It is given by the higher ambisonics (HOA) sound field representation. That is, the compressed sound representation can correspond to the compressed HOA sound (or sound field) representation of the sound or sound field.

For this example, the basic compressed sound field representation (basic compressed sound representation) may contain several components (eg, may be identified by some components). Those components may be monaural signals (for example, they may correspond to monaural signals). Those monaural signals may be quantized monaural signals. Those monaural signals can represent either a dominant sound signal or a coefficient sequence of ambient HOA sound field components.

The basic side information can describe, among other things, for each of these monaural signals how it contributes spatially to the sound field. For example, the basic side information may specify the dominant sound signal as a purely directional signal, i.e. a general plane wave with a certain incident direction. Alternatively, the basic side information may specify the monaural signal as a coefficient sequence of the original HOA representation with an index. The basic side information may be further separated into a first part and a second part as described above.

The first part is the side information (eg, independent basic side information) associated with a particular individual monaural signal. This independent basic side information is independent of the existence of other monaural signals. Such side information may specify, for example, a monaural signal representing a directional signal having a certain incident direction (eg, meaning a general plane wave). Alternatively, the monaural signal may be specified as a coefficient sequence of the original HOA representation with an index. The first part may be referred to as independent basic side information. In general, the first part (eg, basic side information) may specify the decoding of one or more monaural signals of the plurality of monaural signals individually and independently of the other monaural signals.

The second part is the side information (eg, additional basic side information) associated with a particular individual monaural signal. This side information depends on the presence of other monaural signals. Such side signals may be used, for example, when the monaural signal is designated as a vector-based signal (see, eg, Section 12.4.2.4.4 of Non-Patent Document 1). These signals are directionally distributed in the sound field, and the directional distribution can be specified by a vector. In certain modes (ie, CodedVVecLength = 1), certain components of this vector are implicitly set to 0 and are not part of the compressed vector representation. These components are components with an index equal to that of the coefficient sequence of the original HOA representation that is part of the basic compressed sound representation. That is, if the individual components of the vector are coded, the total number depends on the basic compressed sound representation. In particular, the total number depends on which coefficient sequence the original HOA representation contains.

If the coefficient sequence of the original HOA representation is not included in the basic compressed sound representation, the dependent basic side information for each vector-based signal consists of all vector components, with its maximum size. Have. If an indexed coefficient sequence of the original HOA representation is added to the basic compressed sound representation, the vector components with those indexes will be the side information for each vector-based signal. Removed from, thereby reducing the size of dependent basic side information for vector-based signals.

Improved side information (eg, improved side information) includes parameters and / or subband directional signal synthesis and parametric ambient sound replication related to (broadband) spatial prediction (see Section 12.4.2.4.3 of Non-Patent Document 1). May include parameters related to.

Parameters related to (broadband) spatial prediction can be used to (linearly) predict the missing part of the sound field from a directional signal.

Subband directional signal synthesis and parametric ambient sound replication are compression tools recently introduced by revision to the MPEG-H 3D audio standard (see Section 1 of Non-Patent Document 2). These tools allow frequency-dependent parametric prediction of additional monaural signals that should be spatially distributed to complement a compressed HOA representation that is spatially incomplete or missing. The prediction may be based on the coefficient sequence of the basic compressed sound representation.

Note that the above complementary contributions to the sound field are represented within the compressed HOA representation by additional side information of a much smaller size in comparison, rather than by additional quantized signals. It is important to keep in mind. Therefore, the two coding tools mentioned above are particularly suitable for compressing HOA representations at low data rates.

A second example of a compressed representation of one or more monaural signals with the structure described above is encoded for separate frequency bands up to a certain upper frequency limit, which can be considered a basic compressed representation. Spectral information; basic side information that specifies the encoded spectral information (eg, by the number and width of encoded frequency bands); and an improved side that includes (eg, consists of) spectral band replication (SBR) parameters. It may contain information. The parameters of the improved side information parametrically reconstruct the spectral information for higher frequency bands, which is not considered in the basic compressed representation, from the basic compressed representation. Describe.

The present disclosure proposes a method for coding the layered structure of a fully compressed sound (or sound field) representation with the structure described above.

Compression may be frame-based in the sense that it gives a compressed representation (eg, in the form of a data packet, or equivalent but a frame payload) of a set of time intervals. Time intervals can have equal or different sizes. These data packets may be assumed to contain the validity flag, a value indicating its size, in addition to the actual compressed representation of the data. In the following, compression is assumed to be frame-based, without the intent of limitation. In addition, unless otherwise noted, focus on the treatment of a single frame without limited intent. Therefore, the frame index is omitted.

Each frame payload of the fully compressed sound (or sound field) representation considered contains J data packets (or frame payloads), and each data packet contains BSRC _j , j =. It is assumed that it is about one component of the basic compressed sound expression marked 1,…, J. In addition, each data packet is expected to contain a packet with independent basic side information as described by _{BSI I.} BSI _{I specifies the specific components BSRC j} of the basic compressed sound representation, independent of the other components. Optionally, each data packet is further assumed to contain a packet with dependent basic side information (additional basic side information) marked _{BSI D.} BSI _{D specifies the specific components BSRC j} of the basic compressed sound representation, which depends on other components.

The information contained within the two data packets BSI _I and BSI _D may optionally be grouped into a single data packet BSI of basic side information. A single data packet BSI may be said to contain _{, among other things, J parts, each specifying one particular component BSRC j of the basic compressed sound representation.} Each of these parts may be said to include a part of independent side information and optionally a part of dependent side information.

Ultimately, each data packet is labeled ESI, with a description of how to improve or improve the reconstructed sound (or sound field) from a complete basic compressed sound representation. It may include enhancement side information to be provided.

The proposed solution for layered coding is the required to enable both a compressor, including packing of data packets for transmission, and a receiver and decompressor. Work on the stages. Each part will be described in detail below.

First, compression and packing (for example, for transmission) will be described. In particular, the components and elements of a fully compressed sound (or sound field) representation in the case of layered coding are described.

FIG. 1 schematically illustrates a flowchart of an example of a method for compression and packing (eg, an encoding method or a layered encoding method for a compressed sound representation of a sound or sound field). Allocations (eg, allocations) of individual payloads to the base layer and (M-1) enhancement layers may be accomplished by a transport layer packing device. FIG. 2 schematically shows a block diagram of an example of individual payload allocation / allocation.

As shown above, the fully compressed sound representation 2100 may relate, for example, to a compressed HOA representation that includes a basic compressed sound representation. The fully compressed sound representation 2100 includes a plurality of components (for example, monaural signals) 2110-1, ... 2110-J, independent basic side information (basic side information) 2120, and optional improvement side information (improvement side). Information) 2140 and optional subordinate basic side information (additional basic side information) 2130 may be included. The basic side information 2120 may be information for decoding a basic compressed sound representation into a basic reconstructed sound representation of the sound or sound field. The basic side information 2120 may include information that specifies the decoding of one or more components (eg, monaural signals) individually and independently of the other components. The improvement side information 2140 may include parameters for improving (eg, improving) the basic reconstructed sound representation. The additional basic side information 2130 may be (further) information for decoding the basic compressed sound representation into the basic reconstructed sound representation of the plurality of components. It may contain information that specifies the decoding of one or more of the components individually, depending on each other component.

FIG. 2 shows a presupposed assumption that there is a plurality of hierarchical layers, including one basic layer (basic layer) and one or more (hierarchical) improvement layers. For example, there may be a total of M layers, that is, one basic layer and M-1 improvement layers. The plurality of hierarchical layers have a layer index that gradually increases. The lowest value of the layer index (eg layer index 1) corresponds to the base layer. Furthermore, it is understood that the layers are ordered from the basic layer through the various improvement layers to the overall highest improvement layer (ie, the overall top layer).

The proposed method may be performed frame-based (ie, in a frame-by-frame manner). In particular, the compressed sound representation 2100 may be compressed for a series of time intervals, eg, time intervals of equal size. Each time interval can correspond to a frame. The following steps may be performed for each (eg, frame) of a series of time intervals.

In S1010 of FIG. 1, the plurality of components 2110 are subdivided into a plurality of component groups. Each of the plurality of groups is then assigned (eg, added or allocated) to the corresponding one of the plurality of hierarchical layers. Here, the number of groups corresponds to the number of layers. For example, the number of groups may be equal to the number of layers, so that for each layer there may be one group of components. As shown above, the plurality of layers may include a basic layer and one or more (eg, M-1) hierarchical improvement layers.

In other words, the basic compressed sound representation is subdivided into parts assigned to the individual layers. Without loss of generality, grouping _{can be described by the number J m} , m = 0, ..., M of M + 1. Here, J ₀ = 1 and J _M = J + 1, and the component BSRC _j is assigned to the mth layer for _{J m-1} ≤ j <J _m.

In S1020, groups of components are assigned to each layer. In S1030, the basic side information 2120 is added (eg, allocated) to the basic layer (ie, the lowest of the plurality of hierarchical layers).

That is, due to its small size, it is proposed to include complete basic side information (basic side information and optional additional basic side information) in the base layer to avoid its unnecessary fragmentation.

If the compressed sound representation being considered contains subordinate basic side information (additional basic side information), the method further (not shown in FIG. 1) further adds the additional basic side information. It may include decomposing into a plurality of parts 21301, ..., 2130-M of basic side information. Those parts of the additional basic side information may then be added (eg, allocated) to the basic layer. In other words, those parts of the additional basic side information may be included in the basic layer. Each piece of additional basic side information may correspond to each layer, and the decoding of one or more components assigned to each layer is lower than that of each layer and each layer. It may contain information that is specified depending on other components assigned to any layer.

Thus, the independent basic side information BSI _I (basic side information) 2120 remains immutable for allocation, while the dependent basic side information allows correct decoding on the receiver side for layered encoding. And it needs to be treated specially to reduce the size of the transmitted dependent basic side information. It is proposed to decompose the subordinate basic side information into _{M parts (parts) marked by BSI D, m} , m = 1, ..., M. Here, the m-th part contains the subordinate basic side information for each of the basic _{compressed sound expression components BSRC j} and J _m-1 ≤ j <J _{m assigned to the m-th layer.} This assumes that the optional subordinate basic side information exists for the compressed sound representation we are thinking of. In the absence of each dependent side information, BSI _{D, m} is assumed to be empty for that compressed sound representation of the parts. Each part of the dependent basic side information BSI _{D, m} is contained in all layers up to the mth layer (that is, contained in all layers j = 1,…, m) All components BSRC _j , 1 ≤ j <May depend on _{J m.}

If the independent basic side information packet BSI _I is negligibly small in size, it makes sense to keep it as a whole and add (assign) it to the base layer. Arbitrarily, the independent basic side information can be decomposed in the same manner as the dependent basic side information, and packets BSI _{I, m} , m = 1, ..., M are given. This is useful for reducing the size of the base layer by adding (assigning) parts of the independent basic side information to the layer with the corresponding components of the basic compressed sound representation.

In S1040, a plurality of portions 2140-1, ..., 2140-M of the improvement side information may be determined. Each part of the improvement side information contains parameters for improving (eg, improving) the reconstructed sound representation obtained from the data contained in each layer and any layers below the respective layers. May be good.

The reason for performing this step is that in the case of layered coding, it is intended to improve the preliminary decompressed sound (or sound field), so the improvement side information is extra calculated for each layer. It is important to recognize that it needs to be, but it depends on the available layers for decompression. Specifically, the preliminary decompressed sound (or sound field) for a given best decodable layer (best available layer) is the best decompressable layer and the best available layer. It depends on the components contained in any layer below the decodable layer. Therefore, compression needs to provide M individual improved side information data packets (parts of improved side information) _{represented by ESI m, m = 1, ..., M.} Here, the improved side information ESI _m in the m-th data packet is all the data contained in the base layer and the improved layer having an index lower than m (for example, the m-th layer and the m-th layer and below). It is calculated to improve the sound (or sound field) representation obtained from all the data contained in any layer.

In S1050, the plurality of portions 2140-1, ..., 2140-M of the improved side information are assigned (eg, added or allocated) to the plurality of layers. Each portion of the plurality of portions of the improved side information is assigned to each layer of the plurality of layers. For example, each layer of the plurality of layers includes a portion of the improvement side information.

The assignment of basic and / or improved side information to each layer may be indicated in the configuration information generated by the encoding method. In other words, the correspondence between the basic and / or improved side information and each layer may be indicated in the configuration setting information. Further, the configuration setting information may indicate, for each layer, the components of the basic compressed sound expression that are assigned (for example, included) to that layer. The additional parts of the basic side information are contained in the basic layer, but may correspond to a layer different from the basic layer.

In summary, the compression stage provides a frame data packet labeled FRAME with the following composition:

さらに、パケットBSI_Iおよびm＝1,…,MについてのBSI_D,mは単一のパケットBSIに組み合わされてもよい。この場合、FRAMEと記されるフレーム・データ・パケットは次の組成をもつことになる：

In addition, _{the BSI D, m for packets BSI I} and m = 1, ..., M may be combined into a single packet BSI. In this case, the frame data packet labeled FRAME would have the following composition:

フレーム・データ・パケットでの個々のペイロードの順序付けは一般に任意でありうる。

The ordering of individual payloads in frame data packets can generally be arbitrary.

Individual data packets may then be grouped within the payload. The payload is defined as a special data packet that contains the actual compressed representation data as well as a validity flag and a value indicating its size. The use of the payload allows for simple demultiplexing on the receiver side, with the advantage that the old payload can be discarded without having to parse through it. One possible grouping is given by:
-Assign each BSRC _j packet, j = 1, ..., J to a separate payload ( _{marked by BP j} with ￣) (for example, allocate).
-Mth improved side information data packet ESI _m and mth dependent side information data packet BSI _{D, m} are combined into one improved payload (EP _m with ￣, m = 1, ..., M) Assign to (for example, allocate).
• Assign the independent basic side information BSI _I to a separate side information payload (marked by BSIP with ￣).

Optionally, if the size of the independent basic side information is large, each m-th BSI _{I, m} , m = 1, ..., M of the components is in the improved payload (marked by _{EP m with ￣).} It may be assigned (eg, assigned). In this case, the side information payload (BSIP with ￣) is empty and can be ignored.

Another option is to allocate all subordinate basic side information data packets BSI _{D, m} to the side information payload (BSIP with ￣). This makes sense if the size of the dependent basic side information is small.

Finally, a frame data packet written in FRAME with the following composition may be given.

フレーム・データ・パケットでの個々のペイロードの順序付けは一般に任意でありうる。

The ordering of individual payloads in frame data packets can generally be arbitrary.

The method further (not shown in FIG. 1) further, for each of the plurality of layers, the data of each layer (eg, for the basic layer, components, basic side information and improved side information, or the one or more For the improvement layer, generate a transport layer packet (for example, basic layer packet 2200 and M-1 improvement layer packets 2300-1, ..., 2300- (M-1)) including components and improvement side information). May include.

Transport layer packets for different layers may have different transmission priorities. Thus, the method may further (not shown in FIG. 1) include generating a transport stream for the transmission of the plurality of layers of data. Here, the basic layer has the highest transmission priority, and the hierarchical improvement layer has the transmission priority of decrementing. Here, the higher the transmission priority, the greater the degree of error protection, and vice versa.

It is understood that the steps described above may be performed in any order, and the exemplary order shown in FIG. 1 is not limited, unless one step requires another step as a prerequisite.

FIG. 3 shows a method of decoding a compressed sound representation of a sound or sound field for decoding or decompression (unpacking). Examples of corresponding receivers and decompression stages are schematically shown in the block diagrams of FIGS. 4A and 4B.

As can be seen from the above, the compressed sound representation may be encoded in the plurality of hierarchical layers. The plurality of layers may be assigned components of basic compressed sound expression (for example, the components may be included). Those components are assigned to each layer in each component group. The base layer may contain basic side information for decoding the basic compressed sound representation. Each layer is a portion of the above-mentioned portion of the improvement side information that includes parameters for improving the basic reconstructed sound representation obtained from the data contained in the respective layer and any layers below the respective layer. One may be included.

The proposed method may be performed frame-based (ie, in a frame-by-frame manner). In particular, the restored representation of the sound or sound field may be generated for a series of time intervals, eg, time intervals of equal size. Those time intervals may be, for example, frames. The following steps may be performed for each of a series of time intervals (eg, frames).

In S3010, the data payload corresponding to the plurality of layers (for example, a transport layer packet) is received. The data payload may be received as part of a bitstream containing a compressed HOA representation of the sound or sound field that corresponds to the multiple hierarchical layers. Hierarchical layers include basic layers and one or more improvement layers. The plurality of layers may be assigned components of the basic compressed sound expression of the sound or sound field. The components are assigned to each layer in each component group.

The individual layer packets may be multiplexed to provide a received frame packet with a fully compressed sonic representation. The received frame packet is

によって示されてもよい。パケットBSI_Iとm＝1,…,MについてのBSI_D,mとが単一のパケットBSIに組み合わされる代替的な事例では、個別の層パケットは多重化されて、

May be indicated by. In an alternative case where packet BSI _{I and BSI D, m} for m = 1,…, M are combined into a single packet BSI, the individual layer packets are multiplexed and

によって示される完全な圧縮された音表現の受領されたフレーム・パケットを提供してもよい。

It may provide a received frame packet of the fully compressed sound representation indicated by.

With the payload, the received frame packet is

によって与えられてもよい。

May be given by.

The received frame packet may then be passed to the decompressor or decoder 4100. If the transmission of the individual layers is correct, then at least the improved side information payload (corresponding to the portion of the improved side information) is included.

部分の有効性フラグは「真」に設定される。個別の層の伝送に起因する誤りがある場合には、少なくともこの層における向上サイド情報ペイロード内の有効性フラグは「偽」に設定される。よって、層パケットの有効性は、含まれている向上サイド情報ペイロードの有効性から（たとえば、その有効性フラグから）判別できる。

The validity flag of the part is set to "true". If there are errors due to the transmission of individual layers, the validity flag in the improved side information payload at least in this layer is set to "false". Thus, the validity of a tier packet can be determined from the validity of the included improved side information payload (eg, from its validity flag).

In the decompression device 4100, the received frame packet may be decompressed. For this purpose, information about the size of each payload may be utilized to avoid unnecessary parsing through the data of the individual payloads.

In S3020, among the plurality of layers, the first layer index indicating the highest layer (for example, the highest usable layer or the highest decodable layer) decodes the basic compressed sound expression. It is then determined to be used to make the basic reconstructed sound representation of the sound or sound field.

_{Further, in S3020, the value (eg, layer index) N B of the} highest layer (highest available layer) that will be used for decompressing the basic sound representation may be selected. The highest layer of improvement actually used for decompressing the basic sound representation is given by _{N B -1.} Since each layer contains exactly one improved side information payload (part of the improved side information), is the containing layer valid (eg, effectively received) based on the improved side information payload? Whether or not it can be determined. Thus, the selection is for all improved side information payloads ESI _m , m = 1, ..., M (or correspondingly).

m＝1,…,M）を使って達成できる。

It can be achieved by using m = 1,…, M).

In S3030, a basic reconstructed sound expression is obtained. The basic reconstructed sound representation provides basic side information from the components assigned to the highest available layers indicated by the first layer index and any layers below this highest available layer. It may be obtained using (or generally using basic side information).

The basic compressed sound representation components BSRC ₁ ,…, BSRC _J payloads are the basic side information payloads (eg, BSI or BSI _I and BSI _{D, m} , m = 1,…, M) (all) and It may be provided to the basic representation decompression processing unit 4200 together with the value N _B. Basic representation decompression processing unit 4200 (shown in FIGS. 4A and 4B), represented the lowest N _B number of layers, i.e. basic layer and N _B -1 amino enhancing layer (i.e., the first layer index Reconstruct the basic sound (or sound field) representation using only the basic compressed sound representation components contained within the layers). Alternatively, only the payload of the basic compressed sound representation ingredients are contained in the lowest N _B number of layers, with each of the basic side information payload, is provided in the basic representation decompression processing unit 4200 May be good.

The required information about which component of the basic compressed sound (or sound field) representation is contained in each layer is known to the decompressor 4100 from the data packet with the configuration settings information. It is assumed that it has been done. It is assumed that the configuration setting information is transmitted and received before the frame data packet.

Dependent side information data packets _{BSI D, m, m = 1} , ..., in order to provide N _B and increased side information data packets ESI _NE, all improved payloads, together with the value N _E and a value N _B , May be input to the partial parser 4400 (see FIG. 4B) of the decompression device 4100. The parser may drop all payloads and data packets that are not used for the actual decompression. If the value of N _E is equal to 0, all the improved side information data packets may be assumed to be empty.

If the base layer contains at least one dependent basic side information payload (part of the additional basic side information) corresponding to each layer, then each individual dependent basic side information payload (eg BSI _{D, m} , m = Decoding of 1, ..., N _B (part of additional basic side information)) is (i) decoding of said part of additional basic side information lower than its respective layer and each layer. Doing so by referring to the components assigned to any layer (preliminary decoding) and (ii) making the correction of the part of the additional basic side information to the best available layer and the best It may include doing (correction) by referring to the components assigned to any of the layers between the available layers. Here, the additional basic side information corresponding to each layer is that the decoding of one or more components of the components assigned to each layer is lower than that of each layer and each layer. Contains information to specify depending on other components assigned to any layer.

The basic reconstructed sound representation is then given basic side information and the best available layers from the components assigned to any layers below the best available layers and the best available layers. It can be obtained (eg, generated) using the corrected parts of the additional basic side information obtained from the parts of the additional basic side information corresponding to the layers up to.

In particular, _{the preliminary decoding of each payload BSI D, m} , m = 1, ..., N _B is a basic compression of the _{first J m-} 1 contained in the first m layer envisioned in the encoding stage. It may be involved in utilizing the dependence on the sound expression components BSRC ₁ , ..., BSRC _{(Jm) -1.}

Sequential correction of each payload BSI _{D, m} , m = 1,…, N _{B is the first N B} > where the basic sound component is more than expected for preliminary decoding. Also involved in considering the final reconstruction from the _{first J NB-} 1 basic compressed sound expression components BSRC ₁ , ..., BSRC _{(JNB) -1} contained in the m-layer. Good. Therefore, the correction may be achieved by discarding the outdated information. This is possible with the initially assumed attributes of the subordinate basic side information, that is, the subordination of each individual (complementary) component if some complementary component is added to the basic compressed sound representation. This is because of the attribute that the basic side information is a subset of the original.

In S3040, a second layer index may be determined. The second layer index may indicate the portion of the improvement side information (s) that should be used to improve (eg, improve) the basic reconstructed sound representation.

In addition to the first layer index, the index of increased side information payload to be used for decompression (portion of the second improvement information) (second layer index) N _E may be determined. The second layer index N _E may always be equal to or equal to the first layer index N _B. Improvements may always be achieved either according to the basic sonic representations obtained from the highest available layers, or not at all.

In S3050, the reconstructed sound representation of the sound or sound field is obtained (eg, generated) from the basic reconstructed sound representation with reference to the second layer index.

That is, the reconstructed sound representation is one of the improved side information (one of the improved side information) indicated by, for example, a second layer index, by (parametrically) improving or improving the basic reconstructed sound representation. It can be obtained by using the part). As will be described later, the second layer index may indicate that no improvement side information is used at this stage. Then, the reconstructed sound expression corresponds to the basic reconstructed sound expression.

For this purpose, the reconstructed basic sonic representations are all improved side information payloads ESI ₁ , ..., ESI _M , basic side information payloads (eg BSI or BSI _I and BSI _{D, m} , m = 1). , ..., M) and the value N _E are given to the improved representation decompression processing unit 4300 (shown in FIGS. 4A and 4B). The improved expression decompression processing unit 4300 _{uses only the improved side information payload ESI NE} , discards all other improved side information payloads, and calculates the final improved sound (or sound field) representation 2100'. .. Alternatively, instead of all the improved side information payloads, _{only the improved side information payload ESI NE} may be provided to the improved representation decompression processing unit 4300. If the value of N _E is equal to 0, all the improved side information payload is discarded (Alternatively, increased side information payload is not provided). And the reconstructed final improved sonic expression 2100'is equal to the reconstructed basic sonic expression. The improved side information payload ESI _NE may have been obtained by the partial parser 4400.

FIG. 3 also outlines decoding the compressed HOA representation based on the basic side information associated with the basic layer and based on the improved side information associated with one or more hierarchical improvement layers. ing.

It is understood that the steps described above may be performed in any order, and the exemplary order shown in FIG. 3 is not limited, unless one step requires another step as a prerequisite.

Next, details of layer selection (selection of first and second layer indexes) for decompression in steps S3020 and S3040 will be described.

The determination of the first layer index may involve, for each layer, determining whether the layer has been effectively received. The determination of the first layer index may further involve determining the first layer index as the layer index of the layer immediately below the lowest layer that was not effectively received. Whether or not a layer has been effectively received may be determined by assessing whether or not the layer's improved side information payload has been effectively received. This may be done by evaluating the validity flag in the improved side information payload.

The determination of the second layer index generally determines the second layer index to be equal to the first layer index, or does not use any improved side information when obtaining the reconstructed sonic representation. It may be involved in determining the index value (for example, index value 0) indicating that as the second layer index.

Number N of the highest layer (highest available layer) actually used for decompressing the basic sonic representation, where all frame data packets can be decompressed independently of each other. _{Both B and the index N E} of the improved side information payload used for decompression may be set to the highest number L of the valid improved side information payload. L itself can be determined by evaluating the validity flag in the improved side information payload. By leveraging knowledge of the size of each improved side information payload, it is possible to avoid complex parsing through the actual data of the payload to determine validity.

That is, the second layer index may be determined to be equal to the first layer index if the compressed sound representation for a series of time intervals can be independently decoded. In this case, the reconstructed basic sonic representation can be improved based on the improved side information payload of the highest available layer.

If differential decompression with inter-frame dependency is used, further decisions from previous frames need to be considered. In differential decompression, independent frame data packets are typically transmitted at regular time intervals. This is to allow decompression to start from those points. The independent frame data packets, determining the value N _B and N _E becomes frame independently, is performed as described above.

Effective improvement for the kth frame To elaborate on the proposed frame-dependent determination, the highest number (eg, layer index) of the side information payload is L (k), which compresses the basic sound representation. The highest layer number selected and used for _{decompression (eg layer index) is N B} (k) and the number of the improved side information payload used for decompression (eg layer index) is N _E (k). ).

Using this notation, the highest layer number N _B (k) used for decompressing the basic sonic representation is calculated according to the following equation:

N_B(k)がN_B(k−1)およびL(k)より大きくないように選ぶことによって、基本的な音表現の差分圧縮解除のために必要とされるすべての情報が利用可能であることが保証される。

By _{choosing N B} (k) _{not to be greater than N B} (k−1) and L (k), all the information needed for differential decompression of the basic sonic representation is available. Guaranteed to be.

That is, if the compressed sonic representations for a series of time intervals (eg, frames) cannot be decoded independently of each other, determining the first layer index means that for each layer, each layer It is determined whether or not the receipt is valid, and the first layer index for the given time interval is not effectively received with the first layer index for the time interval preceding the given time interval. It may include determining as the smaller of the layer indexes of the layer immediately below the lowest layer.

_{The number N E} (k) of the improved side information payload used for decompression may be determined according to the following equation.

ここで、N_E(k)についての0という選択は、再構成された基本的な音表現が、向上サイド情報を使って改善または向上されないことを示す。

Here, _{the choice of 0 for N E} (k) indicates that the reconstructed basic sonic representation is not improved or improved using the kaizen side information.

That is, specifically, the same corresponding improvement layer number is selected as long _{as the highest layer number N B} (k) used for decompressing the basic sound representation does not change. However, if N _B (k) changes _{, setting N E} (k) to 0 negates the improvement. Due to the expected differential decompression of the improved side information, _{that change based on N B} (k) is not possible. This is because it would be necessary to decompress the corresponding improved side information layer in the previous frame, as it is assumed that such decompression was not performed.

That is, if the compressed sound representations for a series of time intervals (eg, frames) cannot be decoded independently of each other, then the determination of the second layer index is the first for the given time interval. It may include determining whether the layer index of is equal to the first layer index for the preceding time interval. If the first layer index for the given time interval is equal to the first layer index for the preceding time interval, then the second layer index for the given time interval is the given time. It may be determined (eg selected) to be equal to the first layer index for the interval. On the other hand, if the first layer index for the given time interval is not equal to the first layer index for the preceding time interval, then no improvement side information is used when obtaining the reconstructed sonic representation. An index value indicating that may be determined (for example, selected) as the second layer index.

Alternatively, in decompression, if all of the improved side information payload with numbers up to N _E (k) is decompressed in parallel, the selection rule of the formula (4) is
N _E (k) = N _B (k) (9)
Will be replaced by.

Finally, for differential decompression _{, note that the number N B} of the top used layer can only be increased in independent frame data packets, while it can be decreased in any frame.

It is understood that the proposed method of encoding the layered structure of a compressed sound representation can be implemented by an encoder for encoding the layered structure of a compressed sound representation. Such an encoder may have a respective unit adapted to perform each of the above steps. An example of such an encoder 5000 is shown schematically in FIG. For example, such an encoder 5000 is adapted to perform a component subdivision unit 5010 adapted to perform S1010 described above, a component allocation unit 5020 adapted to perform S1020 described above, and S1030 described above. Even if it has a basic side information allocation unit 5030, an improved side information dividing unit 5040 adapted to execute the above-mentioned S1040, and an improved side information allocation unit 5050 adapted to execute the above-mentioned S1050. Good. In addition, each unit of such an encoder is adapted to perform the processing performed by each of the units, i.e., performing some or all of the above-mentioned steps of the proposed encoding method or further steps. It is understood that it may be embodied by the processor 5100 of the computing device adapted as such. The encoder or computing device may further have a memory 5200 accessible by the processor 5100.

In addition, a proposed method of decoding compressed sound representations encoded in multiple layers is by a decoder for decoding compressed sound representations encoded in multiple layers. It is understood that it can be implemented. Such a decoder may have its own unit adapted to perform each of the above steps. An example of such a decoder 6000 is shown schematically in FIG. For example, such a decoder 6000 executes a receiving unit 6010 adapted to perform S3010 described above, a first layer indexing unit 6020 adapted to perform S3020 described above, and S3030 described above. It has a basic reconstruction unit 6030 adapted to perform the above-mentioned S3040, a second layer index determination unit 6040 adapted to perform the above-mentioned S3040, and an improved reconstruction unit 6050 adapted to perform the above-mentioned S3050. You may be. In addition, each unit of such a decoder is adapted to perform the processing performed by each of the units, i.e., performing some or all of the aforementioned steps of the proposed decoding method or any further step. It is understood that it may be embodied by the processor 6100 of the computing device adapted as such. The decoder or computing device may further have a memory 6200 accessible by the processor 6100.

It should be noted that this article and the drawings merely show the principles of the proposed method and device. Therefore, it will be understood that those skilled in the art will be able to embody the principles of the present invention and devise various configurations within their spirit and scope, even if not explicitly described or illustrated in this paper. In addition, all examples given in this article are defined primarily for educational purposes only to help the reader in understanding the principles of the proposed methods and devices and the concepts that the inventor contributes to the advancement of the technology. Intended and construed without limitation to such individually described examples and conditions. Moreover, all statements and individual examples thereof herein describing the principles, aspects and embodiments of the present invention are intended to include their equivalents.

The methods and devices described herein may be implemented as software, firmware and / or hardware. Certain components may be implemented, for example, as software running on a digital signal processor or microprocessor. Other components may be implemented, for example, as hardware and / or as a purpose-built integrated circuit. The signals that emerge in the methods and devices described may be stored on media such as random access memory or optical storage media, such as radio networks, satellite networks, wireless or wired networks, such as the Internet. It may be transferred via.

Claims (15)

A method of decoding a compressed higher-order Ambisonics (HOA) sound representation of a sound or sound field encoded in multiple layers using layered encoding, the method of which is:
At the stage of receiving a bitstream containing the compressed HOA representation corresponding to the plurality of hierarchical layers including the basic layer and at least one hierarchical layer, at least of the plurality of hierarchical layers. One includes the components of the basic compressed sound representation of the sound or sound field, the components of which correspond to multiple monaural signals, with the steps;
Including the step of decoding the compressed HOA representation based on the basic side information associated with the base layer and based on the improvement side information associated with the at least one hierarchical improvement layer.
A method of indicating that the basic side information represents a directional signal having an incident direction in which the first individual monaural signal is present. A non-transitory computer-readable storage medium containing instructions that perform the method of claim 1 when executed by a processor. The basic side information further includes basic dependent side information related to a second individual monaural signal of the plurality of monaural signals that is decoded depending on the other monaural signal of the plurality of monaural signals. , The method according to claim 1. The method of claim 3, wherein the basic dependent side information includes a vector-based signal that is directionally distributed within the sound field, the directional distribution being specified by a vector. The method of claim 4, wherein the vector component is set to 0 and is not part of the compressed vector representation. The method of claim 1, wherein the improved side information includes parameters related to at least one of spatial prediction, subband directional signal synthesis and parametric ambient sound replication. The method of claim 1, wherein the improved side information includes information that allows prediction of a sound or a missing portion of the sound field from a directional signal. For each layer, determine if each layer was effectively received;
Further including determining the layer index of the layer immediately below the lowest layer that has not been effectively received,
The method according to claim 1. A device for decoding high-order Ambisonics (HOA) sound representations of sound or sound fields encoded in multiple layers using layered encoding, which is:
A receiver that receives a bitstream containing the compressed HOA representation corresponding to the plurality of hierarchical layers, including a basic layer and at least one hierarchical layer, of the plurality of hierarchical layers. At least one contains components of the basic compressed sound representation of the sound or sound field, the components of which correspond to multiple monaural signals with a receiver;
With a decoder configured to decode the compressed HOA representation based on the basic side information associated with the base layer and based on the improvement side information associated with the at least one hierarchical improvement layer. Have,
The basic side information comprises designating at least one monaural signal representing a directional signal having a certain incident direction. The basic side information further includes basic dependent side information related to a second individual monaural signal of the plurality of monaural signals that is decoded depending on the other monaural signal of the plurality of monaural signals. , The apparatus according to claim 9. The apparatus according to claim 10, wherein the basic dependent side information includes a vector-based signal directionally distributed in the sound field, and the directional distribution is specified by a vector. 9. The apparatus of claim 9, wherein the vector component is set to 0 and is not part of the compressed vector representation. 9. The apparatus of claim 9, wherein the improved side information includes parameters related to at least one of spatial prediction, subband directional signal synthesis and parametric ambient sound replication. The device according to claim 9, wherein the improved side information includes information that allows prediction of a sound or a missing portion of the sound field from a directional signal. For each layer, determine if each layer was effectively received;
Further including determining the layer index of the layer immediately below the lowest layer that has not been effectively received,
The device according to claim 9.

Images