RU2793271C1 - Systems, methods and equipment for converting from channel-oriented audio to object-oriented audio - Google Patents

Abstract

FIELD: audio processing.

SUBSTANCE: bit stream is received including channel-based audio and associated channel-based audio metadata. The service signal broadcast parameter is parsed from the channel-oriented audio metadata, wherein the service signal broadcast parameter specifies one of a plurality of different object audio metadata (OAMD) representations. Each of the OAMD representations maps one or more channel-based audio channels into one or more audio objects. The channel-specific metadata is converted to OAMD associated with one or more audio objects using the OAMD representation. Channel mixing information is generated based on OAMD channel ordering constraints. The audio channels of the channel-based audio are reordered based on the channel mixing information. The reordered channel-oriented audio and OAMD are coded into a stream of object-oriented audio bits.

EFFECT: increase of audio processing efficiency.

15 cl, 26 dwg

Description

Cross-references to related applications

[0001] This application claims priority of U.S. Provisional Application No. 62/942322, filed December 2, 2019, and EP Application No. 19212906.2, filed December 2, 2019, both of which are incorporated herein in their entirety by reference .

The technical field to which the invention belongs

[0002] This disclosure relates generally to audio signal processing, including converting channel-based audio to object-based audio.

State of the art

[0003] In channel-based audio (CBA) coding, a track set is implicitly assigned to specific speakers by associating the track set with a channel configuration. If the playback speaker configuration differs from the scrambled channel configuration, downmix or upmix specifications are required in order to redistribute audio to the available speakers. This paradigm is known and works when the channel configuration on the decoding side can be predetermined or assumed with reasonable certainty as 2.0, 5.X or 7.X. However, due to the popularity of new speaker layouts, speculation regarding the speaker layout used for playback cannot be made. Therefore, CBA does not offer a sufficient way to adapt a presentation in which the original speaker layout does not match the speaker layout on the decoding side. This presents a problem when trying to author content that plays well regardless of the speaker configuration.

[0004] When encoding object-oriented audio (OBA), rendering is applied to objects that contain the audio entity of the objects in combination with metadata that contains separately assigned properties of the objects. Properties (such as x, y, z position, or channel location) more explicitly indicate how the content creator targets the audio content to be rendered (i.e., they impose restrictions on how to render the entity to the speakers ). Because individual audio elements can be associated with a much richer set of metadata, given the semantic meaning of the elements, the way to adapt to the audio speaker configuration may provide better information on how to prepare by rendering to fewer speakers.

[0005] Several standardized formats are provided for the transmission of CBA content, such as Enhanced AC-3 (E-AC-3) specified in ETSI TS 102 366 [1]. To ensure compatibility with already existing devices, Joint Object Coding (JOC) can be used in conjunction with standardized CBA formats to transport OBAs. JOC delivers immersive audio at low bit rates achieved by transmitting multi-channel downmix of immersive content using perceptual audio coding algorithms along with parametric assistance information that enables audio objects to be recovered from the downmix at the decoder. In some applications, such as television broadcasts, it is desirable to present CBA content as OBA content such that the content is compatible with the installation base of OBA playback devices. However, the standardized bitstream formats for CBA and OBA are not fully compatible.

The essence of the invention

[0006] Embodiments are disclosed for converting CBA content to OBA content and, in a specific embodiment, for converting 22.2-channel content to OBA content for playback on OBA-compatible playback devices.

[0007] In an embodiment, the method comprises: receiving, by one or more audio processing equipment processors, a bit stream including channel-based audio and associated channel-based audio metadata; wherein the one or more processors are configured to: parse a signaling parameter from the channel-oriented audio metadata, wherein the signaling parameter specifies one of a plurality of different object audio metadata (OAMD) representations, each of the OAMD representations converting one or more audio channels channel-wise -oriented audio into one or more audio objects; convert the channel-specific metadata to OAMD associated with one or more audio objects using the OAMD representation that is indicated by a signaling parameter; generate channel mixing information based on OAMD channel ordering constraints; reordering the audio channels of the channel-based audio based on the channel mixing information to generate the re-ordered channel-based audio; and prepare by rendering the reordered channel-oriented audio into the prepared by rendering audio using OAMD; or encode the reordered channel-oriented audio and OAMD into an object-oriented audio bit stream and transmit the object-oriented audio bit stream to a playback device or a source device.

[0008] In an embodiment, the channel-specific audio and metadata are included in the native audio bit stream, and the method further comprises decoding the native audio bitstream in order to recover (i.e., determine or extract) the channel-specific audio and metadata.

[0009] In an embodiment, the channel-based audio and metadata is N.M channel-based audio and metadata, where N is a positive integer greater than nine and M is a positive integer greater than or equal to zero.

[0010] In an embodiment, the method further comprises: determining a first set of channel-based audio channels that are capable of being represented via overlaid OAMD channels; assigning tagged OAMD channels to the first channel set; determining a second set of channel-based audio channels that are not capable of being represented by overlaid OAMD channels; and assigning static positional OAMD coordinates to the second set of channels.

[0011] In an embodiment, the method comprises: receiving, by one or more audio processing equipment processors, a bitstream including channel-based audio and metadata; wherein the one or more processors are configured to: encode the channel-specific audio into a native audio bit stream; parse a signaling parameter from the metadata, wherein the signaling parameter specifies one of a plurality of different object audio metadata (OAMD) representations; convert the channel-oriented metadata to OAMD using the OAMD representation, which is indicated by the signaling parameter; generate channel mixing information based on OAMD channel ordering constraints; generate a bitstream packet that includes a native audio bitstream, channel mixing information, and OAMD; multiplex the packet into a transport layer bitstream; and transmit the transport layer bit stream to the playback device or source device.

[0012] In an embodiment, the channel-based audio and metadata is N.M channel-based audio and metadata, where N is a positive integer greater than seven and M is a positive integer greater than or equal to zero.

[0013] In an embodiment, channels in channel-based audio that can be represented by OAMD-padded channel labels use OAMD-padded channel labels, and channels in channel-based audio that cannot be represented by static object positions use labels. overlaid OAMD channels, where each position of the static object is described in positional OAMD coordinates.

[0014] In an embodiment, the transport bit stream is a Motion Picture Expert Group (MPEG) audio bit stream that includes a signal that indicates the presence of an OAMD in an extension field of the MPEG audio bit stream.

[0015] In an embodiment, a signal that indicates the presence of OAMD in the MPEG audio bit stream is included in a reserved metadata field in the MPEG audio bit stream for transmission in surround sound mode signaling.

[0016] In an embodiment, the method comprises: receiving, by one or more audio processing equipment processors, a transport layer bitstream including a packet; wherein the one or more processors are configured to: demultiplex the transport layer bit stream in order to recover (ie, determine or extract) the packet; decode the packet to recover (ie, determine or extract) the native audio bit stream, channel mixing information, and object audio metadata (OAMD); decode the native audio bit stream in order to recover the channel-specific audio bit stream and metadata; reorder channels of the channel-based audio based on the channel mixing information; and prepare by rendering the reordered channel-oriented audio into the prepared by rendering audio using OAMD; or encode the channel-oriented audio and OAMD into an object-oriented audio bit stream and transmit the object-oriented audio bit stream to the source device.

[0017] In an embodiment, the channel-based audio and metadata is N.M channel-based audio and metadata, where N is a positive integer greater than seven and M is a positive integer greater than or equal to zero.

[0018] In an embodiment, the method further comprises: determining a first set of channel-based audio channels that are capable of being represented via overlaid OAMD channels; assigning tagged OAMD channels to the first channel set; determining a second set of channel-based audio channels that are not capable of being represented by overlaid OAMD channels; and assigning static positional OAMD coordinates to the second set of channels.

[0019] In an embodiment, the transport bit stream is a Motion Picture Expert Group (MPEG) audio bit stream that includes a signal that indicates the presence of an OAMD in an extension field of the MPEG audio bit stream.

[0020] In an embodiment, a signal that indicates the presence of OAMD in the MPEG audio bit stream is included in a reserved data structure field in the MPEG audio bit stream metadata for transmission in surround sound mode signaling.

[0021] In an embodiment, the hardware comprises: one or more processors; and a non-volatile computer-readable storage medium having stored instructions that, when executed by one or more processors, instructs one or more processors to perform the methods described herein.

[0022] Other embodiments disclosed herein are directed to systems, equipment, and computer-readable media. Details of the disclosed implementations are set forth in the accompanying drawings and in the description below. Other features, purposes and advantages should become apparent from the description, drawings and claims.

[0023] The specific embodiments disclosed herein provide one or more of the following advantages. The existing installed base of OBA-compatible playback devices can convert CBA content to OBA content using existing standardized native audio and transport bitstream formats, without changing the hardware components of the playback devices.

Description of drawings

[0024] In the accompanying drawings, referred to below, various embodiments are illustrated in block diagrams, flowcharts, and other diagrams. Each block in the flowcharts or block may represent a module, program, or piece of code that contains one or more executable instructions to perform the specified logical functions. Although these blocks are illustrated in specific sequences for performing the steps of the methods, they may not necessarily be performed strictly in accordance with the illustrated sequence. For example, they may be performed in reverse order or simultaneously, depending on the nature of the respective operations. It should also be noted that the flowcharts and/or each block in the flowcharts, and combinations thereof, may be implemented by a dedicated software or hardware system to perform said functions/operations, or by a combination of specialized hardware and computer instructions.

[0025] FIG. 1A is a table showing sub-channel and object positions for two different object audio metadata (OAMD) representations, according to an embodiment.

[0026] FIG. 1B is a table showing sub-channel assignment and channel ordering for two different OAMD representations, according to an embodiment.

[0027] FIG. 2A is a table showing sizing metadata according to an embodiment.

[0028] FIG. 2B is a table showing trim/balance controls according to an embodiment.

[0029] FIG. 3 is a block diagram of a system for converting a 22.2-channel audio bitstream into audio objects and OAMD without using bitstream coding, according to an embodiment.

[0030] FIG. 4 is a block diagram of a system for converting a 22.2-channel audio bitstream into audio objects and OAMD using bitstream coding, according to an embodiment.

[0031] FIG. 5 is a block diagram of a system for converting a 22.2-channel audio bit stream into audio objects and OAMD for rendering at a source device, according to an embodiment.

[0032] FIG. 6A and 6B are block diagrams of a system for converting a 22.2-channel audio bit stream into audio objects and OAMD for high definition multimedia interface (HDMI) transmission for external rendering, according to an embodiment.

[0033] FIG. 7A-7C are block diagrams of a system for converting a 22.2-channel audio bit stream into audio objects and OAMD, wherein the channel mixing and OAMD information is packaged in a native audio bit stream, according to an embodiment.

[0034] FIG. 8A and 8B are a block diagram of a system for converting a 22.2-channel audio bit stream into audio objects and OAMD, wherein the churn information and OAMD are packaged in a native audio bit stream for rendering at a source device, according to an embodiment.

[0035] FIG. 9A-9C are block diagrams of a system for converting a 22.2-channel audio bit stream into audio objects and OAMD, wherein the channel mixing and OAMD information is embedded in the transport layer for delivery to source devices and then packaged in the native audio bit stream for transmission over HDMI, according to embodiment.

[0036] FIG. 10A and 10B are block diagrams of a system for converting a 22.2-channel audio bit stream into audio objects and OAMD, with channel mixing and OAMD information embedded in the transport layer for rendering in source devices, according to an embodiment.

[0037] FIG. 11 is a flowchart for a CBA-OBA conversion process, according to an embodiment.

[0038] FIG. 12 is a flowchart for an alternative CBA-OBA conversion process, according to an embodiment.

[0039] FIG. 13 is a flowchart for an alternative CBA-OBA conversion process, according to an embodiment.

[0040] FIG. 14 is a flowchart for an alternative CBA-OBA conversion process, according to an embodiment.

[0041] FIG. 15 is a flowchart for an alternative CBA-OBA conversion process, according to an embodiment.

[0042] FIG. 16 is a flowchart for an alternative CBA-OBA conversion process, according to an embodiment.

[0043] FIG. 17 is a block diagram of an exemplary audio system architecture that includes mapping audio channels to audio objects, according to an embodiment.

[0044] The same reference number used throughout the drawings indicates like elements.

Detailed description of the invention

General view

[0045] Object audio metadata (OAMD) is a representation of coded metadata bitstreams for OBA processing, such as, for example, the metadata described in ETSI TS 103 420 v1.2.1 (2018-10). The OAMD bit stream may be carried in an Extensible Metadata Delivery Format (EMDF) container, for example, as specified in ETSI TS 102 366[1]. OAMDs are used to render the audio object. Rendering information can change dynamically (eg gain and position). The elements of the OAMD bit stream may include content description metadata, object property metadata, property update metadata, and other metadata.

[0046] In an embodiment, the content description metadata includes the syntax version of the working OAMD data, the total number of objects, object types, and program composition. Object property metadata includes the object's position in room-referenced, screen-referenced, or speaker-referenced coordinates, object size (width, depth, height), priority (imposes ordering by importance on objects where higher priority indicates higher importance for the object), gain (used to apply a individually adjustable gain value to the object), channel blocking (used to restrict the rendering of the object to a single speaker, resulting in non-diffused, tone-neutral audio reproduction), zone restrictions (indicate zone or sub-volume in the listening environment in which an object is excluded or included), object divergence (used to convert an object into two objects, with the energy spread along the x-axis), and object adjustment (used to reduce the level of off-screen elements , which are specified in the mix).

[0047] In an embodiment, property update metadata signaling timing data applicable to updates for all transmitted objects. The time synchronization data of the transmitted property update indicates the start time for the update, along with the update context with previous or subsequent updates, and the time duration for the interpolation process between successive updates. The OAMD bit stream syntax supports up to eight property updates per object per codec frame. The number of updates signaled or the start and stop time of each property update is the same for all objects. The metadata indicates the OAMD duration ramp value that indicates the time period in audio samples to interpolate from the signaled property values of the objects of the previous property update to the values of the current update.

[0048] In an embodiment, the timing data also includes a sample offset value and a block offset value, which are used by the decoder to calculate an initial sample value offset and a frame offset . The sample offset is the time offset in samples relative to the first PCM audio sample to which the data in the OAMD payload is applied, for example, as specified in ETSI TS 102 366 [1], sections H.2.2.3.1 and H.2.2.3.2. The block offset value specifies a period of time in samples as an offset from the sample offset common to all property updates.

[0049] In an embodiment, the decoder provides an interface to an OBA containing the audio entity audio data of the objects and timestamped metadata updates for the corresponding properties of the objects. At the interface, the decoder provides the decoded metadata per object in timestamped updates. For each update, the decoder provides the data specified in the metadata update structure.

Sample CBA-OBA conversion

[0050] In the disclosure below, technologies are disclosed for converting CBA content to OBA using OAMD. In an exemplary embodiment, 22.2-channel ("22.2ch") content is converted to OBA using OAMD. In this embodiment, the 22.2-channel content has two predetermined ways in which the channels are positioned and therefore downmixed/prepared by rendering. The choice of method may depend on the value of a parameter, such as the parameter dmix_pos_adj_idx embedded in the 22.2-channel bitstream. The format converter that converts the 22.2-channel location to an OAMD representation chooses one of the two OAMD representations based on the value of this parameter. The selected representation is carried in an OBA bitstream (eg, a Dolby® MAT bitstream) that is input to a playback device (eg, a Dolby® Atmos® playback device). An exemplary 22.2 channel system is a Hamasaki 22.2. Hamasaki 22.2 is a Super Hi-Vision surround sound component, which is a television standard developed by NHK Science and Technical Research Laboratories that uses 24 speakers (including two subwoofers) arranged in three layers.

[0051] Although the following disclosure is directed to an embodiment in which 22.2-channel content is converted to OBA content using OAMD, the disclosed embodiments are applicable to any CBA or OBA bit stream format including standardized or proprietary formats. bit stream, and to any playback device or system. Additionally, the following disclosure is not limited to a 22.2-channel-to-OBA transform, but is also applicable to transforming any N.M channel-based audio, where N is a positive integer greater than seven and M is a positive integer greater than or equal to zero.

[0052] As used herein, the term "includes" and variations thereof should be read as multivariate terms that mean "includes, but not limited to". The term "or" should be read as "and/or" unless the context clearly indicates otherwise. The term "based on" should be read as "at least partially based on". The terms "one exemplary embodiment" and "exemplary embodiment" should be read as "at least one exemplary embodiment". The term "another embodiment" should be read as "at least one other embodiment". In addition, in the description below and in the claims, unless otherwise indicated, all technical and scientific terms used herein have the same meaning as is commonly understood by those skilled in the art to which this disclosure belongs.

Assignment of programs and positions of objects

[0053] In this application, 22.2-channel content 305 (eg, a file or a live stream) is received by a format converter 301. Content 305 includes audio and associated metadata. The metadata includes a dmix_pos_adj_idx parameter to select one of two OAMD representations based on the value of this parameter. Channels that can be represented by overlaid OAMD channel labels use overlaid OAMD channel labels. Channels that cannot be represented by sub-OAMD channel labels use static object positions, where each static object position is described in OAMD positional coordinates [x, y, z], for example, as described in ETSI TS 103 420 v1. 2.1 (2018-10). As used herein, a "sub-channel" is a group of multiple sub-objects, and a "sub-object" is a static object whose spatial position is fixed by assigning a speaker to a playback system.

[0054] FIG. 1A is a table showing sub-channel and object positions for two different OAMD representations, according to an embodiment. The top row of the table includes twenty-four 22.2-channel labels, the middle row of the table includes the sub-channel labels and object positions for the first OAMD representation signaled by dmix_pos_adj_idx=0, and the bottom row of the table includes the sub-channel labels and object positions for the second OAMD representation signaled by dmix_pos_adj_idx=1. Note that the dmix_pos_adj_idx signal is an exemplary signal, and any type of signaling may be used, including, but not limited to, boolean flags and signals encoded with one or more bits.

[0055] Referring to the table in FIG. 1A, some examples of 22.2-channel tags include front left (FL), front right (FR), front center (RC), low frequency effects 1 (LFE1), rear left (BL), rear right (BR), front Left Center (FLc), Front Right Center (FRc), Rear Center (BC), LFE2 (LFE2), Left Wide (SIL), Right Wide (SIR), Top Front Left (TpFL), Top Front Right ( TpFR), Top Anterior Center (TpFC), Top Center (TpC), Top Back Left (TpBL), Top Back Right (TpBR), Top Side Left (TpSIL), Top Side Right (TpSIR), Top Back Center (TpBC) , between front left (BtFL), between front right (BtFR) and between front center (BtFC). It should be noted that these labels are converted either into labels of embedded OAMD channels or into positions of static objects [x, y, z]. For example, for the first OAMD representation (dmix_pos_adj_idx=0), a 22.2-channel FL label is mapped to a static object position [0,0.25,0], a 22.2-channel FR label is mapped to a static object position [1, 0.25, 0], 22.2 The FC-channel label is mapped to the label C of the overlaid OAMD-channel, and so on. The OAMD representation maps one or more audio channels to one or more audio objects based on (eg, the value of) a transmission parameter in the overhead. One or more audio objects may be dynamic or static audio objects. As defined above, a static audio object is an audio object having a fixed spatial position. A dynamic audio object is an audio object whose spatial position can change over time. In the example above, the OAMD representation contains channel labels, sub-channel labels, and static object positions. The OAMD representation maps channel labels to either sub-channel labels or static object positions based on (eg, the value of) a transmission parameter in the signaling.

Assignment of programs and positions of objects

[0056] OAMDs assume that embedded objects precede dynamic objects. Additionally, the underlying objects appear in a specific order. For these reasons, audio for 22.2-channel content is reordered by the audio channel mixing unit 303 in order to satisfy OAMD ordering constraints. The audio channel mixing unit 303 receives the channel mixing information from the metadata generator 304 and uses the channel mixing information to reorder the 22.2 channels.

[0057] FIG. 1B is a table showing sub-channel assignment and channel ordering for two different OAMD representations, according to an embodiment. The top row of the table shows the suggested channel order (0-23 channels) and channel labels for 22.2 channel content (Hamasaki 22.2). The middle row of the table shows the sub-assignment labels for the first OAMD representation, and the bottom row of the table shows the sub-assignment labels for the second OAMD representation. The converted audio and OAMD metadata is output, with reference to FIG. 3 through the format converter 301 to an audio object renderer 302 that generates rendered audio.

[0058] Referring to the table in FIG. 1B, the first two channels (0, 1) of 22.2-channel content are FL and FR. For the first OAMD representation (dmix_pos_adj_idx=0), the first two channels (0,1) are reordered ("mixed") into OAMD channels 15 and channel 16, respectively. For the second OAMD representation (dmix_pos_adj_idx=1), the first two channels (0,1) of the 22.2-channel content are reordered into the overlaid OAMD channels L and R, respectively. In this example, for the first OAMD view (dmix_pos_adj_idx=0), for the first output channel at index 0, to associate the first OAMD view with it, input channel index 6 (e.g. Hamasaki 22.2) is reordered/shuffled such that it becomes channel index 0. In other words, in this example, if the left channel (L) is present in the input padded channels, then the left channel in the first OAMD representation is forced to be the first channel (with channel index 0). All spoofed channels, if any, appear in a specific order when presented to OAMD. After the padded channels are reordered, the dynamic objects are reordered as a result of the padded channels reordering. The reordering satisfies certain restrictions on the order of the OAMD representation. The restrictions depend on the OAMD specification that is used by the OBA device/playback system. For example, for an OBA device/playback system compatible with Dolby Atmos, the OAMDs transmitted in systems and codecs containing Dolby Atmos content are specified by Dolby Atoms OAMD specifications. These specifications/constraints define the order of the overlaid OAMD channels as being, for example, what is shown in FIG. 1A, and as follows, with the respective channel labels in brackets: Left (L), Right (R), Center (C), Low Frequency Effects (LFE), Surround Left (Ls), Surround Right (Rs), Left surround back (Lrs), right surround back (Rrs), left front high (Lfh), right front high (Rfh), left top middle (Ltm), right top middle (Rtm), left back high (Lrh), right rear high (Rrh) and low frequency effects 2 (LFE2).

Sizing Metadata

[0059] FIG. 2A is a table showing sizing metadata according to an embodiment. To ensure that rendering of 22.2-channel content to OBA content is performed as closely matching the downmixes indicated by the 22.2-channel specification, sizing metadata is included in the OAMD that is attached to the 22.2-channel content delivered to the OBA renderer. . Object trimming is used to lower the level of off-screen elements that are included in the mix. This may be required when the immersive mix is played back in layouts with a small number of speakers.

[0060] In an embodiment, the first metadata field includes a warp_mode parameter which, if set to a value of 0, indicates normal rendering (ie, no warping) of objects in output 5.1.X configurations. If warp_mode is set to "1", warp is applied to objects in the output 5.1.X configuration. Curvature refers to how the renderer handles issues related to content being panned between the midpoint and the back of the listening environment (such as a room). By curving, content is presented at a constant level in the surround speakers between the rear side and midpoint of the listening environment, completely eliminating the need for ghost imaging until it is in the front half of the listening environment.

[0061] The second metadata field in the sizing metadata table includes trim/balance controls for each configuration for eight speaker configurations (e.g., 2.0, 5.1.0, 7.1.0, 2.1.2, 5.1.2, 7.1.2 , 2.1.4, 5.1.4, 7.1.4), as shown in Fig. 2b. Metadata fields are provided for auto trim (auto_trim), center trim (center_trim), surround trim (surround_trim), height trim (height_trim), and front/rear balance trim (fb_balance_ohfl, fb_balance_surr).

[0062] With reference to the table of FIG. 2A, the third metadata field includes an object_trim_bypass parameter which has a value that applies to all padded and dynamic objects in the 22.2 channel content. If object_trim_bypass is set to "1", trimming is not applied to inline and dynamic objects.

Object Gain

[0063] OAMD allows each object to have a separate object gain (described by the object_gain field). This gain is applied by the audio object renderer 302 . Object Gain provides the ability to compensate for differences between 22.2-channel content downmix values and rendering OAMD representations of 22.2-channel content. In an embodiment, the object gain is set to -3 dB for objects with sub-channel assignments in LFE1 or LFE2, and to 0 dB for all other objects. Other values for object gain may be used depending on the application.

Exemplary Applications

Listening to 22.2 content as an OBA

[0064] FIG. 3 is a block diagram of an exemplary system 300 for converting a 22.2-channel audio bitstream to audio and OAMD without using bitstream coding, according to an embodiment. System 300 is used in an application in which 22.2 channel content is heard as OBA content on an OBA playback system (Dolby® Atmos®).

[0065] System 300 includes a format converter 301 and an audio object renderer 302 . The format converter 301 further includes an audio mixing module 303 and an OAMD metadata generator 304 . Some examples of OAMD metadata include, but are not limited to, content description metadata, property update metadata, and customization data. The 22.2-channel content 305 (eg, file or live stream) includes 22.2-channel audio and metadata that is input to the format converter 301 . The OAMD metadata generator 304 converts the 22.2-channel metadata to OAMD, for example, in accordance with the principles as described with respect to FIG. 1A and generates channel mixing information. The channel mixing information describes the channel reordering of the 22.2-channel content that is applied by the audio channel mixing unit 303, for example, in accordance with the principles as described with respect to FIG. 1b. The output of the audio channel mixing module 303 is the reordered audio channels. The output of the format converter 301 is the reordered audio and OAMD channels that are input to the audio object renderer 302 . The audio object renderer 302 processes the audio using OAMD to tailor it to a particular speaker layout.

Transferring 22.2 content as an OBA

[0066] FIG. 4 is a block diagram of an exemplary system 400 for converting a 22.2-channel audio bitstream into audio objects and OAMD using bitstream coding, according to an embodiment. In this application, instead of transmitting 22.2-channel content, 22.2-channel content is subjected to format conversion and transmitted as an OBA using an OBA codec.

[0067] The system 400 includes a format converter 401 and an OBA encoder 402. The format converter 401 further includes an OAMD metadata generator 404 and an audio mixing module 403. Some examples of OAMD metadata include, but are not limited to, content description metadata, property update metadata, and customization data. The 22.2-channel content 405 (eg, file or live stream) includes 22.2-channel audio and metadata that is input to the format converter 401 . The OAMD metadata generator 404 converts the 22.2-channel metadata to OAMD, for example, in accordance with the principles as described with respect to FIG. 1A and generates channel mixing information. The channel mixing information describes the channel reordering of the 22.2-channel content that is applied by the audio channel mixing unit 403, for example, in accordance with the principles as described with respect to FIG. 1b. The output of the audio channel mixing module 403 is the reordered audio channels.

[0068] The output of the format converter 401 is the reordered audio and OAMD channels that are input to the OBA encoder 402. The OBA encoder 402 encodes the audio using OAMD (eg, using JOC) to generate an OBA bit stream 406, which may be sent to the downstream OBA player, where it is rendered by an audio object renderer that processes the audio to adapt it to a particular speaker layout.

Converting the transmitted 22.2 content to OBA for rendering on the source device

[0069] FIG. 5 is a block diagram of an exemplary system for converting a 22.2-channel audio bit stream into audio objects and OAMD for rendering at a source device, according to an embodiment. In this application, a source device such as a set-top box (STB) or an audio/video recorder (AVR) receives 22.2-channel content from its own audio bit stream, and after format conversion by a format converter, the content is prepared by rendering using a renderer audio objects. An exemplary native audio bitstream formant is the Advanced Audio Coding (AAC) bitstream format.

[0070] The system 500 includes a format converter 501 and an audio object renderer 502 and a decoder 506. The format converter 501 further includes an OAMD metadata generator 504 and an audio channel mixing module 503. Some examples of OAMD metadata include, but are not limited to, content description metadata, property update metadata, and customization data. An audio bit stream 505 (eg, AAC/MP4) includes 22.2-channel audio and metadata that is input to a decoder 506 (eg, an AAC/MP4 decoder). The output of the decoder 506 is 22.2-channel audio and metadata that is input to the format converter 501. The OAMD metadata generator 504 converts the 22.2-channel metadata to OAMD, for example, in accordance with the principles as described with respect to FIG. 1A and generates channel mixing information. The channel mixing information describes the channel reordering of the 22.2-channel content that is applied by the audio channel mixing unit 503, for example, in accordance with the principles as described with respect to FIG. 1b. The output of the audio channel mixing module 503 is the reordered audio channels. The output of the format converter 501 is the reordered audio and OAMD channels that are input to the audio object renderer 502 . The audio object renderer 502 processes the audio using OAMD to tailor it to a particular speaker layout.

Converting 22.2 transmitted content to OBA for HDMI transmission for external rendering (STBA/VR/SB)

[0071] FIG. 6A and 6B are a block diagram of an exemplary system for converting a 22.2-channel audio bit stream into audio objects and OAMD for transmission over a High Definition Multimedia Interface (HDMI) for external rendering, according to an embodiment. In this application, the channel mixing information as well as the OAMD are generated in the encoder and packetized in the native audio bit stream (eg AAC) to be transmitted. In this configuration, the format conversion that occurs is simplified into an audio mixing unit. The shuffled audio along with the OAMD is sent to the OBA encoder for bitstreaming over HDMI. On the receiver side, the bitstream is decoded and rendered by the audio object renderer.

[0072] Referring to FIG. 6A, the encoding system 600A includes a format converter 601 and an OBA encoder 602 and a decoder 606. The format converter 601 further includes an OAMD metadata generator 604 and an audio channel mixing module 603. Some examples of OAMD metadata include, but are not limited to, content description metadata, property update metadata, and customization data. Native audio bit stream 605 (eg, AAC/MP4) includes 22.2-channel audio and metadata that is input to a decoder 606 (eg, AAC/MP4 decoder). The output of the decoder 606 is 22.2-channel audio and metadata that is input to the format converter 601 . The OAMD metadata generator 604 converts the 22.2-channel metadata to OAMD, for example, in accordance with the principles as described with respect to FIG. 1A and generates channel mixing information. The channel mixing information describes the channel reordering of the 22.2-channel content that is applied by the audio channel mixing unit 603, for example, in accordance with the principles as described with respect to FIG. 1b. The output of the audio channel mixing module 603 is the reordered audio channels. The output of the format converter 601 is the reordered audio and OAMD channels that are input to the OBA encoder 602. The OBA encoder 602 encodes the audio and OAMD and outputs an OBA bit stream that includes audio and OAMD.

[0073] Referring to FIG. 6B, the decoding system 600B includes an OBA decoder 607 and an audio object renderer 608. The OBA bit stream is input to the OBA decoder 607, which outputs audio and OAMD, which are input to the audio object renderer 608 . The audio object renderer 608 processes the audio using OAMD to tailor it to a particular speaker layout.

Transmission of pre-calculated 22.2-OAMD via proprietary HDMI bitstream

[0074] FIG. 7A-7C are block diagrams of exemplary systems for converting a 22.2-channel audio bit stream into audio objects and OAMD, wherein the channel mixing and OAMD information is packaged in a native audio bit stream, according to an embodiment. In the previous exemplary applications, OAMDs are generated after a decoder (eg, an AAC decoder). However, it is possible to embed the channel mixing and OAMD information in the transmission format (in the native audio bitstream or in the transport layer), as an alternative implementation. In this application, the channel mixing information as well as the OAMD are generated in the encoder and packetized in the native audio bitstream (eg, AAC bitstream) to be transmitted. In this configuration, the format conversion that occurs is simplified into an audio mixing unit. The shuffled audio along with the OAMD is sent to the OBA encoder for transmission over HDMI. On the receiving side, the OBA bit stream is decoded and rendered using an audio object renderer.

[0075] Referring to FIG. 7A, encoding system 700A includes an encoder 701 (eg, an AAC encoder) and a transport layer mux 706. The encoder 701 further includes a core encoder 702, a format converter 703, and a bitstream packetizer 705. The format converter 703 further includes an OAMD metadata generator 704, which may be a Dolby Atmos metadata generator, for example. Some examples of OAMD metadata include, but are not limited to, content description metadata, property update metadata, and customization data.

[0076] Native audio bit stream 707 (eg, AAC/MP4) includes 22.2-channel audio and metadata. The audio is input to the base encoder 702 of the encoder 701, which encodes the audio into a native audio format and outputs the encoded audio to the bitstream packetizer 705 . The OAMD metadata generator 704 converts the 22.2-channel metadata to OAMD, for example, in accordance with the principles as described with respect to FIG. 1A and generates channel mixing information. The channel mixing information describes channel reordering of 22.2-channel content, for example, in accordance with the principles as described with respect to FIG. 1b. The channel mixing information is input to the bit stream packetizer 705 along with the OAMD. The output of the bitstream packetizer 705 is a native audio bitstream that includes channel mixing and OAMD information. The native audio bit stream is input to the transport layer multiplexer 706, which outputs a transport stream that includes the native audio bit stream.

[0077] Referring to FIG. 7B, the decoding/encoding system 700B includes a transport layer demultiplexer 708, a decoder 709, an audio mixing unit 710, and an OBA encoder 711. The transport layer demultiplexer 708 demultiplexes the audio and OAMD from the transport bit stream and inputs the audio and OAMD to the decoder 709, which decodes audio and OAMD from native audio bitstream. The decoded audio and OAMD are then input to an OBA encoder 711, which encodes the audio and OAMD into an OBA bitstream.

[0078] Referring to FIG. 7C, the decoding system 700C includes an OBA decoder 712 and an audio object renderer 713. The OBA bit stream is input to the OBA decoder 712, which outputs audio and OAMD, which are input to the audio object renderer 713 . The audio object renderer 713 processes the audio using OAMD to tailor it to a particular speaker layout.

Passing precomputed OAMDs for rendering on the source device

[0079] FIG. 8A and 8B are block diagrams of exemplary systems for converting a 22.2-channel audio bit stream into audio objects and OAMD, where the churn information and OAMD are packaged in a native audio bit stream for rendering at a source device, according to an embodiment. In this application, the channel mixing information as well as the OAMD are generated in the encoder and packetized in the native audio bit stream (eg, AAC bit stream) to be transmitted through the transport layer. In this configuration, the format conversion that occurs is simplified into an audio mixing unit. The shuffled audio along with the OAMD is sent to the audio object renderer for rendering.

[0080] Referring to FIG. 8A, encoding system 800A includes an encoder 801 (eg, an AAC encoder) and a transport layer mux 807. The encoder 801 further includes a core encoder 803, a format converter 802, and a bitstream packetizer 805. The format converter 802 further includes an OAMD metadata generator 804, which may be a Dolby Atmos metadata generator, for example. Some examples of OAMD metadata include, but are not limited to, content description metadata, property update metadata, and customization data.

[0081] Native audio bit stream 806 (eg, AAC/MP4) includes 22.2-channel audio and metadata. The audio is input to the base encoder 803 of the encoder 801, which encodes the audio into a native audio format and outputs the encoded audio to the bitstream packetizer 805. The OAMD metadata generator 804 converts the 22.2-channel metadata to OAMD, for example, in accordance with the principles as described with respect to FIG. 1A and generates channel mixing information. The channel mixing information describes channel reordering of 22.2-channel content, for example, in accordance with the principles as described with respect to FIG. 1b. The channel mixing information is input to the bit stream packetizer 805 along with the OAMD. The output of the bitstream packetizer 805 is a native audio bitstream that includes channel mixing and OAMD information. The native audio bit stream is input to the transport layer multiplexer 807, which outputs a transport stream that includes the native audio bit stream.

[0082] Referring to FIG. 8B, the decoding system 800B includes a transport layer demultiplexer 808, a decoder 809, an audio mixing unit 810, and an audio object rendering unit 811. The transport layer demultiplexer 808 demultiplexes the audio and OAMD from the transport bit stream and inputs the audio and OAMD to a decoder 809 which decodes the audio and OAMD from the native audio bit stream. The decoded audio and OAMD are then input to the audio object renderer 811 . The audio object renderer 811 processes the audio using OAMD to adapt it to a particular speaker layout.

Transfer of precomputed OAMDs through the transport layer for transmission over HDMI

[0083] FIG. 9A-9C are block diagrams of exemplary systems for converting a 22.2-channel audio bit stream into audio objects and OAMD, where the channel mixing and OAMD information is embedded in the transport layer for delivery to source devices and then packaged in the native audio bit stream for transmission over HDMI, according to the embodiment.

[0084] The OAMDs used to represent 22.2-channel content are static to the program. For this reason, it is desirable to prevent the OAMD from being sent frequently so as to prevent the data rate from increasing in the audio bitstream. This can be achieved by sending static OAMD and channel mixing information in the transport layer and transmitted in the transport layer. On reception, OAMD and channel mixing information are used by the OBA encoder for subsequent transmission over HDMI. An exemplary transport layer is the Basic Media File Format (BMFF) described in ISO/IEC 14496-12-MPEG-4 Part 12, which defines a common structure for temporary media files such as video and audio. In an embodiment that uses MPEG DASH, the OAMDs are included in the manifest.

[0085] Referring to FIG. 9A, encoding system 900A includes an encoder 902 (eg, an AAC encoder), a format converter 905, and a transport layer multiplexer 903. The format converter 905 further includes an OAMD metadata generator 904 . Some examples of OAMD metadata include, but are not limited to, content description metadata, property update metadata, and customization data.

[0086] Native audio bit stream 901 (eg, AAC/MP4) includes 22.2-channel audio and metadata. The audio is input to an encoder 902 which encodes the audio into a native audio format and outputs the encoded audio to a transport layer multiplexer 903 . The OAMD metadata generator 904 converts the 22.2-channel metadata to OAMD, for example, in accordance with the principles as described with respect to FIG. 1A and generates channel mixing information. The channel mixing information describes the channel reordering of the 22.2-channel content, for example, in accordance with the principles as described with respect to FIG. 1b. The channel mixing information is input to the transport layer multiplexer 903 along with the OAMD. The output of the transport layer multiplexer 903 is a transport bit stream (eg, an MPEG-2 transport stream) or a batch file (eg, an ISO BMFF file) or a description of the media representation (eg, an MPEG-DASH manifest) that includes its own stream of audio bits.

[0087] Referring to FIG. 9B, decoding system 900B includes a transport layer demultiplexer 906, a decoder 907, an audio mixing module 908, and an OBA encoder 909. The transport layer demultiplexer 906 demultiplexes the audio, channel mixing information, and OAMD from the transport bitstream. The decoded audio is input to the audio bitstream at a decoder 907 (eg, an AAC decoder) that decodes the audio to reconstruct (ie, determine or extract) its own audio bitstream. The native audio bit stream is then input to the audio mixing module 908 along with the channel mixing information output by the transport layer demultiplexer 906 . Reordered audio is output from the audio channel mixing module 908 and input to the OBA encoder 909 along with the OAMD. The output of the OBA encoder is a stream of OBA bits.

[0088] Referring to FIG. 9C, the decoding system 900C includes an OBA decoder 910 and an audio object renderer 911. The OBA bit stream is input to the OBA decoder 910, which outputs audio and OAMD, which are input to the audio object renderer 911 . The audio object renderer 911 processes the audio using OAMD to adapt it to a particular speaker layout.

Passing precomputed OAMDs through the transport layer for rendering on the source device

[0089] FIG. 10A and 10B are block diagrams of exemplary systems for converting a 22.2-channel audio bitstream into audio objects and OAMD, with channel mixing information and OAMD embedded in the transport layer for rendering in source devices (e.g., STB, AVR), according to an embodiment. The OAMDs used to represent 22.2 channel content are static to the program. For this reason, it is desirable to prevent the OAMD from being sent frequently so as to prevent the data rate from increasing in the audio bit stream. This can be achieved by sending static OAMD and channel mixing information in the transport layer and transmitted in the transport layer. On reception, the OAMD and channel mixing information are used by the audio object renderer to render the content. An exemplary transport layer is the Basic Media File Format (BMFF) described in ISO/IEC 14496-12-MPEG-4 Part 12, which defines a common structure for temporary media files such as video and audio. In an embodiment, the OAMDs are included in the MPEG-DASH manifest.

[0090] Referring to FIG. 10A, encoding system 1000A includes an encoder 1001 (eg, an AAC encoder), a format converter 1002, and a transport layer multiplexer 1004. The format converter 1002 further includes an OAMD metadata generator 1003 . Some examples of OAMD metadata include, but are not limited to, content description metadata, property update metadata, and customization data.

[0091] Native audio bit stream 1005 (eg, AAC/MP4) includes 22.2-channel audio and metadata. The audio is input to an encoder 1001 which encodes the audio into a native audio format and outputs the encoded audio to a transport layer multiplexer 1004. The OAMD metadata generator 1003 converts the 22.2-channel metadata to OAMD, for example, in accordance with the principles as described with respect to FIG. 1A and generates channel mixing information. The channel mixing information describes the channel reordering of the 22.2-channel content, for example, in accordance with the principles as described with respect to FIG. 1b. The channel mixing information is input to the transport layer multiplexer 1004 along with the OAMD. The output of transport layer multiplexer 1004 is a transport stream that includes its own audio bit stream.

[0092] Referring to FIG. 10B, the decoding system 1000B includes a transport layer demultiplexer 1006, a decoder 1007, an audio mixing unit 1008, and an audio object rendering unit 1009. The transport layer demultiplexer 1006 demultiplexes the audio and OAMD from the transport bitstream and inputs the audio and OAMD to a decoder 1007 which decodes the audio and OAMD from the native audio bitstream. The decoded audio and OAMD are then input to the audio object renderer 1009 . The audio object renderer 1009 processes the audio using OAMD to tailor it to a particular speaker layout.

Sample Process

[0093] FIG. 11 is a flowchart for a CBA-OBA conversion process 1100. Process 1100 may be implemented using the audio system architecture shown in FIG. 3. Process 1100 includes receiving a bitstream including channel-specific audio and metadata (1101), parsing a signaling parameter from the bitstream indicating an OAMD representation (1102), converting the channel-oriented metadata to OAMD to based on the signaled OAMD representation (1103), generating channel mixing information based on the OAMD ordering constraints (1104), channel reordering of the channel based audio based on the channel mixing information (1105), and preparing by rendering the reordered channel based audio using OAMD (1106). Steps 1103 and 1104 above may be performed using, for example, the OAMD representations and sub-channel assignments/ordering shown in FIG. 1A and 1B, respectively, and the audio system architecture shown in FIG. 3. Some examples of OAMD metadata include, but are not limited to, content description metadata, property update metadata, and customization data.

[0094] FIG. 12 is a flowchart for a CBA-OBA conversion process 1200. Process 1200 may be implemented using the audio system architecture shown in FIG. 4. Process 1200 includes receiving a bitstream including channel-specific audio and metadata (1201), parsing a signaling parameter from the bitstream indicating an OAMD representation (1202), converting the channel-oriented metadata to OAMD to based on the signaled OAMD representation (1203), generation of channel mixing information based on OAMD ordering constraints (1204), channel reordering of the channel based audio based on the channel mixing information (1205), and encoding of the reordered channel specific audio and OAMD into a stream OBA bits (1206) for transmission to the playback device, wherein the audio is prepared by rendering by an audio object renderer using OAMD. Steps 1203 and 1205 above may be performed using, for example, the OAMD representations and sub-channel assignments/ordering shown in FIG. 1A and 1B, respectively, and the audio system architecture shown in FIG. 4. Some examples of OAMD metadata include, but are not limited to, content description metadata, property update metadata, and customization data.

[0095] FIG. 13 is a flowchart for a CBA-OBA conversion process 1300. Process 1300 may be implemented using the audio system architecture shown in FIG. 5. Process 1300 includes receiving a native audio bit stream including channel-specific audio and metadata in a native audio format (1301), decoding the native audio bitstream to recover the channel-specific audio and metadata (1302), parsing the parameter signaling from a bitstream indicating an OAMD representation (1303), transforming channel-specific metadata to OAMD based on the signaled OAMD representation (1304), generating channel mixing information based on OAMD ordering constraints (1305), reordering channels channel oriented audio based on the channel mixing information (1306), preparing by rendering the reordered channel oriented audio using OAMD (1307). Steps 1304 and 1305 may be performed using, for example, the OAMD representations and sub-channel assignments/sequencing shown in FIG. 1A and 1B, respectively, and the audio system architecture shown in FIG. 5.

[0096] FIG. 14 is a flowchart for a CBA-OBA conversion process 1400. Process 1400 may be implemented using the audio system architecture shown in FIG. 6A and 6B. Process 1400 begins by receiving a native audio bitstream including channel-specific audio and metadata in a native audio format (1401), decoding the native audio bitstream in order to recover, i.e. determine, or extract, the channel-specific audio and metadata (1402), parse the signaling parameter from the bitstream indicating the OAMD representation (1403), and convert the channel-oriented metadata to OAMD based on the signaled OMD representation ( 1404), generation of channel mixing information based on OAMD ordering constraints (1405), channel reordering of channel oriented audio based on channel mixing information (1406), encoding of the reordered channel oriented audio and OAMD into an OBA bit stream (1407) for transmission to a playback device, wherein the audio is prepared by rendering by an audio object renderer using OAMD. Steps 1404 and 1405 may be performed using, for example, the OAMD representations and sub-channel assignments/sequencing shown in FIG. 1A and 1B, respectively, and the audio system architecture shown in FIG. 6A and 6B.

[0097] FIG. 15 is a flowchart for a CBA-OBA conversion process 1500. Process 1500 may be implemented using the audio system architecture shown in FIG. 7A-7C. Process 1500 begins by receiving a channel-based audio bit stream including channel-based audio and metadata (1501), encoding the channel-based audio into a native audio bitstream (1502), parsing a signaling parameter from the channel-based metadata indicating OAMD representation (1503), mapping channel-oriented metadata to OAMD based on signaling OMD representation (1504), generating channel mixing information based on OAMD ordering constraints (1505), combining native audio bit stream, channel mixing information and OAMD into the combined audio bit stream (1506), including the combined audio bit stream into the transport layer bit stream (1507) for transmission to a rendering playback device or a rendering source device (eg, STB, AVR). Details of the above steps are described with respect to FIG. 1A, 1B, 7A, 7C, 8A, 8B, 9A-9C, 10A and 10B.

[0098] FIG. 16 is a flowchart for a CBA-OBA conversion process 1600. Process 1600 may be implemented using the audio system architecture shown in FIG. 8A, 8B, 9A-9C, 10A, 10B. Process 1600 begins by receiving a transport layer bitstream including the native audio bitstream and metadata (1601), extracting the native audio bitstream and metadata, channel mixing information, and OAMD from the transport bitstream (1602), decoding the native audio bitstream to restore, i.e. determine or extract, channel-based audio (1603), channel-based audio channel reordering using channel mixing information (1604), optionally encoding the re-ordered channel-based audio and OAMD into an OBA-bit stream (1605) to be passed to a playback device or the source device, or optionally decoding the OBA bitstream to recover the reordered channel-based audio and OAMD (1606), and preparing by rendering the reordered channel-based audio using OAM (1607) and passing to the playback device. Details of the above steps are described with respect to FIG. 8A, 8B, 9A-9C, 10A and 10B.

Transmission of precomputed OAMDs in MPEG-4 audio bit streams or MPEG-D audio bit streams

[0099] In an embodiment, OAMDs representing 22.2 content are carried in their own audio bit stream, eg, in an MPEG-4 audio bit stream (ISO/IEC 14496-3). An exemplary syntax for three embodiments is provided below.

MPEG-4 syntax alternative #1

Syntax Number of bits Mnemonics data_stream_element(){
element_instance_tag;
data_byte_align_flag;
reserved;
cnt=count;
if(cnt==255)
cnt+=esc_count;
if(data_byte_align_flag)
byte_alignment();
for (i=0; i<cnt; i++)
data_stream_byte[element_instance_tag][i];
} 4
1
3
8
8 uimsbf
uimsbf
uimsbf
uimsbf
uismsbf

MPEG-4 syntax alternative #2

Syntax Number of bits Mnemonics fill_element(){
cnt=count;
if(cnt==15)
cnt+=esc_count-1;
while(cnt>0){
cnt-=extension_payload(cnt);
}
} 4
8 uimsbf
uimsbf

MPEG-4 syntax alternative #3

Syntax Number of bits Mnemonics extension_payload(cnt)
{
extension_type;
align=4;
switch(extension_type){
[...] Case EXT_OAMD_INFO:
return oamdInfo;
[...] }
} 4 usimsbf

[00100] In the above exemplary syntax, the element_instance_tag element is a number to identify the data flow element, and the extension_payload(int) element may be contained in the fill element (ID_FIL). Each of the above three syntax embodiments describes a "tag" or "extension_type" to indicate the meaning of the additional data. In an embodiment, a signal may be inserted into a bitstream signaling that additional OAMD and channel mixing information is present in one of the three bitstream extension zones to avoid the need for a decoder to check these bitstream zones. For example, the MPEG4_ancillary_data field contains a dolby_surround_mode field with the following semantics. A similar signaling transmission syntax can be used to indicate to the decoder that OAMDs are present in the bitstream.

Defining the dolby_surround_mode signal

dolby_surround_mode Description "00" Dolby surround mode not specified "01" 2ch audio part is not encoded in Dolby surround mode "10" 2ch the audio part is encoded in Dolby surround mode "eleven" reserved

[00101] In an embodiment, a reserved field in the above table is used to indicate that precomputed OAMD payload data is embedded somewhere in the extended bitstream data. The reserved value (dolby_surround_mode="11") is used to indicate to the decoder that the extended data fields contain the required OAMD and channel information required to convert 22.2 to OBA (eg, Dolby® Atmos®). Alternatively, the reserved field indicates that the content is OBA compatible (eg, Dolby® Atmos® compatible) and conversion of 22.2 channel content to OBA is possible. Thus, if the dolby_surround_mode signal is set to the reserved value "11", then the decoder must know that the content is OBA-compatible and convert the 22.2-channel content to OBA for additional encoding and/or rendering.

[00102] In an embodiment, OAMDs representing 22.2 content are carried in their own audio bit stream, eg, in an MPEG-D USAC audio bit stream (ISO/IEC 23003-3). An exemplary syntax for such an embodiment is provided below.

Syntax Number of bits Mnemonics UsacExtElementConfig { usacExtElementType=escapedValue(4,8,16); usacExtElementConfigLength=escapedValue(4,8,16); usacExtElementDefaultLengthPresent; 1 uimsbf if(usacExtElementDefaultLengthPresent){ usacExtElementDefaultLength=escapedValue(8,16,0)+1; } else { usacExtElementDefaultLength=0; } usacExtElementPayloadFrag; 1 uimsbf switch(usacExtElementType) { case ID_EXT_ELE_FILL: break; case ID_EXT_ELE_MPEGS: SpatialSpecificConfig(); break; case ID_EXT_ELE_SAOC: SaocSpecificConfig(); break; case ID_EXT_ELE_AUDIOPREROLL: /* No config element */ break; case ID_EXT_ELE_UNI_DRC: uniDrcConfig(); break; case ID_EXT_ELE_OAMD: oamdInfo(); break; default: Note while(usacExtElementConfigLength} { tmp; 8 uimsbf } break; } } Note: The default entry for usacExtElementType is used for the unknown extElementTypes so that legacy decoders can accommodate future extensions.

Exemplary audio system architecture

[00103] FIG. 17 is a block diagram of an exemplary audio system architecture that includes mapping audio channels to audio objects, according to an embodiment. In this example, the architecture is for STB or AVR. The STB/AVR 1700 includes an input 1701, an analog-to-digital converter (ADC) 1702, a demodulator 1703, a synchronizer/decoder 1704, an MPEG demultiplexer 1707, an MPEG decoder 1706, a memory 1709, a control processor 1710, an audio mixing module 1705, OBA encoder 1711 and video encoder 1712. In this example, STB/AVR 1700 implements the applications described in FIG. 9A-9C and 10A, 10B, the precomputed OAMDs are carried in an MPEG4 audio bit stream.

[00104] In an embodiment, the low noise unit collects radio waves from a satellite dish and converts them into an analog signal, which is sent via a coaxial cable to the input port 1701 of the STB/AVR 1700. The analog signal is converted to a digital signal by the ADC 1702. The digital signal is demodulated by a demodulator 1703 (eg, QPSK demodulator) and synchronized and decoded by synchronizer/decoder 1704 (eg, synchronizer plus Viterbi decoder) in order to recover the MPEG transport bit stream, which is demodulated by MPEG demultiplexer 1707 and decoded by MPEG decoder 1706 in order to recover channel-oriented audio and video audio bit streams and metadata including channel mixing information and OAMD. The audio channel mixing module 1705 reorders the audio channels according to the channel mixing information, for example, in accordance with the principles as described with respect to FIG. 1b. An OBA encoder 1711 encodes audio using reordered channels into an OBA audio bit stream (eg, Dolby® MAT) that is transmitted to a playback device (eg, Dolby® Atmos® device) for preparation by rendering by an audio renderer of objects in the playback device . A video encoder 1712 encodes the video into a video format that is supported by the playback device.

[00105] It should be noted that the architecture described with respect to FIG. 17 is only an exemplary architecture. The conversion from CBA to OBA may be performed by any device that includes one or more processors, a storage device, appropriate I/O interfaces, and software modules and/or hardware (e.g., an ASIC) to perform the format conversion and channel reordering described in this document.

[00106] Although this document contains many specific implementation details, they should not be construed as limitations on the scope of what may be claimed as claims, but instead as descriptions of features that may be specific to particular embodiments. Certain features that are explained in this detailed description in the context of separate embodiments may also be implemented in combination in one embodiment. Conversely, various features that are described in the context of one embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. In addition, although the features may be described above as working in certain combinations and even initially set as such in the claims, one or more features from the claimed combination may in some cases be excluded from the combination, and the claimed combination may be directed to a subcombination or variation of the subcombination. . The logical sequences of operations illustrated in the drawings do not require the specific order shown or sequential order in order to achieve the desired results. In addition, other steps may be provided or steps may be omitted from the described sequences of operations, and other components may be added to or removed from the systems described. Accordingly, other implementations are within the scope of the appended claims.

[00107] Various aspects of the present invention may be taken into account from the following enumerated exemplary embodiments (EEE):

EEE 1. A method comprising:

- receiving, by means of one or more processors of the audio processing equipment, a bit stream including channel-oriented audio and metadata;

- wherein one or more processors are configured to:

- parse a signaling parameter from the metadata, wherein the signaling parameter specifies one of a plurality of different object audio metadata (OAMD) representations;

- convert the channel-specific metadata to OAMD using the OAMD representation, which is indicated by the signaling parameter;

- generate channel intermixing information based on OAMD channel ordering constraints;

- reorder the channels of the channel-based audio based on the channel mixing information; And

- prepare by rendering reordered channel-oriented audio into prepared by rendering audio using OAMD; or

- encode the channel-oriented audio and OAMD into an object-oriented audio bit stream, and pass the object-oriented audio bit stream to a playback device or source device.

EEE 2: The EEE 1 method, wherein the channel-based audio and metadata are included in the native audio bitstream, and the method further comprises decoding the native audio bitstream in order to recover the channel-based audio and metadata.

EEE 3: An EEE 2 method wherein the native audio bitstream is an Advanced Audio Coding (AAC) bitstream.

EEE 4. The method as in any one of EEE 1-3, wherein the channel oriented audio and metadata is N.M channel oriented audio and metadata, where N is a positive integer greater than nine and M is a positive integer greater than or equal to zero.

EEE 5. A method according to any one of EEE 1-4, wherein the source device is a set-top box or an audio/video receiver.

EEE 6. A method according to any one of EEE 1-5, further comprising:

- defining a first set of channel-oriented audio channels that are capable of being represented by overlaid OAMD channels;

- assignment of labels of embedded OAMD channels to the first set of channels;

- defining a second set of channel-oriented audio channels that do not allow representation via overlaid OAMD channels; And

- assignment of static positional OAMD coordinates to the second set of channels.

EEE 7. A method as in any one of EEE 1-6, wherein the OAMD includes sizing data to lower the volume levels of one or more off-screen audio objects in the rendered audio.

EEE 8. A method as in any one of EEE 1-7, wherein the OAMDs include object gains used to compensate for differences between the channel oriented audio downmix values and the rendering of the OAMD representations of the channel oriented audio.

EEE 9. A method comprising:

- receiving, by means of one or more processors of the audio processing equipment, a bit stream including channel-oriented audio and metadata;

- wherein one or more processors are configured to:

- encode channel-oriented audio into a native audio bitstream;

- parse a signaling parameter from the metadata, wherein the signaling parameter specifies one of a plurality of different object audio metadata (OAMD) representations;

- convert the channel-specific metadata to OAMD using the OAMD representation, which is indicated by the signaling parameter;

- generate channel intermixing information based on OAMD channel ordering constraints;

- generate a packet of bit streams, which includes its own audio bit stream, channel mixing information and OAMD;

- multiplex the packet into a transport layer bitstream; And

- transmit the transport layer bit stream to the playback device or source device.

EEE 10: The EEE 9 method wherein the native audio bit stream is an Advanced Audio Coding (AAC) bit stream.

EEE 11. A method according to EEE 9 or EEE 10, wherein the channel-based audio and metadata is N.M channel-based audio and metadata, where N is a positive integer greater than seven and M is a positive integer greater than or equal to zero.

EEE 12. The method of any one of EEE 9-11, wherein the source device is a set-top box or an audio/video receiver.

EEE 13. A method as in any one of EEE 9-12, wherein channels in channel-based audio that can be represented by OAMD sub-channel labels use OAMD sub-channel labels, and channels in channel-based audio that cannot be represented through the positions of static objects, the labels of the embedded OAMD channels are used, and each position of the static object is described in positional OAMD coordinates.

EEE 14. A method as in any one of EEE 9-13, wherein the OAMD includes sizing data to lower the volume levels of one or more off-screen audio objects in the rendered audio.

EEE 15. A method as in any one of EEE 9-14, wherein the OAMDs include object gains used to compensate for differences between the channel oriented audio downmix values and the rendering of the OAMD representations of the channel oriented audio.

EEE 16. The method of any one of EEE 9-15, wherein the transport bit stream is a Motion Picture Expert Group (MPEG) audio bit stream that includes a signal that indicates the presence of an OAMD in an extension field of the MPEG audio bit stream.

EEE 17. The EEE 16 method, wherein a signal that indicates the presence of an OAMD in an MPEG audio bit stream is included in a reserved metadata field in the MPEG audio bit stream for transmission in surround sound mode signaling.

EEE 18. A method comprising:

- receiving, by means of one or more processors of the audio processing equipment, a transport layer bit stream including the packet;

- wherein one or more processors are configured to:

- demultiplex the transport layer bit stream in order to recover the packet;

- decode the packet in order to recover its own audio bitstream, channel mixing information and object audio metadata (OAMD);

- decode the native audio bit stream in order to recover the channel-specific audio bit stream and metadata;

- reorder the channels of the channel-based audio based on the channel mixing information; And

- prepare by rendering reordered channel-oriented audio into prepared by rendering audio using OAMD; or

- encode the channel-oriented audio and OAMD into an object-oriented audio bit stream and transmit the object-oriented audio bit stream to the source device.

EEE 19: The EEE 18 method, wherein the native audio bit stream is an Advanced Audio Coding (AAC) bit stream.

EEE 20. A method according to EEE 18 or EEE 19, wherein the channel-based audio and metadata is N.M channel-based audio and metadata, where N is a positive integer greater than seven and M is a positive integer greater than or equal to zero.

EEE 21. A method according to any one of EEE 18-20, further comprising:

- defining a first set of channel-oriented audio channels that are capable of being represented by overlaid OAMD channels;

- assignment of labels of embedded OAMD channels to the first set of channels;

- defining a second set of channel-oriented audio channels that do not allow representation via overlaid OAMD channels; And

- assignment of static positional OAMD coordinates to the second set of channels.

EEE 22. A method as in any one of EEEs 18-21, wherein the OAMD includes size adjustment data to lower the volume levels of one or more off-screen objects in the rendered audio.

EEE 23. A method as in any one of EEEs 18-22, wherein the OAMDs include object gains used to compensate for differences between the channel oriented audio downmix values and the rendering of the OAMD representations of the channel oriented audio.

EEE 24. The method of any one of EEEs 18-23, wherein the transport bit stream is a Motion Picture Expert Group (MPEG) audio bit stream that includes a signal that indicates the presence of an OAMD in an extension field of the MPEG audio bit stream.

EEE 25. A method as in any of EEEs 18-24, wherein a signal that indicates the presence of OAMD in the MPEG audio bit stream is included in a reserved data structure field in MPEG audio bit stream metadata for transmission in surround sound mode signaling.

EEE 26 Equipment comprising:

- one or more processors; And

- a non-volatile computer-readable storage medium having stored instructions that, when executed by one or more processors, instruct one or more processors to perform the methods of any of the preceding EEE 1-25.

EEE 27. A non-volatile computer-readable storage medium having stored instructions that, when executed by one or more processors, instructs one or more processors to perform the methods of any of the preceding EEE 1-25.

Claims (41)

1. An audio processing method (1100; 1200), comprising the step of: - receive (1101; 1201), by one or more processors of the audio processing equipment, a bit stream including channel-based audio and associated channel-based audio metadata; - wherein one or more processors are configured to: - parse (1102; 1202) a signaling parameter from channel-oriented audio metadata, wherein the signaling parameter specifies one of a plurality of different object audio metadata (OAMD) representations, each of the OAMD representations converting one or more channel-oriented audio audio channels to one or more audio objects; - convert (1103; 1203) channel-specific metadata to OAMD associated with one or more audio objects using an OAMD representation that is indicated by a signaling parameter; - generate (1104; 1204) channel mixing information based on OAMD channel ordering constraints; - reorder (1105; 1205) one or more audio channels of channel-based audio based on the channel mixing information in order to generate reordered channel-based audio; And - prepare (1106) by rendering reordered channel-oriented audio into prepared by rendering audio using OAMD; or - encode (1206) the reordered channel-oriented audio and OAMD into an object-oriented audio bit stream and transmit the object-oriented audio bit stream to a playback device or source device (600B). 2. The method of claim 1, wherein the bitstream is a native audio bitstream, and the method further comprises decoding the native audio bitstream to determine channel-oriented audio and metadata. 3. The method of claim 2, wherein the native audio bit stream is an advanced audio coding (AAC) bit stream. 4. A method according to any one of the preceding claims, wherein the channel-based audio and associated channel-based audio metadata are N.M channel-based audio and channel-based audio metadata associated with the N.M channel-based audio, respectively, and where N is a positive integer greater than nine, and M is a positive integer greater than or equal to zero. 5. The method of claim 4, wherein the channel-based audio is 22.2. 6. An audio processing method (1500), comprising: receive (1501), by one or more processors of the audio processing equipment, a bit stream including channel-based audio and associated channel-based audio metadata; - wherein one or more processors are configured to: - encode (1502) channel-oriented audio in its own stream of audio bits; - parse (1503) a signaling parameter from the channel-based audio metadata, wherein the signaling parameter specifies one of a plurality of different object audio metadata (OAMD) representations, each of the OAMD representations converting one or more channel-based audio audio channels into one or more audio objects; - convert (1504) channel-oriented metadata to OAMD associated with one or more audio objects, using the OAMD representation, which is indicated by a signaling parameter; - generate (1505) channel mixing information based on OAMD channel ordering constraints; - generate (1506) a packet of bit streams that includes its own audio bit stream, channel mixing information and OAMD, where the channel mixing information provides reordering of one or more channel-based audio audio channels based on the channel mixing information in the playback device or source device ( 700B; 800B) to generate reordered channel-based audio; - multiplex (1507) a packet of bitstreams into a transport layer bitstream; And - transmit the transport layer bit stream to the playback device or source device (700B; 800B). 7. The method of claim 6, wherein the native audio bit stream is an advanced audio coding (AAC) bit stream. 8. The method according to any one of paragraphs. 6 or 7, wherein the channel specific audio and associated channel specific audio metadata are N.M channel specific audio and channel specific audio metadata associated with N.M channel specific audio, respectively, and where N is a positive integer, greater than seven, and M is a positive integer greater than or equal to zero. 9. The method of claim 8, wherein the channel-based audio is 22.2. 10. An audio processing method (1600), comprising: receiving (1601), by one or more processors of the audio processing equipment, a transport layer bitstream including a bitstream packet, wherein the bitstream packet contains its own audio bitstream containing encoded channel-specific audio, channel mixing information, and object audio metadata (OAMD ); - wherein one or more processors are configured to: - demultiplex the bitstream of the transport layer in order to determine the packet of bitstreams; - decode (1606) the packet of bit streams in order to determine channel-oriented audio, channel mixing information and object audio metadata (OAMD); - reorder (1604) the audio channels of the channel-based audio based on the channel mixing information in order to generate a re-ordered channel-based audio; And - prepare (1607) by rendering reordered channel-oriented audio into prepared by rendering audio using OAMD; or - encode (1605) the reordered channel-oriented audio and OAMD into an object-oriented audio bit stream and transmit the object-oriented audio bit stream to the source device. 11. The method of claim 10, wherein the native audio bit stream is an advanced audio coding (AAC) bit stream. 12. The method of claim 10 or 11, wherein the channel-based audio is N.M channel-based audio, wherein N is a positive integer greater than seven and M is a positive integer greater than or equal to zero. 13. The method of claim 12, wherein the channel-based audio is 22.2. 14. Equipment for audio processing, comprising: - one or more processors; And a non-volatile computer-readable storage medium having instructions stored thereon which, when executed by one or more processors, instructs one or more processors to perform the methods of any one of the preceding claims. 15. A non-volatile computer-readable storage medium having instructions stored thereon which, when executed by one or more processors, instructs one or more processors to carry out the methods of any one of claims. 1-13.

Images