BR122020017000B1 - METHOD OF GENERATING AN AUDIO SIGNAL FOR AN AUDIOVISUAL CONTENT ITEM AND APPARATUS FOR GENERATING AN AUDIO SIGNAL FOR AN AUDIOVISUAL CONTENT ITEM - Google Patents

Abstract

is a transmission apparatus that generates an audiovisual content item data stream (e.g., transport stream) comprising a plurality of individual audiovisual data streams with audiovisual components for the content item. a generator (301-307) generates a first stream comprising both mandatory audio data and substitutable audio data for the audio representation, with substitutable audio data being data that can be replaced with alternative audio data. a combiner (309) includes the resulting stream in the content item data stream. a receiving apparatus includes an extractor (403) that extracts the required audio data from the received stream. a substitutor (415) can replace the substitutable audio data with alternative audio data, and an output (415) can generate an audio signal from the mandatory and alternative signal data. The approach can specifically provide an improved and more flexible data flow for audiovisual content.

Description

[001] This patent application consists of a split application from the invention patent application BR 112016021214-2, dated 03/16/2015.

FIELD OF INVENTION

[002] The invention relates to audiovisual content item data streams and, in particular, but not exclusively, audiovisual content item data streams such as the MPEG Transport Stream -two.

BACKGROUND OF THE INVENTION

[003] Digital coding of various source signals has become increasingly important in recent decades, as digital signal communication and representation have increasingly replaced analog communication and representation. For example, mobile phone systems, such as the Global System for mobile communication, are based on digital voice coding. Additionally, the distribution of media content such as video and music is increasingly based on digital content encoding.

[004] Generally, an audiovisual content item comprises a series of different audiovisual components and data types. For example, a content item corresponding to a film or television program may include at least one video signal component, generally a plurality of different audio components, control data, synchronization data, metadata, for example, characterizing the content etc. For example, a film may include a main video component, a plurality of audio tracks (e.g. for different languages), subtitle data, main actors, etc. Therefore, it is often necessary to include a relatively large number of different data types in a single combined data stream for the audiovisual content item.

[005] In order to accommodate a representation of an audiovisual content item that includes a range of data of different types, an audiovisual content item data stream of (sub)streams may frequently be generated ( streams) of audiovisual data that provide audiovisual components to the audiovisual content item. Additionally, data streams comprising data, metadata, etc. may be included.

[006] The audiovisual content item data stream may comprise all data related to the rendering of the content item. The data stream of the audiovisual content item is often referred to as a transport stream, or possibly as a system stream, program stream, or container stream. The individual audiovisual data stream is often called an elementary data stream.

[007] To provide an efficient representation of the audiovisual content item, it is important that an effective data structure be defined for the audiovisual content item data stream. The use of a data structure comprising a series of separate audiovisual data streams that each represent audiovisual components provides a flexible yet efficient approach. The approach enables, for example, flexible inclusion of different audio tracks for a given video component, for example, audio signals corresponding to different languages can be provided in different audiovisual data streams.

[008] An example of a data structure is provided in ETSI DVB (Digital Video Broadcasting) Standard EN 300 468 V I.14.1, where section 6.4.10 specifically refers to the MPEG transport stream used for DVB. ENGDEGÂRD JONAS ET AL: “MPEG Spatial Audio Object Coding The ISO/MPEG Standard for Efficient Coding of Interactive Audio Scenes”, AES CONVENTION 129; NOVEMBER 2010, AES, 60 EAST 42ND STREET, ROOM 2520 NEW YORK 10165-2520, USA, November 4, 2010 (2010-11-04), XP040567234, provides an introduction and overview of the ISO-MPEG SAOC specification. Steffens et al: “Interactivity in MPEG-H 3D Audio Content - Proposal for Extension of OAM Format and Test Procedure” January 16, 2013, XP030056683 proposes that reveals the use of an interactivity flag in an encoder/decoder-based approach object to indicate that a specific audio object should be made available separately from all other audio objects on the receiver side.

[009] A number of different structures for audiovisual content item data streams have been standardized. One of the most widely distributed and frequently used structures for audiovisual content item data streams is the MPEG-2 Transport Stream, which is used, for example, for broadcast digital television or Blu-rays.

[010] The MPEG-2 Transport Stream is an example of a data structure in which the data stream is made up of a plurality of time-multiplexed sequential data packets. Each data packet can provide data for a specific component of the audiovisual content item.

[011] However, a problem with the conventional approach to audiovisual content item data streams lies in that the data structure is sub-optimal for some purposes, and, in particular, tends not to provide optimal flexibility.

[012] For example, audiovisual content item data streams such as MPEG-2 Transport streams support alternative audio representations for a given scene by enabling different audio representations to be provided in different elementary streams. Then, a receiver can select between these alternative elementary streams to provide a desired audio track. For example, an MPEG-2 Transport Stream may comprise an elementary stream comprising a video component together with two elementary streams that each provide an audio representation that can be rendered along with the video component. For example, one elementary audio stream may comprise the audio for the video component in the original language, while another elementary audio stream may comprise the audio for the video component, but with the voice dubbed in a different language. Then, a decoder or renderer can select between alternative audio tracks for the video by selecting the appropriate elementary stream.

[013] However, such an approach does not provide optimal audio flexibility and also results in a relatively high data rate due to the parallel audio representations provided by alternative elementary streams.

[014] Consequently, it would be desirable to provide an improved approach to audiovisual content item data streams and, in particular, an approach that provides additional flexibility and/or a reduced data rate.

[015] However, a critical challenge is how such improvement is possible while maintaining a high degree of common attributes with existing approaches. For example, it is desirable to be able to further improve the MPEG-2 Transport stream, but such enhancement should ideally maintain as much backward compatibility as possible. Considerations needed to improve approaches to audiovisual content item data streams may furthermore not be limited to just what additional data should be provided, how, or in what format it should be provided. Instead, there are additional challenges in determining how such data should be included in an audiovisual content item data stream to not only achieve an efficient audiovisual content item data stream, but also enable efficient operation and, ideally, , optimized backward compatibility.

[016] Thus, an improved approach to audiovisual content item data streams would be advantageous.

SUMMARY OF THE INVENTION

[017] Consequently, the Invention preferably seeks to mitigate, alleviate or eliminate one or more of the disadvantages mentioned above, individually or in any combination.

[018] According to another aspect of the invention, there is provided a method of generating an audio signal for an audiovisual content item, the method comprising: receiving an audiovisual content item data stream comprising a plurality of individual audiovisual data streams, wherein the plurality of individual audiovisual data streams comprises audiovisual components for an audiovisual content item; wherein each audiovisual data stream comprises encoded data representing at least one audiovisual component and synchronization data; wherein the audiovisual content item data stream comprises a first individual audiovisual data stream comprising first audio data for an audio representation for the audiovisual content item, the first audio data comprising audio data required for the audio representation and substitutable audio data for the audio representation, the mandatory audio data corresponding to an audio component that must be rendered when the audio representation is rendered, and the substitutable audio data corresponding to a substitutable audio component that may be replaced by an alternative audio component when the audio representation is rendered, the audiovisual content item data stream further comprising a content item map table for the audiovisual content item, the audiovisual content item map table content comprising a list of audiovisual data stream packet identifiers for data packets for audiovisual data streams linked to the audiovisual content item, and the content item map table comprising a common data stream packet identifier for the audiovisual content item for a set of at least two of the plurality of individual audiovisual data streams; the method further comprising: extracting the required audio data from the first audio data in response to the common data stream packet identifier; recover alternative audio data; replace the substitutable audio data with alternative audio data; generating the audio signal from the mandatory audio data and the alternative audio data; wherein the method further comprises: generating a single individual audiovisual data stream from the plurality of single individual audiovisual data streams; generating an audio data processing descriptor for second audio data from at least one of the audiovisual data streams, inserting the audio data processing descriptor into the single audiovisual data stream, and feeding the single audiovisual data stream to a decoder audio; and wherein generating the audio signal comprises the audio decoder generating the audio signal from audio data of the single individual audiovisual data stream, extracting the audio data processing descriptor from the second audiovisual data stream; and generating the audio signal in response to the audio data processing descriptor.

[019] The invention may enable enhanced audio generation for a content item. More efficient and/or flexible processing, communication or functionality can generally be achieved.

[020] More efficient audio selection and generation can be achieved in many scenarios.

[021] The approach can often provide improved backwards compatibility. It should be considered that the comments provided previously regarding the generation of the audiovisual content item data stream will apply equally, mutatis mutandis, to the processing of the audiovisual content item data stream.

[022] Replacing the substitutable audio data with alternative audio data may occur through a selection of the alternative audio data and a de-selection (or non-selection) of the substitutable audio data for processing. In some embodiments, the selection/de-selection may be a selection/de-selection of audio components that correspond to the replaceable audio data and alternative audio data, i.e., by a selection/de-selection of an audio component. replaceable and an alternative audio component generated by decoding the replaceable audio data and alternative audio data, respectively. In other embodiments, the replacement may be, for example, by overlaying the replaceable audio data with alternative audio data, for example, in a data stream/bitstream or in memory.

[023] In some embodiments, the method may comprise recovering only one of the replaceable audio data and the alternative audio data, and using only the recovered audio data for generating the audio signal.

[024] In some embodiments, recovery comprises recovering alternative audio data from the first audiovisual data stream. In some embodiments, the recovery comprises recovering the alternative audio data from a different audiovisual data stream among the plurality of individual audiovisual data streams of the first audiovisual data stream.

[025] In some embodiments, retrieval comprises retrieving alternative audio data from a data stream other than the data stream of the audiovisual content item.

[026] A data stream may, for example, correspond to a separate file or a segment of a file.

[027] According to one aspect of the invention, there is provided an apparatus for generating an audio signal for an audiovisual content item, the apparatus comprising: a receiver for receiving an audiovisual content item data stream comprising a plurality of individual audiovisual data streams, wherein the plurality of individual audiovisual data streams comprises audiovisual components for an audiovisual content item; wherein each audiovisual data stream comprises encoded data representing at least one audiovisual component and synchronization data; wherein the audiovisual content item data stream comprises a first audiovisual data stream comprising first audio data for an audio representation for the audiovisual content item, the first audio data comprising audio data required for the representation audio data and substitutable audio data for the audio representation, the mandatory audio data corresponding to an audio component that must be rendered when the audio representation is rendered, and the substitutable audio data corresponding to a substitutable audio component that can be replaced by an alternative audio component when the audio representation is rendered, the audiovisual content item data stream further comprising a content item map table for the audiovisual content item, the content item map table comprising a list of audiovisual data stream packet identifiers for data packets for audiovisual data streams linked to the audiovisual content item, and the content item map table comprising a common data stream packet identifier for the item of audiovisual content for a set of at least two of the plurality of individual audiovisual data streams; the apparatus further comprises: an extractor for extracting the required audio data from the first audio data in response to the common data stream packet identifier; a recoverer recovering alternative audio data; a replacer for replacing the replaceable audio data with the alternative audio data; an output for generating the audio signal of the mandatory audio data and the alternative audio data; an audio decoder for generating the audio signal from audio data of a single individual audiovisual data stream, the audio decoder comprising the extractor, the recoverer, the replacer and the output; and wherein the receiver is arranged to generate the single individual audiovisual data stream from the plurality of individual audiovisual data streams and to generate an audio data processing descriptor for second audio data from at least one of the data streams audiovisual data streams and to insert the audio data processing descriptor into the single audiovisual data stream and to feed the single audiovisual data stream to the audio decoder; and the audio decoder further comprises an audio data processing descriptor extractor for extracting the audio data processing descriptor from the second audiovisual data stream; and the audio decoder is arranged to generate the audio signal in response to the audio data processing descriptor.

[028] A method for generating an audiovisual content item data stream comprises a plurality of individual audiovisual data streams, the plurality of individual audiovisual data streams comprising audiovisual components for an audiovisual content item; each audiovisual data stream comprising encoded data representing at least one audiovisual component and synchronization data; comprising: generating a first audiovisual data stream comprising first audio data for an audio representation for the audiovisual content item, wherein the first audio data comprises mandatory audio data for the audio representation and substitutable audio data for the audio representation, wherein substitutable audio data is data substitutable with alternative audio data; and including the first audiovisual data stream in the audiovisual content item data stream.

[029] The approach can provide an enhanced audiovisual content item data stream. In particular, it is possible to achieve a transport flow with greater flexibility and/or reduced data rate in many scenarios.

[030] A more efficient provision of alternative audio can be achieved in many scenarios. Generally, the approach can be implemented as an enhancement to or further development of existing transport flows, while requiring a relatively low level of modification. Greater backward compatibility can often be achieved.

[031] As a specific example, the approach can enable efficient and practical enhancement of transport streams such as the MPEG-2 Transport stream. The transport stream can maintain the general design approach and philosophy of traditional MPEG-2 Transport Streams while allowing it to be improved to provide a more flexible and efficient representation of alternative audio representations.

[032] The replaceable audio data can be replaced/replaced/swapped/changed with the alternative audio data. In this way, the transport stream provides alternative audio representations for the content item with one alternative, for example, being a default representation represented by the required audio data and the replaceable audio data, and the other being an alternative representation represented by the mandatory audio data and alternative audio data.

[033] In many scenarios, the method may further comprise including the alternative audio data in the transport stream, often in the first audiovisual data stream or in another data stream of the audiovisual content item of the plurality of audiovisual data streams. . In this way, the alternative audio data can be comprised in an audiovisual data stream of the plurality of individual audiovisual data streams.

[034] Mandatory audio data may define a mandatory audio component for the audio representation. Replaceable audio data may define a replaceable audio component for the audio representation. Alternate audio data can define an alternate audio component for the audio representation.

[035] In this way, the first audio data may comprise mandatory audio data for a mandatory audio component of the representation and replaceable audio data for a replaceable audio component for the representation, the replaceable audio component being a data which can be replaced with alternative audio data for an alternative audio component in the replaceable audio component.

[036] The replaceable audio data may correspond to a standard audio track for the content item.

[037] The transport stream may include an assignment of one or more audio representations to the content item. An audio representation can correspond to a set of audio components that is sufficient to provide an audio output for the content item. Therefore, the transport stream may assign at least one audio sound track/representation to the content item. In many embodiments, at least one designated (or defined) audio representation in the transport stream comprises a plurality of audio components, wherein at least one audio component is a required audio component and at least one audio component is a replaceable audio component. The mandatory audio component is represented by mandatory audio data, and the replaceable audio component is represented by replaceable audio data. The required audio component must be included in the audio output for the content item when representation is selected. However, when a receiver selects this representation, the replaceable audio component may be included or may be replaced with an alternative audio component. The alternate audio component is represented by alternate audio data, often (but not always/necessarily) comprised in the transport stream itself.

[038] In the approach, audio data for a defined/specified/described/designated audio representation in the transport stream may consequently comprise mandatory audio data for at least one mandatory audio component and substitutable audio data for the one less replaceable audio component. When the designated audio representation in the transport stream is selected for, for example, rendering, the mandatory audio component needs to be included, while the replaceable audio component needs to be added or replaced with an alternative audio component.

[039] An audio representation for a content item may correspond to a set of audio components that is sufficient to provide an audio presentation/track for the content item. Specifically, an audio representation for a content item may correspond to a set of audio components that is indicated in the transport stream as sufficient to provide a sound track for the content item. A set of audio components that is not defined in the transport stream as sufficient to provide an audio representation for the content item, unless combined with at least one other audio component, does not form an audio representation.

[040] In this way, the transport stream may include data that defines one or more audio representations for the content item. Mandatory audio data may be mandatory for a first audio representation of the defined audio representations. Replaceable audio data provides audio data required for the first audio representation unless replaced by the alternate audio data. The transport stream can possibly assign more than one audio representation to the content item. In this case, the mandatory audio data need not be mandatory for a second audio representation.

[041] A data stream may, for example, correspond to a separate file or a segment of a file.

[042] The method may further comprise the step of including the alternative audio data in the first audiovisual data stream.

[043] This can provide an efficient and flexible approach and data flow of the audiovisual content item. In particular, in many scenarios this can provide a very efficient data structure for alternative audio. The approach may provide, for example, improved grouping of alternative audio that, in some embodiments, may facilitate operation.

[044] The step of including the alternative audio data in the first audiovisual data stream may comprise including the alternative audio data in an audiovisual data stream other than the plurality of individual audiovisual data streams of the first audiovisual data stream.

[045] This can provide an efficient approach, and an efficient and flexible audiovisual content item data flow. In particular, in many scenarios this can provide a very efficient data structure for alternative audio. The approach can facilitate, for example, the generation of the transport flow and/or facilitate the extraction of desired data from there.

[046] In some scenarios, the different stream of audiovisual data may provide an alternative audio representation of the audio scene. For example, the different stream of audiovisual data may be designated in the transport stream as an elementary stream that provides an audio representation for a content item. Specifically, it may comprise data for a set of audio components sufficient to generate an audio output for the content item without including any audio components for other streams.

[047] The data stream of the audiovisual content item may be a transport stream and the first audiovisual data stream may be an elementary stream.

[048] The approach can provide an improved transport flow that is based on a series of individual elementary flows. The approach can provide a flexible yet efficient structure for representing a content item. Furthermore, it can enable improved backwards compatibility with conventional systems.

[049] The substitutable audio data may represent a speech audio component in a first language and the alternative audio data represents a speech audio component in a second language.

[050] The approach can provide a particularly efficient approach to providing a plurality of different/alternative languages for a given content item.

[051] The mandatory audio data may represent a background audio component and the replaceable audio data represents a voice audio component.

[052] The approach may provide a particularly efficient approach to providing differentiated sound for background audio and voice audio, and in particular, may facilitate support of alternative voice components without increasing the data rate to an unacceptable extent.

[053] Replaceable audio data and alternative audio data may represent the same as audio content encoded using different encoding characteristics.

[054] Different coding characteristics can be, for example, different data rates, different coding algorithms, etc. Specifically, in some embodiments, the substitutable audio data and the alternative audio data may represent the same content or audio component encoded with different encoding characteristics, such as different bit rates.

[055] The approach can enable greater flexibility and can specifically enable flexible adaptation of characteristics of a transport flow. Furthermore, the approach can enable the transport flow to be modified relatively easily by, for example, intermediate entities or systems.

[056] Alternative audio data may represent, for example, an audio object that may include additional sub-objects.

[057] The method may further comprise the step of including rendering control data in the data stream of the audiovisual content item, wherein the rendering control data provides a rendering parameter for a rendering that uses the replaceable audio data different from a rendering that uses the alternate audio data.

[058] This may provide advantageous functionality and/or improved audio rendering for the content item.

[059] The method may specifically comprise the step of including rendering control data in the data stream of the audiovisual content item, wherein the rendering control data is indicative of a rendering adjustment for a rendering of the replaceable audio data different from that for a rendering of the alternative audio data.

[060] The method may further comprise the step of including an indication in the data stream of the audiovisual content item of the first audiovisual data stream comprising replaceable audio data.

[061] This may enable efficient adaptation of the functionality of a receiver that processes the data stream of the audiovisual content item.

[062] The method may further comprise the step of including extraction data for the substitutable audio data in the data stream of the audiovisual content item.

[063] This may enable efficient adaptation of the functionality of a receiver that processes the data stream of the audiovisual content item.

[064] The method may further comprise the steps of: generating a content item map table, wherein the content item map table for a content item comprises a list of audiovisual data stream packet identifiers for data packets for audiovisual data streams linked to the content item; and include the content item map table in the audiovisual content item data stream; wherein the step of generating the content item map table comprises generating the content item map table to include a data stream packet identifier for the content item of at least two of the plurality of streams (streams) of individual audiovisual data.

[065] This can provide improved and/or facilitated operation in many modalities. Specifically, it can enable a stream of content item data to be generated that can be processed with reduced complexity by different equipment, such as multiplexers, data extractors, etc.

[066] The step of generating the content item map table may comprise including only an audiovisual data stream packet identifier for the content item in the content item map table.

[067] This can provide improved and/or facilitated operation in many modalities. In particular, it may enable multiple functionalities to process the audiovisual content item data stream as if only one included a single audiovisual data stream.

[068] The step of generating the content item map table may comprise including only an audiovisual data stream packet identifier for the content item in the content item map table.

[069] This can provide improved and/or facilitated operation in many modalities. In particular, it may allow multiple functionalities to process the audiovisual content item data stream as if only one included a single audiovisual data stream.

[070] The step of generating the content item map table may comprise, for the common data stream packet identifier, including a plurality of secondary packet identifiers, each secondary packet identifier identifying a data stream audiovisual data from the set of at least two of the plurality of individual audiovisual data streams.

[071] This can provide greater flexibility and functionality. For example, it can enable efficient access and selection of individual audiovisual data streams. The primary and secondary package identifiers may, in some embodiments, be included in a structured or nested data structure or may, in other embodiments, for example, be provided in a flat structure, where, for example, the primary package identifiers and secondary are provided as sequential entries in a single list.

[072] The method may additionally comprise the steps of: generating an audio data processing descriptor for the second audio data of at least one of the audiovisual data streams, wherein the audio data processing descriptor is indicative the second audio data is alternative audio data or additional audio data; and include the content item map table in the audiovisual content item data stream.

[073] This can provide greater functionality and can enable, in particular, an efficient way to indicate how different audio data can be processed by a decoder.

[074] The method may further comprise the step of generating a content item map table, wherein the content item map table for a content item comprises a list of audiovisual data stream packet identifiers for packets of data for streams of audiovisual data linked to the content item; and wherein the step of generating the content item map table comprises generating the content item map table to include the audio data processing descriptor.

[075] This can provide particularly advantageous representation of information in many modalities.

[076] According to an optional feature of the invention, the method further comprises receiving a second data stream of the audiovisual content item comprising a plurality of audiovisual data streams, the plurality of audiovisual data streams comprising audiovisual components; and that the recovery comprises recovering the alternative audio data of an audiovisual data stream from the second data stream of the audiovisual content item.

[077] This can provide advantageous functionality and/or operation in many embodiments.

[078] According to an optional feature of the invention, the step of generating the audio signal comprises: decoding the mandatory audio data in a first audio decoder to generate a first decoded audio signal; decoding the alternative audio data in a second audio decoder to generate a second decoded audio signal; and generating the audio signal in response to the combination of at least the first decoded audio signal and the second decoded audio signal.

[079] This can provide improved and/or facilitated operation in many modalities.

[080] The audiovisual content item data stream comprises a content item map table, wherein the content item map table for a content item comprises a list of audiovisual data stream packet identifiers for data packets for audiovisual data streams linked to the content item, wherein the content item map table comprises a common data stream packet identifier for the content item for a set of at least two of the plurality of individual audiovisual data streams; and by the method further comprising the step of extracting audio data for the content item in response to the common data stream packet identifier.

[081] This can provide improved and/or facilitated operation in many modalities. Specifically, it can enable a stream of content item data to be generated that can be processed with reduced complexity by different equipment, such as multiplexers, data extractors, etc.

[082] According to an optional feature of the invention, the content item map table comprises a plurality of secondary packet identifiers, each secondary packet identifier identifying an audiovisual data stream from the set of at least two of the plurality of individual audiovisual data streams; and by the step of extracting audio data for the content item comprising extracting audio data for different audio components in response to the secondary packet identifiers.

[083] This can provide greater flexibility and functionality. For example, it can enable efficient access and selection of individual audiovisual data streams. The primary and secondary package identifiers may, in some embodiments, be included in a structured or nested data structure or may, in other embodiments, for example, be provided in a flat structure, where, for example, the primary package identifiers and secondary are provided as sequential entries in a single list.

[084] An apparatus for generating an audiovisual content item data stream comprising a plurality of individual data streams, wherein the plurality of individual audiovisual data streams comprises audiovisual components for an audiovisual content item ; wherein each audiovisual data stream comprises encoded data representing at least one audiovisual component and synchronization data; may comprise: a generator for generating a first audiovisual data stream comprising first audio data for an audio representation for the audiovisual content item, the first audio data comprising audio data required for the audio representation and audio data substitutable audio for the audio representation, wherein substitutable audio data is data substitutable with alternative audio data; and a combiner for including the first audiovisual data stream in the audiovisual content item data stream.

[085] According to an optional feature of the invention, the audio data processing descriptor is indicative of whether or not the second audio data replaces the replaceable data.

[086] This can provide a particularly advantageous and efficient implementation in many embodiments, while also enabling a high degree of flexibility and control in processing audio data from an audiovisual content item data stream.

[087] The approach can provide, in particular, an efficient interface between a receiver and an audio decoder. In many embodiments and scenarios, it can enable generic audio decoders to be used while providing flexibility in audio rendering and without requiring complex and/or proprietary control functions to be implemented between the receiver and the audio decoder.

[088] According to an optional feature of the invention, the audio data processing descriptor is indicative of whether or not the second audio data replaces the replaceable data.

[089] This can provide a particularly advantageous and efficient implementation in many embodiments, while enabling a high degree of flexibility and control in processing audio data from an audiovisual content item data stream.

[090] According to an optional feature of the invention, the audio data processing descriptor is indicative of whether the second audio data is designated alternative audio data or additional audio data, and the audio decoder is arranged so as to recover the second audio data as alternate audio data if the first audio data processing descriptor indicates that the second audio data is designated as alternate audio data and to retrieve the second audio data as additional audio data if the first data processing descriptor indicates that the second audio data is designated as additional audio data; and replacing the substitutable audio data with the second audio data if the second audio data is designated as substitutable audio data, and the output is arranged to generate the audio signal by combining the required audio data, the second data audio data and replaceable audio data, if the first audio data processing descriptor indicates that the second audio data is designated as additional audio data.

[091] This can provide a particularly advantageous and efficient approach to controlling audio rendering adaptation.

[092] According to an optional feature of the invention, the receiver is arranged to generate the first audio data processing descriptor from a second audio data processing descriptor received in the data stream of the content item audio-visual.

[093] This may enable both the data stream provider of the audiovisual content item and the local renderer or user to control the generated audio. The second audio data processing descriptor may describe suitable audio output that can be used, for example, by the user to determine suitable rendering.

[094] The approach can also facilitate implementation. For example, the receiver need not have any detailed knowledge or consideration of how specific audio processing is performed, but instead may simply base the first audio data processing descriptor on the second audio data processing descriptor. .

[095] According to an optional feature of the invention, the receiver is arranged to copy the second audio data processing descriptor to the second audiovisual data stream.

[096] The approach can also facilitate implementation. For example, the receiver does not need to have any detailed knowledge or consideration of how specific audio processing is performed, but instead can simply copy the second audio data processing descriptor.

[097] According to an optional feature of the invention, the second audio data processing descriptor is comprised in the content item map table.

[098] This can provide a particularly advantageous representation.

[099] According to an optional feature of the invention, the data stream of the audiovisual content item comprises a plurality of possible audio data processing descriptors for the second audio data; and by the receiver being arranged to select the first audio data processing descriptor from the plurality of possible audio data processing descriptors.

[0100] This may enable both the data stream provider of the audiovisual content item and the local renderer or user to control the generated audio. The second audio data processing descriptor may, for example, describe permitted or recommended combinations of the provided audio components, thereby providing alternative audio presentations for the content item. A user may simply select a suggested alternative audio presentation and the receiver may, in response, cause the selected audio presentation to be provided simply by copying the corresponding second audio data processing descriptor and including it in the data stream. supplied to the audio decoder.

[0101] These and other aspects, features and advantages of the invention will become evident and will be elucidated with reference to the embodiment(s) described hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

[0102] Embodiments of the invention will be described, by way of example only, with reference to the drawings, in which:

[0103] Figure 1 is an illustration of a transport flow, according to the prior art;

[0104] Figure 2 is an illustration of a transport flow, according to some embodiments of the invention;

[0105] Figure 3 is an illustration of an apparatus for generating a transport flow;

[0106] Figure 4 is an illustration of an apparatus for generating a transport flow;

[0107] Figure 5 is an illustration of a transport flow, according to some embodiments of the invention;

[0108] Figure 6 is an illustration of a transport flow, according to some embodiments of the invention;

[0109] Figure 7 illustrates the syntax of a supplementary audio descriptor of an MPEG-2 Transport Stream;

[0110] Figure 8 illustrates the syntax of the mix_type and editorial_classification data items of an MPEG-2 Transport stream;

[0111] Figure 9 illustrates a modified syntax of a mix_type data item, in accordance with some embodiments of the invention;

[0112] Figure 10 illustrates a modified syntax of an editorial_classification data item, in accordance with some embodiments of the invention;

[0113] Figure 11 illustrates a syntax of an AAC descriptor component data item, in accordance with some embodiments of the invention;

[0114] Figure 12 illustrates an identification of an object in an audio payload header;

[0115] Figure 13 illustrates an example of an object descriptor syntax to enable an object to delegate its content to a SAOC object;

[0116] Figure 14 illustrates an example of a reception apparatus comprising a receiver coupled to an audio decoder, the receiver receiving the data stream of the audiovisual content item;

[0117] Figure 15 illustrates an example of an operation that uses audio data processing for an MPEG (modified) transport stream;

[0118] Figure 16 illustrates another example of an operation that uses audio processing for a (modified) MPEG transport stream;

DETAILED DESCRIPTION OF SOME EMBODIMENTS OF THE INVENTION

[0119] The following description focuses on embodiments of the invention applicable to enhancing transport streams for audiovisual content, such as an MPEG-2 Transport Stream. However, it should be understood that the invention is not limited to this application, but that it can be applied to many other audiovisual content item data streams.

[0120] As a specific reference, an example of an MPEG DVB (Digital Video Broadcast) Transport (TS) stream as defined in ISO/IEC 13818-1 extended with standard DVB content, as in EN 300468, is illustrated in Figure 1. It should be considered that although the description focuses on embodiments that can, for example, improve such audiovisual content item data flow, the approach can be applied to many other transport technologies, including, e.g. , TCP/IP-based transport streams in the context of ISO/IEC 23009-1 DASH (Dynamic Adaptive Streaming over HTTP).

[0121] In the example, a transport stream is made up of a plurality of data packets. The transport stream data is divided into a plurality of elementary streams that correspond to an audiovisual component data, control or information (metadata). Each of the elementary streams is associated with a given identity, and each packet comprises a header that includes a Packet IDentifier (PID) that identifies the elementary stream for which the packet comprises data.

[0122] In the specific example, the transport stream is a time-multiplexed stream of packets with each packet containing 188 bytes. Each packet is labeled by a PID. A package may contain metadata or compressed audiovisual data. Packets with a given PID represent sequential data that needs to be joined together to create a single audiovisual component. For example, to transport an audiovisual component in the transport stream, the data for the audiovisual component is divided by the payloads of a group of data packets. Data packets from that group, and only from that group, will then use the same unique PID.

[0123] The transport stream additionally comprises a Program Association Table (PAT) that provides information about the programs that are included in the specific transport stream. Each program is characterized by a Program Map Table (PMT) that comprises information about the individual programs. Specifically, the PMT identifies the elementary streams comprised in the transport stream for the individual program and, in particular, lists the PIDs for the data packets that belong to the different elementary streams for the individual program.

[0124] Figure 1 illustrates an example of how elementary audio streams for a program can be provided in the transport stream. In the example, three audio components are provided to the program. Audio components can be rendered alongside a video component of the transport stream (not shown in Figure 1). Each of the audio components corresponds to a complete representation of the content item's audio sound platform, that is, each elementary stream provides the complete audio sound platform data needed to render (the term audio sound platform refers to - is to the audio of the entire rendered content item and is not limited to, for example, a specific audio scene).

[0125] Each of the elementary streams provides a single integrated audio component that needs to be fully rendered. This way, if an elementary stream is selected for rendering, the renderer encodes all audio data from the elementary stream to generate the single audio component of the elementary stream. This audio component is then rendered (possibly in some cases after being mixed with audio from another stream).

[0126] Each of the audio components is provided in a separate elementary stream identified by a separate PID. The audio data for each audio representation is provided in a series of data packets (referred to as PR1p1, PR1p2 PR2p1, PR2p2 PR3p1, PR3p2 etc., with each packet having its own identification p1, p2.. and each group of packets receives the identification of the audio presentation to which it belongs “PR”x). In this way, each of the audio components is represented in the transport stream by data packets comprising the audio data encoded for the audio component corresponding to the complete audio representation. Each audio representation is provided as a single audio component in an individual elementary stream, and as such, each audio representation is associated with a separate PID. In the example in Figure 1, the audio representations are associated respectively with PID 1, PID 2 and PID 3.

[0127] As illustrated in Figure 1, data packets are time multiplexed in the transport stream. A receiver can receive the data packets, extract the audio data from each data packet associated with a specific PID, decode the audio data, and thereby recreate the individual audio component/representation corresponding to that elementary stream.

[0128] To enable the receiver to synchronize audio data originating from different data packets and synchronize the resulting audio component, for example, with the associated video stream, each data packet comprises timing or synchronization data from the audio segment. audio represented by the audio data of the data packet.

[0129] An advantage of providing a plurality of elementary audio streams is that it is possible to provide more flexible audio. For example, Figure 1 illustrates an example in which three elementary audio streams are provided. Each of the elementary streams provides a complete representation of the audio scene including, for example, background data, music, dialogue, etc. The complete representation is provided by a single audio component. However, the three elementary streams provide different representations, for example, the three elementary streams may correspond to the same video sequence, but with audio in three different languages. For example, PID 1 may represent the original audio scene with dialogue in the original language, for example, English, while PID 2 and PID 3 may represent the audio scene, but with the dialogue dubbed in, for example, French and German , respectively. PMT can provide data indicating the language associated with each individual elementary stream. Consequently, a user can select a preferred language and the receiver can therefore select the corresponding elementary stream. In this way, the receiver extracts the audio data from all packets with the corresponding PID and then generates the audio output signal from this audio data. For example, a user may indicate a preference for dialogue in French, and therefore the renderer may extract audio data from elementary stream packets with PID2 and then generate audio output from this audio data.

[0130] Consequently, multiple alternative streams may be provided, with each elementary stream providing a single audio component that provides a complete representation of the audio for the content item (i.e., with the audio for the program). Then, a receiver can select between individual streams. Thus, in conventional transport streams for audio broadcasts, each alternative language is provided as a complete and independent audio representation given by a single audio component in a single elementary stream. The receiver can just select between the different available streams and once a stream is selected, all the audio from that stream is rendered.

[0131] However, this approach results in a low degree of flexibility, as only a single audio component corresponding to the complete audio platform is available to be rendered at the receiver. Additionally, it results in a high data rate of data streams as complete audio representations are provided.

[0132] MPEG-2 Transport Streams enable supplementary audio components to be defined to provide additional Audio Description audio (additional audio that provides descriptive commentary for visually impaired users). Such supplementary audio components cannot be rendered by themselves, but can be added to a full representation audio component of another elementary stream.

[0133] In this way, traditional MPEG-2 Transport stream elementary audio streams can be one of two types. The first type is the most common type, where a single audio component provides a complete audio representation for the content item. This audio can be rendered directly and is called “mixed broadcast” audio. The second type provides only a single component with supplemental audio that needs to be added to (mixed with) an audio component of the first type. In this way, the second elementary data type provides additional data that can be added to a complete audio representation. This is called “receiver mixed” audio.

[0134] To be able to decode a received transport stream, the transport stream includes information that defines the data contained in the transport stream. In more detail, information about the programs included in the stream is indicated in PAT. For each program, the number and types of audio components/tracks are made known to the receiver (e.g., a DVB Stream Decoder) through PMT signaling. PMT information is based on PID and specifically comprises a data structure link that contains all signaling required by PID. For each PID, there can be specifically a language descriptor and a codec specific descriptor. The language descriptor can be an ISO 639 Language descriptor or a Supplementary Audio descriptor. A supplemental audio description provides a clear differentiation between the mixed broadcast and receiver audio signals. The codec-specific audio description provides information about the contents of the packetized elementary stream of the given PID. This information is mainly about whether the component is a multichannel or stereo component.

[0135] Thus, in the current approach, each elementary stream provides a separate audio representation for the content item, except that the indicated elementary streams are mixed by the receiver. These do not provide an audio presentation, but instead provide an audio signal that needs to be added to a mixed broadcast audio signal.

[0136] Next, an approach that provides a more flexible and efficient transport flow will be described. The description will focus on an example where a plurality of languages are provided within the transport stream, but it should be considered that the approach can be used with other types of audio as well.

[0137] Figure 2 illustrates an example of a modified transport flow. Similar to the approach of Figure 1, the transport stream of Figure 2 comprises a plurality of elementary audio streams. However, unlike the prior art, elementary streams are not limited to complete audio representations or supplementary audio data and are not limited to simple audio components. Instead, an elementary stream may comprise different types of audio data and, specifically, may comprise both data that is mandatory for the audio representation and replaceable audio data.

[0138] Specifically, the transport stream may include data that defines/designates/describes one or more audio representations. A first elementary stream may be associated with an audio representation for the audiovisual content item represented by the transport stream. This first elementary stream comprises audio data mandatory for the audio representation, as well as substitutable audio data for the audio representation. Mandatory audio data corresponds to a mandatory audio component that needs to be rendered when the audio representation of the elementary stream is rendered. Conversely, replaceable data can be replaced with alternative audio for the audiovisual content item. Thus, the first elementary stream comprises both a mandatory audio component and a replaceable audio component for audio representation, with the replaceable audio component being replaceable by another audio component.

[0139] The first elementary audiovisual data stream may therefore comprise a plurality of audio components or objects, with an audio component/object being represented by the required audio data. This audio component is mandatory and must always be rendered when the specific audio representation (associated with the first audiovisual data stream) is selected, and thus when the first elementary stream is selected for rendering.

[0140] Another audio component is represented by replaceable audio data. This replaceable audio component is consequently not mandatory for representation, but may be replaced by an alternative audio component. However, the replaceable audio component cannot simply be erased or discarded while providing audio presentation, but can instead be replaced with an alternative audio component. The replaceable audio component may correspond to an audio source required for the audio representation, but with different versions of the audio source being included in the transport stream.

[0141] In many embodiments, the transport stream may provide a standard representation of the audiovisual content item when the mandatory audio component, represented by the mandatory audio data, and the replaceable audio component, represented by the replaceable audio data, together , provide a standard audio representation for the content item (possibly along with other audio or data components). In this way, the replaceable audio data generally corresponds to a standard audio component. When rendering the audio representation, the receiver will therefore, by default, select mandatory audio data and replaceable audio data. Consequently, the receiver can decode the mandatory audio component and the replaceable audio component, mix the components, and render the resulting audio. This will result in the receiver rendering the default version of the audio representation of the content item.

[0142] However, the transport stream may comprise, at the same time, data for alternative audio. However, this alternate audio need not be provided as complete alternate audio representations in separate elementary streams providing a complete audio representation. Instead, alternative audio data is provided that may simply define an alternative audio component that can replace the replaceable audio component of the first elementary stream. In this way, the transport stream may comprise alternative audio data that may replace part of the substitutable audio data of the first elementary stream.

[0143] Thus, the first elementary stream comprises two types of data, namely, mandatory audio data for (at least) one mandatory audio component and replaceable audio data for (at least one) replaceable audio component.

[0144] Specifically, the first elementary stream may comprise data packets comprising obligatory audio data and data packets comprising replaceable audio data. Each data packet may specifically comprise one of mandatory audio data or replaceable audio data, that is, in many embodiments, a data packet will comprise only audio data for an audio component.

[0145] The mandatory audio component and the replaceable audio component are thus time-parallel components, i.e., for a given content item rendering/time segment, the first elementary stream may comprise both audio data mandatory audio data for a mandatory audio component, and substitutable audio data for a substitutable audio component. However, mandatory audio data and substitutable audio data for the same time segment are generally provided in different data packets that are sequential in time in the first elementary stream and the transport stream.

[0146] Similarly, the alternative audio data may be provided in data packets that are sequential in time in the transport stream to the data packets of the mandatory audio data, and the substitutable audio data of the first elementary stream provides, also, audio data for the same time segment (rendering).

[0147] In this way, the timing of data packets and the data in the transport stream can be independent (and clearly not identical) from the timing of the corresponding audio in the audio components. Specifically, for at least one time segment, the elementary stream comprises separate audio data for a mandatory audio component and a replaceable audio component, that is, it comprises both mandatory audio data and replaceable audio data for the time segment. time.

[0148] Figure 2 illustrates an example in which PID 1 is assigned to a first elementary flow. The first elementary stream not only comprises a single audio component, but instead comprises audio data for both a mandatory audio component and a replaceable audio component.

[0149] Specifically, the required audio component may correspond to general background audio (including, for example, music) for the content item. Data packets comprising mandatory audio data are referred to in Figure 2 as BG1p1, BG1p2, BG1p3, BG1p4.

[0150] The replaceable audio component may correspond to dialogue in a standard language, e.g., English. Data packets comprising replaceable audio data are referred to as AO A in Figure 2.

[0151] As illustrated, the mandatory audio data and the replaceable audio data are provided in individual data packets that are time multiplexed to the transport stream. In the example, the mandatory audio data and the substitutable audio data are provided in separate data packets, that is, a data packet of the first elementary stream comprises mandatory audio data or substitutable audio data. However, it should be considered that in some embodiments, a data packet may comprise both mandatory audio data and replaceable audio data.

[0152] A receiver that performs rendering of the standard audio representation with the standard language therefore only needs to access the first elementary stream, that is, it only needs to decode and process the data packets with PID1. Then, it may individually decode the mandatory audio data to generate the mandatory audio component and the substitutable audio data to generate the substitutable audio component. The audio signal for the content item can then be generated by mixing the required audio component and the replaceable audio component.

[0153] However, in addition to the first elementary stream, the transport stream comprises other elementary streams that comprise alternative audio data for the replaceable audio data, that is, it comprises data that defines at least one alternative audio component that can replace the replaceable audio component.

[0154] In the example of Figure 2, the transport stream comprises a second elementary audio stream identified by PID 2 and a third elementary audio stream identified by PID 3. Consequently, the data packets corresponding to the third and third elementary streams are time multiplexed in the data stream with the data packets for the first elementary stream. The second elementary stream comprises data packets (called AO B) with alternative first audio data, and the third elementary stream comprises data packets (called AO C) with alternative second audio data. The first alternative audio data may be decoded to generate a first alternative audio component and the second alternative audio data may be decoded to generate a second alternative audio component.

[0155] Instead of generating an audio signal for the content item using the replaceable audio component, a renderer may use one of the alternative audio components, that is, audio generation for the content item may be based on one of the alternative audio data sets rather than being based on the replaceable audio data.

[0156] For example, the substitutable audio data may provide English language dialogue, the first alternative audio data may provide German dialogue, and the second alternative audio data may provide French dialogue. So a receiver, by default, will decode the first elementary stream, generate the background audio component and the English audio component, and mix them to generate the audio for the content item. However, for example, in response to user-entered data, the receiver may replace the substitutable audio data with the first alternative audio data, that is, it may proceed to decode the required audio data to generate the required audio component and the first alternative audio data to generate the first alternative audio component. Then, the output audio can be generated by combining the mandatory audio component and first alternative audio component. This way, instead of the content item having dialogue in English, dialogue in German is provided.

[0157] The approach enables very efficient communication and audio coding in a transport stream. Reduced data rates can be achieved and/or greater flexibility can be provided. Furthermore, easy processing is achieved, in terms of both generation and processing of the transport stream.

[0158] Notably, the specific approach can provide a high degree of compatibility with existing transport streams, such as the MPEG-2 Transport stream. Specifically, an approach is provided in which existing data structure and organization can be maintained, and specifically the same elementary streams approach can be used to provide greater flexibility. In fact, the approach will minimize (or at least reduce) the amount of standardization effort required to adopt the innovative approach. What's more, it can increase backwards compatibility and reduce or even minimize the changes required to support additional functionality.

[0159] Specifically, the approach may use the same transport flow approach comprising a PAT defining one or more content items and specifically providing a PID to a PMT for each content item. The PMT for each content item can define a PID for each elementary stream associated with the content item (in this way, the content item can be made up of the elementary streams for which a PID is included in the PMT). Then, the receiver can retrieve the audiovisual data for the content item by selecting data packets with a PID indicated in PMT. However, because the content item may have alternative audio tracks, the receiver may, for example, select to ignore or extract data from a given PID, depending on whether that PID understands necessary audio data or not. For example (and with reference to Figure 2), if a German dialogue track is provided as alternative data in data packets with PID 2, the receiver may ignore these data packets if English dialogue of PID 1 is desired. This way, if the standard audio scene is desired, the receiver in this example only needs to consider PID1 data packets to provide the desired audio. However, at the same time, if the alternative German dialogue audio track is preferred, the receiver can simply extract this audio data from the PID 2 data packets and replace the substitutable audio data from the PID 1 data packets with this audio data.

[0160] The approach follows a high degree of operational efficiency. In fact, the system can simply render a single elementary stream for nominal operation, and in this way the approach not only provides different options, but can also treat an option as a preferred option. For example, the original dialogue is included in the same elementary stream as the background audio, so it will always be provided unless a user specifically requests an alternative.

[0161] Figure 3 illustrates an example of an apparatus for generating an audiovisual content item data stream, and specifically a transport stream such as the MPEG-based transport stream of Figure 2. The content item data stream audiovisual will be called transport flow. The content item is made up of a series of audiovisual components, such as one or more audio components and usually one or more video components. A content item can also include components that support interactivity. Each audio component can be an audio signal and in many cases can be an audio object. Each video component may be a video signal corresponding to a single moving image, or may be, for example, a moving image in multiple views. A content item can also include components that support interactivity. For example, an interactivity component may be available to assist the user in their selection of an audio object and/or video component.

[0162] The apparatus generates a transport stream comprising a plurality of individual audiovisual data streams. The audiovisual components of the content item are consequently distributed across a plurality of individual audiovisual data streams. Each of the individual audiovisual data streams therefore corresponds to an elementary stream, as previously described, and will be referred to as an elementary stream. Each elementary flow can be associated with a unique packet identifier (within the transport flow), such as a PID specifically.

[0163] The plurality of elementary streams (audiovisual data streams) consequently comprises the audiovisual components of the content item. In the approach shown in Figure 1, each audiovisual component is comprised of a single elementary flow. However, on the other hand, the apparatus of Figure 2 is arranged to generate the transport stream, so that at least one of the elementary streams comprises a plurality of audiovisual components.

[0164] The audiovisual components are comprised in elementary flows by elementary flows comprising encoded data for the audiovisual components. In this way, each elementary stream comprises coded data that represents at least one audiovisual component. In some embodiments, some elementary streams comprising metadata such as control data, interactive data, etc. may be provided.

[0165] As previously described, the elementary streams are made up of a plurality of time-multiplexed data packets and, consequently, the data encoded for the audiovisual components are included in the data packets of the transport stream. Therefore, the timing of the individual data packet in the transport stream does not directly correspond to the timing of the audiovisual component segment in the data packet when it is rendered. To ensure correct rendering timing, each elementary stream comprises synchronization information that allows it to synchronize with audiovisual components in other elementary streams. Specifically, each data packet may include synchronization data that defines the timing of the content segment represented by the data packet.

[0166] The description and illustration of Figure 3 will focus on the part of the transport flow related to audio components. It should be considered that, in addition, the transport stream may be generated to include metadata, control data, video content items (and possibly other audio items), etc. For example, the well-known approach of several conventional MPEG-2 Transport stream implementations can be used for such data.

[0167] The apparatus of Figure 3 comprises an audio source 301 that provides encoded audio data to a plurality of audio components. In some embodiments, the audio source 301 may receive a plurality of audio components and may encode them individually. The encoding may use the same encoding algorithm or may, in many embodiments, use different encoding algorithms and standards for different audio components. Furthermore, encoding parameters, such as the resulting data rate, may be the same or different for different audio components.

[0168] In other embodiments, the audio source 301 may directly provide the encoded audio data. For example, audio source 301 may be local storage in which various encoded audio components are stored.

[0169] Audio source 301 specifically provides encoded audio data for audio components representing the content item. It typically provides all audio to be played when playing a video component included in the transport stream. For example, for a movie, one or more elementary streams may provide the video for the movie (e.g., different elementary streams may provide different resolutions of the video. Audio source 301 may provide the associated audio divided into multiple audio components.

[0170] In the approach of Figure 3, a single elementary stream is generated that comprises a complete representation of a sound platform to be rendered in conjunction with the rendering of the content item. For example, an elementary stream is generated that comprises all the audio necessary to render the audio of, for example, a movie or television program.

[0171] The transport stream may be specifically generated so as to explicitly comprise data defining one or more audio representations. The definition of an audio representation may be provided, for example, by a direct or indirect identification of audio data corresponding to mandatory audio data and substitutable audio data for the audio representation. Generally, the definition of the audio representation may also include an identification of alternative audio data that can replace the substitutable audio data.

[0172] In the approach, the first elementary stream does not comprise a single audio component, but instead comprises both a mandatory audio component and a replaceable audio component. In this way, the elementary stream is generated to include both mandatory audio data for the mandatory audio component and substitutable audio data for the substitutable audio component, as previously described. The required audio component can specifically correspond to background audio, and the overridable audio component can correspond to the original dialogue in the original language (for example, English).

[0173] In the example of Figure 3, audio source 301 provides mandatory audio data for the mandatory audio component to a first packager 303. The first packager 303 proceeds to package the mandatory audio data. This packaging includes dividing the required audio data into appropriately sized blocks. It also includes adding sync data to data packages. Furthermore, the first packetizer 303 adds a header with relevant information that enables the correct decoding of the data packet. The header is generated specifically to include a PID, and the PID allocated to the first elementary stream is consequently included in the data packet.

[0174] Similarly, audio source 301 provides substitutable audio data for the substitutable audio component to a second packager 305. The second packager 305 proceeds to package the substitutable audio data. This packaging similarly includes dividing the replaceable audio data into appropriately sized blocks, summing synchronization data, and a packet header. As for the first wrapper 303, the second wrapper 305 generates the header to include a PID corresponding to the PID allocated to the first elementary stream. This way, the data packets for the mandatory audio component and replaceable audio component have the same PID and are consequently part of the same elementary stream.

[0175] The audio source 301 further provides alternative audio data to a third packer 307. The third packer 307 proceeds to package the replaceable audio data. This packaging similarly includes dividing the replaceable audio data into appropriately sized blocks, summing synchronization data, and a packet header. As for the first wrapper 303, the third wrapper 307 generates the header to include a PID. Alternative audio data is, as previously described, an alternative to replaceable audio data. In this way, alternative audio data is data that encodes an alternative audio component that can be used instead of the replaceable audio component. For example, the alternate audio component may correspond to a dubbed language (e.g., German) and this may replace the original language (e.g., English) when rendering the content item comprised in the transport stream.

[0176] In the specific example, the alternate audio data is included in a different elementary stream than the first elementary stream and consequently the alternate audio data is generated to have a different PID than the data packets for the mandatory audio data and replaceable audio data. However, as will be described later, alternative audio data may, in some embodiments, be included in the first elementary stream itself, and thus, in some embodiments, data packets may be generated so that they have the same PID as the mandatory audio data and alternative audio data.

[0177] The first, second and third packers 303, 305, 307 are coupled to a stream generator 309, which proceeds to generate the transport stream by time multiplexing the data packets from the packers 303, 305, 307. It should be considered that, generally, several other audio components can also be encoded. These audio components (including, possibly, either mandatory audio components for other audio representations or alternative audio components) may be packaged in a similar manner. The stream generator 309 may be arranged to additionally include such data packets in the time multiplexed transport stream. Similarly, data packets can be generated for video components, possibly mixed audiovisual components, metadata or control data, and the resulting data packets can be included in the transport stream by time multiplexing.

[0178] The apparatus further comprises a control data generator 311 which is arranged to generate control data for the transport flow. The control data generator 311 specifically generates control data that describes what data is included in the transport stream. Specifically, control data generator 311 can generate PATs and PMTs. The resulting data packets are fed into stream generator 309 which proceeds to include the data in the transport stream. The control data generator 311 may specifically include data that defines or describes data associated with a specific audio representation for the content item.

[0179] In this way, the apparatus of Figure 3 can generate a transport stream as described with reference to Figure 2, that is, a transport stream can be generated in which at least one first elementary stream comprises data for a plurality of of audio components, with at least one audio component being mandatory for the audio representation of the content item, while the other audio component may be replaced by another audio component. In this way, the advantages described previously can be obtained by the device in Figure 3.

[0180] Figure 4 illustrates an example of an apparatus for generating an audio signal for an audiovisual content item. Specifically, the apparatus receives the transport stream from the apparatus of Figure 3 (specifically a transport stream is shown in Figure 2) and generates an audio output signal for the content item. The output signal can generally be used to render audio and the apparatus of Figure 4 will consequently be referred to as a renderer.

[0181] The renderer comprises a transport stream receiver 401 that receives the transport stream.

[0182] The renderer further comprises a first data extractor 403 which is arranged to extract the required audio data from the received transport stream. Specifically, the first data extractor 403 is arranged to detect data packets with the PID corresponding to the first elementary transport stream. The extractor can then, from these data packets, extract the data from the subset of data packets to which the data corresponds to the required audio component.

[0183] The renderer also comprises a second data extractor 405 which is arranged to extract the required audio data from the received transport stream. Specifically, the second data extractor 405 is arranged to detect data packets with the PID corresponding to the first elementary stream. The extractor can then, from these data packets, extract data from the subset of data packets to which the data corresponds to the replaceable audio component.

[0184] Information may be provided about the individual data packet comprising substitutable audio data or mandatory audio data, in some embodiments, by control or transport stream metadata not comprised in the first elementary stream. For example, a PMT may be provided that, for the first elementary stream, defines an order of data packets in the elementary stream. For example, the PMT may define a repeated sequence of two data packets with mandatory audio data followed by a data packet of replaceable audio data. In other embodiments, the information may be provided, alternatively or additionally, in individual data packets. For example, a marker in the header of each data packet may indicate whether the data packet comprises mandatory audio data or replaceable audio data. In simple embodiments, a single bit marker may be provided. In other embodiments, an identity of the individual audio component of the elementary stream may be provided.

[0185] The renderer also comprises a third data extractor 407 that is arranged to extract alternative audio data from the received transport stream. Specifically, in examples where alternative audio data is provided in a different elementary stream, the third data extractor 407 may be arranged to detect data packets with a PID corresponding to that different elementary stream. The third extractor can then, from these data packets, extract the alternative audio data.

[0186] The first data extractor 403 is coupled to a first decoding unit 409 to which the required audio data is provided. The first decoding unit 409 then proceeds to decode the required audio data to generate the required audio component.

[0187] Similarly, the second data extractor 405 is coupled to a second decoding unit 411 to which replaceable audio data is provided. The second decoding unit 411 then proceeds to decode the substitutable audio data to generate the substitutable audio component.

[0188] Correspondingly, the third data extractor 407 is coupled to a third decoding unit 413 to which alternative audio data is provided. The third decoding unit 413 then proceeds to decode the alternative audio data to generate the alternative audio component.

[0189] In some embodiments, the first, second and third decoding units 409, 411, 413 may utilize the same decoding algorithm, that is, the same algorithm or coding pattern may have been used to generate the encoded audio data . However, even in such embodiments, different decoding units (and thus encoders at the source end) may have different encoding/decoding parameters. Specifically, the data rate of the audio data may be different for mandatory audio data, replaceable audio data and/or alternative audio data, and consequently the decoding units may decode audio data with different data rates. .

[0190] In some embodiments, at least two of the first, second and third decoding units 409, 411, 413 may use different decoding algorithms, that is, different algorithms or coding patterns may have been used to generate the decoding data. encoded audio.

[0191] In this way, the approach can enable individual optimization or adaptation to the characteristics and preferences of each individual audio component. The decision to encode adjustments/algorithms can be performed dynamically, for example, based on the characteristics of the audio components.

[0192] The first, second and third decoding units 409, 411, 413 are coupled to a combiner 415 which is arranged to generate an output signal by combining the required audio component with one of the replaceable audio component and the alternate audio component. The combination can be, in many scenarios, a simple (possibly weighted and/or frequency-selective) sum or mix.

[0193] Specifically, when operating in standard mode, the output signal is generated by combining the mandatory audio component and the replaceable audio component. This way, the output signal is generated to match the standard audio representation for the content item. In the specific example, an audio track with dialogue in English is generated for the video component.

[0194] However, for example, in response to data implicitly entered by the user, the apparatus of Figure 4 may instead generate the output signal by combining the mandatory audio component and the alternative audio component. Therefore, in this scenario, the device proceeds to replace the replaceable audio component with the alternative audio component. This could result, for example, in the generation of an audio track with German-dubbed dialogue for the video component.

[0195] In this way, the output audio signal can, for example, be rendered or reproduced by driving a speaker based on the output audio signal. Such drive may be via a suitable drive circuit including, for example, a power amplifier, filter etc., as is well known in the art. It should also be considered that the approach can be used for multichannel signals - for example, by having all data, components and processing directly in the multichannel domain (for example, audio data can represent encoded multichannel signals that are decoded to generate multichannel audio components that can then be selected and combined by a multichannel combiner). As another example, the described approach can be performed individually for each individual channel of a multichannel signal.

[0196] As yet another example, single-channel and multi-channel audio components may be mixed, for example, the mandatory audio component may be a multi-channel signal and the substitutable and alternative audio components may be single-channel signals. .

[0197] In the previous example, the substitutable audio data is replaced by the alternative audio data when combiner 415 makes the selection to combine the alternative audio component and the mandatory audio component. It should be considered that in some such embodiments, the apparatus of Figure 4 may not extract or decode data that is not being used. Specifically, if standard operation is selected, the device will not extract the alternate audio data and decode the alternate audio component. Similarly, if alternative audio data is used to generate the output signal, the apparatus may not extract the substitutable audio data or decode the substitutable audio component.

[0198] In the example, replacing (or not replacing) the replaceable audio data with the alternative audio data is accomplished by selecting the audio component combined with the required audio component. However, it should be considered that selection and replacement can be performed directly in the domain of encoded audio data. For example, the same decoder may be used for all audio components being decoded and the apparatus may be arranged to directly replace substitutable audio data with alternative audio data in a data stream being fed into the decoding unit.

[0199] In some embodiments, the combination of mandatory audio and substitutable (standard) audio or alternative audio can be performed in the compressed and/or encoded domain. In such an embodiment, a combiner 415 may be provided with the relevant compressed/encoded audio data from the data extractors 403, 405, 407 and the same may proceed to select and combine the desired audio data. Then, the combined audio data can be fed to a simple decoding unit that decodes the combined audio data.

[0200] Alternative audio data may be provided in different parts of the transport stream in different embodiments, or may even, in some cases, be provided external to the transport stream.

[0201] Thus, in some embodiments, the apparatus may be arranged to receive a second transport stream from the same or a different source. The apparatus may be arranged to process such a second transport stream and may therefrom extract an alternative audio component that may replace the substitutable audio component of the first transport stream, as previously described for an alternative audio component. included in the same transport flow. The second transport stream may be received, for example, from another source via an Internet connection.

[0202] Such an approach may enable, for example, the replaceable audio component to be retrieved from a different provider, for example, over an Internet connection. The approach could, for example, enable a third party to provide language dubbing for films, for example.

[0203] A specific example of a second transport stream may thus be a transport stream received or retrieved from a different data source or from a different file than the first transport stream. For example, the first transport stream may be received from a primary data source by a broadcast transmission. For example, the first transport stream may be from a television station that broadcasts the first transport stream to a large number of receivers and users. Thus, this communication can be, for example, a radio broadcast from one point to multiple points. However, the second transport stream comprising alternative audio data may be transmitted directly to the user via a different communication medium, such as via the Internet. The second transport stream can be transmitted as a point-to-point transmission directly to the individual receiver/user. In fact, the second transport stream may even be specifically requested by the user and provided through an interactive medium such as the Internet. Such an approach can enable efficient distribution and communication, while at the same time enabling a very high degree of customization by the potential user.

[0204] Consequently, the second transport stream may possibly be recoverable from or through a different source than the first transport stream. In the case of files, the transport streams can be from different files. A different source for the second transport stream may also be, for example, a different medium (e.g., the first transport stream may be retrieved from a Blu-ray Disc and the second transport stream may be from a radio broadcast). ). The primary source providing the first transport stream may be a broadcast, or multicast signal, while the second source providing the second transport stream may be one directed to a specific individual.

[0205] However, in many embodiments, the alternative audio data will be part of the same transport stream.

[0206] Specifically, the apparatus that generates the transport stream, in some embodiments includes the alternative audio data in a different elementary stream in the transport stream. This can provide a particularly efficient approach and can, in many scenarios, facilitate operation and signaling, and can provide more efficient transport flow. In such embodiments, a receiver may extract the alternative audio data by identifying data packets with a PID corresponding to the elementary stream in which the alternative audio data is comprised.

[0207] An example of such a scenario is shown in Figure 5. In the example, an elementary stream with a PID of 1, comprises two audio components, namely, a mandatory audio component corresponding to background sounds, and the obligatory audio component corresponding to background sounds. replaceable audio, corresponding to a default language. The mandatory audio component is represented by audio data in data packets referred to as BG1p1, BG1p2, etc. and the substitutable audio component is provided by audio data in data packets referred to as OA C. Furthermore, a second elementary stream with a PID of 2 comprises an alternative audio component represented by audio data in data packets referred to as OA A. A third elementary stream with a PID of 3 also provides an alternative audio component represented by audio data in data packets referred to as OA B. It should be noted that in Figure 5, the data packets for OA A and OA B are shown as being concurrent with the data packets for OA C. This may reflect scenarios in which data packets may be included simultaneously (for example, with channel interleaving in the data packets). However, it must also be considered that in many embodiments, data packets can be time multiplexed. The two alternate audio components can correspond to different languages that can override the language of the replaceable audio component. Specifically, as illustrated in Figure 5, PID 2 elementary stream substitutable audio data can replace the OA C elementary stream substitutable audio data.

[0208] In this way, a renderer can extract the background audio data (the mandatory audio data) from the PID 1 elementary stream and decode it to generate a mandatory audio component corresponding to background audio. Furthermore, it can extract alternative audio data from the PID 2 elementary stream and decode it to generate an alternative audio component corresponding to one of the dubbed languages. The two components are then combined to generate the output signal that is rendered/played.

[0209] As indicated in Figure 5, the data packets for the substitutable audio data and the alternative audio data do not necessarily correspond to each other. For example, they may be encoded differently (e.g., different encoding parameters, data rate, or encoding algorithms) and may specifically correspond to different time slots or segments. However, if, for example, separate decoding followed by combining the audio components is used, such differences can be handled by adapting the different decoding paths and combining the decoded signals.

[0210] In some embodiments, the second elementary stream may not only comprise the alternative audio component, but may also comprise, for example, multiple alternative audio components.

[0211] Indeed, in some embodiments, the second elementary stream may additionally comprise audio data required for a different representation for the content item. Thus, in some embodiments, the second elementary stream may be an elementary stream that itself provides a different audio representation for the content item. Consequently, a receiver may be able to generate a soundtrack for the content item using only the audio data from the second elementary stream. For example, the second elementary stream may comprise mandatory audio data and substitutable audio data for an alternative audio representation for the content item. In such an example, the alternative audio data for the first elementary stream (and thus the first representation) may be, for example, substitutable audio data from the second elementary stream (i.e., for the second representation).

[0212] In some embodiments, the alternative audio data may be provided in the first elementary stream itself, that is, it may be part of the same elementary stream that includes the mandatory audio data and the replaceable audio data. In this way, the apparatus of Figure 3 can be arranged to include alternative audio data in the first elementary stream by time multiplexing data packets, including alternative audio data in the transport stream, the PID of these data packets being It is the same for data packets with mandatory audio data and replaceable audio data. The apparatus of Figure 4 may, as a standard, extract the audio data from the data packets comprising the mandatory audio data and the replaceable audio data and proceed to generate the output audio signal from this data. However, for example, in response to data input by the user, the device may instead extract alternative audio data from the appropriate data packets and generate the audio output signal using this data, rather than the audio data. replaceable.

[0213] An example of such an approach is shown in Figure 6, where the first elementary stream comprises data packets with mandatory audio data (BG1p1...), with replaceable audio data (OA A) and with alternative audio data (OA B, OA C). In the example, the mandatory audio data is extracted to generate the mandatory audio component, and the OA B data packet data is extracted and decoded to generate an alternative audio component instead of the default substitutable audio data. The output audio is generated by combining the decoded mandatory audio component and decoded alternate audio component.

[0214] The previous description focused on examples where alternative audio data corresponds to different audio components, such as different languages. However, it should be considered that the described approach is not limited to such an example.

[0215] In fact, in some embodiments, the substitutable audio data and the alternative audio data may represent the same audio content encoded using different encoding characteristics. Specifically, the substitutable audio data and the alternative audio data may correspond to the same audio content and, specifically, the same audio component, but encoded at different bitrates.

[0216] This can provide greater flexibility and can enable, for example, a receiver to receive a transport stream and efficiently and with low complexity generate a transport stream corresponding to the same audio content but at a different data rate . Thus, the selection between substitutable audio data and alternative audio data may depend on characteristics of the data itself, such as dependent on the specific data rate and the desired data rate. The approach can enable greater flexibility and can specifically enable flexible adaptation of the characteristics of a transport flow.

[0217] In some embodiments, the alternative audio data may be, for example, an audio object, such as a SAOC audio object, representing multiple sub-objects.

[0218] In some embodiments, the apparatus of Figure 3 may be arranged to include the rendering of control data in the transport stream. The rendering control data can specify a required, desired, or suggested rendering parameter for a rendering of the transport stream audio. For example, rendering control data may provide an indication of an absolute or relative level of audio components.

[0219] The rendering control data may specifically provide a different rendering parameter for when substitutable audio data is used than when alternative audio data is used.

[0220] The rendering control data may describe, for example, a relative gain or signal level for the output signal when standard audio is rendered, that is, when the output signal is generated based on the audio data mandatory and audio data. They may additionally describe, for example, a relative gain or different signal level for the output signal when alternative audio is rendered, that is, when the output signal is generated based on required audio data and alternative audio data. .

[0221] Such an approach can be used, for example, to provide differences between rendering, for example, audio can be rendered louder, when the standard audio signal is rendered, than when the alternative audio signal is rendered. However, the approach can also be used to homogenize the output signal. For example, in situations where the level of the replaceable audio component is higher than the level of the alternate audio component, the render control data can increase the total gain of the output signal when using the alternate audio component, resulting in the reduction of the level difference.

[0222] In some embodiments, the rendering control data may describe different renderings for the replaceable audio component and the alternate audio component. For example, the gain for the replaceable audio component may be different from the gain for the alternate audio component. This can, for example, enable customization. As a specific example, German-language TV broadcasts may tend to have higher relative levels of commentator sound relative to crowd noise for sporting events than English-language TV broadcasts. Consequently, different gains can be adjusted for two different language components so that the same effect can be provided by a single transport stream with alternative languages.

[0223] In some embodiments, the rendering control data may provide different rendering parameters for (possibly only) the required audio data/required audio component, depending on whether the signal is generated based on the replaceable audio data/required audio component. replaceable audio or alternative audio data/alternative audio component.

[0224] To enable a receiver to fully utilize the provided transport stream, the transmitter may include additional data that may signal information about the different data comprised in the transport stream.

[0225] Specifically, in some embodiments, the apparatus of Figure 3 may be arranged to include an indication that the first elementary stream comprises replaceable audio data. This indication may be provided, for example, as part of separate configuration data for the transport stream. For example, data packets may be provided that comprise configuration packets that specify the different elementary flows of the transport stream. Specifically, configuration data that includes PMTs may be provided. In such a case, the part of the PMT that defines the individual elemental stream may be enhanced to also include information that defines which audio components are represented in the elemental stream, and specifically, whether the individual elemental stream comprises more than one audio component. For example, a marker may be defined to indicate that a given elementary stream comprises replaceable audio data.

[0226] In some embodiments, the transport stream may be generated to include extraction data for the substitutable audio data in the elementary stream. Such extraction data may specify, for example, how substitutable audio data may be identified. They may comprise, for example, an indication of which data packets include substitutable audio data and which data packets include mandatory audio data. The description can be provided, for example, by describing the audio components included in the elementary stream and associating each with an identity. Each data packet may then include an audio component identity in the header to indicate for which audio component the data packet comprises audio data. The extraction data can provide the audio component identity to each audio component, thereby allowing the receiver to extract the relevant information.

[0227] Specifically, for an MPEG-2 Transport stream, the supplemental audio descriptor can be extended by defining an object. An example of the syntax of a supplemental audio descriptor is provided in Figure 7. Figure 8 illustrates the conventional use of, respectively, mix_type and editorial_classification data items

[0228] In this example, the mix_type data data structure can be changed from that of Figure 8 to that of Figure 9. The editorial_classification can be changed to that of Figure 10. Furthermore, an AAC descriptor component, such as that in Figure 11, can be used.

[0229] This example shows that the reserved bit, under the editorial_classification data structure, can be used to indicate whether the audio contains objects that can be manipulated. In the case where the mixed_type is equal to 1, the object bit value 1 indicates that the audio track is a background object and needs to be mixed with an object. If the object bit is 0, which is the current case, this indicates that the audio track is complete and can be supplemented with an object or another mix_type 0 stream.

[0230] With the editorial_classification data structure, the use-case of a replaceable object can be signaled. The identification of that replaceable object can be placed in the AAC descriptor. The only signal that needs to be given at this level is the identification that a replaceable object exists.

[0231] The AAc descriptor extension may specifically include a link of objects with a descriptor per object describing its attributes, such as specifically: - The language of an object. - Replaceability: This can signal that the object is the default object in the flow and can be replaced by another object (and optionally, a link can be included with that alternative object). - Whether this is an object that a user can select to add or choose to leave out. For example, it could be a special soundtrack for a person with a disability or contain a special commentary. This gives an operator the ability to offer a commentary track at a very limited bandwidth cost. - A text indication that can be displayed with the object to make it clear to a user that it is an interactive object. For example, it may be possible for a user to select this object, place it in a different position or make it louder or softer, for example. - Provide a link to a video object that is coupled to the audio object. An interactive object may also have the ability to be moved along with its video counterpart. For example, it may be possible to “zoom in” on an object. Zoom can be viewed by the video object and noise by the audio object. So, the user has the illusion that the object is, in fact, closer, because only the audio from that object becomes louder.

[0232] Furthermore, an object in the payload header (UsacConfig) can be extended with a unique identifier through the metadata descriptor. This identifier can be included in the descriptor in the PMT to enable object identification. This identifier can also be an XML description in a file that describes the audio content.

[0233] Figure 12 illustrates how an object can be identified in UsacConfig.

[0234] Figure 13 illustrates an example of an object descriptor syntax to enable an object to delegate its content to a SAOC object.

[0235] In many embodiments, the data stream of the audiovisual content item may consequently have data, including audiovisual data, control data, metadata, etc., structured in a hierarchical arrangement.

[0236] Specifically, the audiovisual content item data stream (specifically a transport stream) may comprise a data stream content item descriptor that defines the content items that are included in the content item data stream audio-visual. The data stream content item descriptor may be specifically assigned to a predetermined packet identifier, which may be specifically standardized, and thus will be known at the transmitting end and the receiving end. In this way, a receiver can detect the data packets with the predetermined packet identifier and can, from these data packets, determine which content items are included in the data stream of the audiovisual content item. For example, for an MPEG transport stream, the transport stream may include a PAT that is included in data packets with a predetermined PID of 0.

[0237] The data flow content item descriptor may provide, specifically for each content item, a pointer to an item map table for the content item. The content item map table for a content item may comprise a list of audiovisual data stream packet identifiers for data packets for data streams linked to the content item. In this way, the content item map table comprises a packet identifier for each audiovisual data stream that provides audiovisual data for the content item, including, for example, video data, required audio data, replaceable audio data , alternate audio data, and additional audio data. Additionally, the content item map table may comprise pointers and, specifically, package identifiers to data streams that provide, for example, metadata, control data, etc. for the content item. The content item map table can be a program map table.

[0238] Specifically, for an MPEG-type data stream, the PAT may include, for each content item, a packet identifier for a PMT. The PMT for a program/content item may then include a PID for each elementary data stream that provides audiovisual data to the program.

[0239] Accordingly, audiovisual data streams are provided in data packets with packet identifiers corresponding to those indicated in the content item map table. For example, as previously described, mandatory and substitutable audio data may be provided in one audiovisual data stream by endowing the audio data in the packets with a packet identifier, the first alternative data may be provided in another audiovisual data stream. by providing the audio data in packets with another packet identifier, the second alternative data may be provided in another audiovisual data stream by providing the audio data in packets with a third packet identifier, etc. The Content Item Map Table is generated to include the bundle identifiers of the respective audiovisual data streams.

[0240] The approach provides a high degree of flexibility and enables a source to generate audiovisual content item data streams that are suitable for the particular scenario and that provide the desired content items in a desirable form. The approach can also substantially facilitate data extraction by a receiver. Specifically, a receiver essentially only needs to monitor the packet identifiers to select the required data and can simply ignore all data packets that do not contain required data.

[0241] For example, a receiver may initially monitor received packets for the packet identifier of the content item descriptor of the data stream, that is, specifically, it may monitor the predetermined PID of PAT (PID 0 in the case of MPEG). .

[0242] When this is detected, the receiver extracts the data and selects the desired content item. The packet identifier for the content item map table, specifically the PID for PMT for MPEG, is then extracted for that content item.

[0243] The receiver then scans for data packets with that packet identifier. The data for these data packets is extracted, thereby providing the packet identifiers for the audiovisual data streams that comprise data for the content item. Then, the receiver can proceed to scan the data packets that contain these identifiers, ignoring all other data packets. The approach in particular enables practical and low-complexity multiplexing and de-multiplexing of audiovisual data.

[0244] In this way, the approach provides a low-complexity, low-resource approach to extracting relevant data from audiovisual content item data streams.

[0245] However, in some embodiments, the audiovisual content item data stream may be generated to enable multiple audiovisual data streams to be treated as a single audiovisual data stream. Specifically, instead of using standard package identifiers, the approach may use a combined or concatenated identification approach. Specifically, an audiovisual data stream may be associated with both a primary identifier and a secondary identifier, with the primary identifier being an identifier common to at least two of the audiovisual data streams.

[0246] The primary identifier may specifically be a common data stream packet identifier, that is, it may be a packet identifier that identifies a (combined) data stream. Because the primary identifier is common to a plurality of audiovisual data streams, the primary identifier can be considered to define a common or combined audiovisual data stream that is a combination of all audiovisual data streams that share the primary identifier.

[0247] In many embodiments, the primary identifier is included in the content item map table. In this way, the content item map table is generated to include the primary identifier, and thus the content item map table provides a pointer to a single data stream that is a combined data stream from a plurality of individual audiovisual data streams.

[0248] Indeed, in many embodiments, the apparatus of Figure 3 may be arranged to generate the content item map table to include only a single primary identifier for all audiovisual data streams comprising audio, or, indeed, for all audiovisual data streams, for the content item represented by the content item map table.

[0249] In fact, in some embodiments, the content item map table may be generated so as to include only an audiovisual data stream packet identifier for the content item in the content item map table, or to include only an audiovisual data stream packet identifier for audio data for the content item in the content item map table. In this way, for a given content item, the common bundle identifier represents all audio or audiovisual content for the content item, and in fact the content item can be processed based on the primary identifier as if it comprised only a single audiovisual data stream (or a single audiovisual data stream for audio).

[0250] As a specific example, for an MPEG-type transport stream, the PMT for a content item may be generated to include only one PID for all elementary audio streams, or, for all elementary audio streams, audio or video, for the content item. This way, the content item will only be bound to a single PID for all audio or for all audio and video.

[0251] Consequently, in some embodiments, the audiovisual data stream may appear, to processing devices such as receivers, multiplexers, de-multiplexers, etc., to have fewer audiovisual data streams than is actually the case. Specifically, the audiovisual data stream may appear to have only a single elementary audio or audiovisual stream for a content item. For example, for the MPEG-like example, the transport stream may appear to the processing circuitry as including only a single elementary stream for all audio (or audiovisual) components for a content item.

[0252] However, the audiovisual content item data stream may additionally comprise a secondary identifier for each individual audiovisual data stream that has been combined into the common audiovisual data stream. The secondary identifier is unique within the common audiovisual data stream, that is, two audiovisual data streams of a common audiovisual data stream will not have the secondary identifier. Thus, the secondary identifier enables the individual audiovisual data stream to be identified.

[0253] The secondary identifier may be, in many embodiments, a packet identifier. Thus, in many embodiments, individual packets may be represented by a common primary packet identifier and a common secondary packet identifier. Each packet may include a field in which the common primary packet identifier is comprised and a second field in which the individual secondary identifier is comprised.

[0254] In this way, packets can be processed by circuitry, such as de-multiplexers, routers, etc., as if the packets represented a single stream of audiovisual data. Specifically, by including the common PID in the PMT and the PID field of the data packets for an MPEG-type transport stream, the transport stream can be processed as if it comprised only a single elementary audio or audiovisual stream. The functionality may simply process, for example, all audio data packets for the content item in the same way, and as if there were only a single elementary stream. However, at the same time, appropriately equipped functionality is capable of accessing individual audiovisual data streams and processing them as separate audiovisual data streams. Specifically, such functionality may additionally evaluate the field comprising the secondary identifier and may accordingly select how to process the data packet. Specifically, it can decide whether to ignore or process the data packets, depending on whether the audiovisual data stream is necessary to provide the desired rendering of the content item.

[0255] In this way, when processing the audiovisual content item data stream, a receiver can detect the correct data packets for a given content item based on the primary/common packet identifier. Based on the information in the received content item map table, it is possible, in particular, to identify all data packets that belong to the common audiovisual data stream and thus to the individual audiovisual data streams that form the audiovisual data stream common. This way, audio data extraction is based on the primary identifier.

[0256] In embodiments that use such a two-part packet identifier approach, the transport stream may comprise additional data that provides information of individual audiovisual data streams, including information of corresponding secondary identifiers and, generally, of characteristics of the streams. individual audiovisual data associated with individual audiovisual data streams.

[0257] For example, data may be provided that, for a given primary identifier, lists all secondary identifiers of the audiovisual data streams that share the same primary identifier. The data may additionally indicate characteristics of the individual audiovisual data stream, such as whether it comprises mandatory audio data, replaceable audio data, additional audio data, alternative audio data, etc.

[0258] In some embodiments, such information may be provided within data packets of the common audiovisual data stream. For example, at regular intervals, a data package may be provided that includes all information regarding the secondary identifier. A receiver can initially detect and extract data from all data packets comprising the primary identifier. When a data packet with information regarding the use of the secondary identifier is received, this information can be extracted and used by the receiver to identify individual audiovisual data streams. Then, it may switch to also consider the secondary identifiers of the data packets that comprise the primary packet identifier and, specifically, to extract data only from data packets that have the desired secondary identifiers.

[0259] In other embodiments, the data may be provided outside of the ordinary audiovisual data stream, and in fact the information may be provided in the content item map table.

[0260] Specifically, for each common or primary identifier, the content item map table may be generated to include a plurality of secondary packet identifiers, where each secondary packet identifier identifies an individual audiovisual data stream of the data streams of the common audiovisual data stream.

[0261] For example, following the listing of the primary identifier, several optional fields may be included in the content item map table, where each optional field provides a secondary identifier for an individual audiovisual data stream with the common primary identifier. If the primary identifier represents a single audiovisual data stream that is not a combination of audiovisual data streams, no optional fields may be included. Each optional field may, in addition to the secondary packet identifier for an individual audiovisual data stream, also comprise metadata for the audiovisual data stream, such as data that specifically defines characteristics of the audiovisual data of the audiovisual data stream (such as, e.g., if it contains alternative audio data, mandatory audio data, replaceable audio data, or additional audio data).

[0262] Specifically, the content item map table may be modified or generated for each primary identifier to include a set of fields, with each field representing a stream of individual combined audiovisual data. Each field may comprise, for example, data representing:

[0263] - a secondary identifier for the individual audiovisual data stream, and

[0264] - descriptors for that individual audiovisual data stream.

[0265] For example, for an MPEG-like approach, the PMT may comprise a single PID for all audio for a content item. Following the PID in the PMT, there may be a set of fields that, for each individual audiovisual data stream, provide the secondary identifier and metadata that characterize the content of that individual audiovisual data stream.

[0266] In some embodiments, primary/common identifiers and secondary packet identifiers may be provided in a hierarchical arrangement. For example, a primary packet identifier may be provided in a content item map table at a first level that will be examined by a receiver looking for elementary streams. Secondary packet identifiers may be provided in optional fields that are linked to the primary packet identifier, such as by including a list of secondary packet identifiers for each primary packet identifier. In such embodiments, the receiver will therefore first scan the standard list of primary packet identifiers. When the desired combined elementary stream is identified, the receiver can proceed through the list of secondary packet identifiers that are stored in the identified primary packet identifier. In this way, secondary package identifiers are identified in dependence on the corresponding primary package identifier.

[0267] Thus, in some embodiments, a telescoping structure of the primary and secondary packet identifiers may be used. For example, a PMT may include a list of PIDs in a set of dedicated fields, and the list may include, for example, only a single primary PID for a combined elementary stream. However, the PMT may further comprise a second list of secondary PIDs for the individual elementary streams of the combined elementary stream. These secondary PIDs are provided in dedicated data fields for secondary PIDs and are linked to the primary PID. This way, the receiver will only access secondary PIDs based on the primary PID, and in fact, the receiver will differentiate between primary PIDs and secondary PIDs.

[0268] However, in other embodiments, a flat structure of primary and secondary packet identifiers may be used. In fact, a content item map table can comprise a list of identifiers that can contain both primary and secondary identifiers. In fact, the content item map table can include a simple list of all possible audiovisual data streams, including both combined audiovisual data streams and individual audiovisual data streams. Furthermore, a given individual audiovisual data stream may be represented by the secondary packet identifier for the individual audiovisual data stream and may also be represented by the primary packet identifier (i.e., selection of either packet identifier will result in in selecting the audiovisual data stream). In fact, for each individual audiovisual data stream, the content item map table may include an entry for the primary packet identifier of any combined audiovisual data stream to which the individual audiovisual data stream belongs, as well as the identifier of secondary packet for the individual flow. Thus, in this example, a list may comprise, for example, an entry with a packet identifier for all audiovisual data streams, including both combined and individual audiovisual data streams.

[0269] Accordingly, in some embodiments, the secondary packet identifiers may be encapsulated in data linked to a primary packet identifier, while in other embodiments, the secondary packet identifier may be provided in parallel with the primary packet identifiers. A primary packet identifier may therefore be any identifier that provides a packet identifier for a combined audiovisual data stream comprising a plurality of audiovisual data streams. A secondary packet identifier may be any packet identifier for a stream of individual audiovisual data and need not be encapsulated, embedded or dependent on a primary packet identifier.

[0270] For example, rather than embedding secondary packet identifier information in data for a simple primary packet identifier, the content item map table can be generated to include all possible packet identifiers on a link /sequence. In such embodiments, the receiver may simply select the packet identifier that corresponds to the specific data that is required and may choose, for example, to select a single combined audiovisual data stream or an individual audiovisual data stream simply by selecting the identifier. of suitable package from the list.

[0271] In such embodiments, the receiver 1401 may be specifically arranged to select the desired audio by selecting the required packet identifiers, rather than merely transmitting all audio information in a single combined stream.

[0272] The approach may combine properties and characteristics of having a single audiovisual data stream with those of having a plurality of audiovisual data streams in a transport stream.

[0273] This can provide a significant advantage, as a batch of equipment in the field is not capable of reliably handling multiple streams of audiovisual data.

[0274] Figure 14 illustrates an example of a receiving apparatus comprising a receiver 1401 coupled to an audio decoder 1403. In the example, the receiver 1401 receives the audiovisual content item data stream and is capable of processing it to extract, for example, multiple audiovisual data streams. In this way, receiver 1401 may receive a transport stream and may be arranged to process this to generate, for example, individual elementary audio and video data streams. Receiver 1401 may specifically correspond directly to receiver 1401 of Figure 4.

[0275] The audio decoder 1403 is arranged to receive one or more data streams, comprising audio data, and to generate an output audio signal from that audio data. The audio decoder 1403 may specifically comprise functionality for receiving audio data from the audiovisual data streams of the content item data stream. As such, the audio decoder 1403 may specifically comprise the functionality of extracting obligatory audio data from an elementary stream, to recover alternative audio data from the same or another elementary stream, to replace received substitutable audio data with the alternative data, and to generate the audio signal from the resulting audio data.

[0276] Specifically, the audio decoder 1403 may comprise the remaining blocks of Figure 4 and may specifically comprise the data extractors 403, 405, 407, the decoding units 409, 411, 413 and the replacer/combiner 415.

[0277] In the example of Figure 14, the audio decoder 1403 may be provided with different audiovisual data streams/elementary data streams, where each stream is an audio stream comprising different audio data. However, in many embodiments, receiver 1401 may be arranged to generate a single data stream comprising all audio data, and feed that data stream to audio decoder 1403. Receiver 1401 may combine, e.g. the elementary audio streams into a single combined audio stream comprising different elementary streams. In some scenarios, the single combined audio stream may further comprise individual audiovisual data streams, for example, the individual audiovisual data streams may be encapsulated into a single data stream. In other scenarios, receiver 1401 may extract, for example, relevant audio data and combine it into an audio data stream without including all aspects of the input audiovisual data streams. For example, receiver 1401 may generate a new data stream that comprises different control data, syntax, and/or structure for the data stream. In some cases, audio data can be combined into a new audiovisual data stream, that is, a new elementary stream. This audiovisual data stream/elemental stream combination may comprise the original elementary stream of the received transport stream or may include some or all of the audio data in an independent elementary stream. For the sake of brevity, the following description will refer primarily to transport streams and elementary streams, rather than audiovisual content item audio data streams and audiovisual data streams. Furthermore, the description will refer to a scenario where the receiver 1401 generates a single elementary stream to the audio decoder 1403.

[0278] In this way, the audio decoder 1403 will receive an elementary audio stream and generate the output audio stream.

[0279] In many practical implementations, a receiver and decoder are relatively independent and separate functional blocks, rather than closely integrated functions. In fact, in many scenarios, the two entities can be designated and developed independently and separately. For example, for many content rendering devices, audio decoders are generic decoders that are reused for many different devices. For example, many content rendering device manufacturers may purchase audio decoder functions rather than developing them in-house.

[0280] Consequently, the interface between the receiver 1401 and the audio decoder 1403 is of critical importance in many scenarios and it is desirable to use a simple and standardized, or at least common, approach rather than relying on, for example, interfacing and control dedicated and private.

[0281] Traditionally, an audio decoder has been considered to be in the structure of a relatively simple device that receives a stream of bits that is decoded according to the decoding pattern implemented by the audio decoder and without any other modification or interaction by the external circuitry . Such an approach can be used, for example, in the embodiment of Figure 4, where each of the decoding units 409 to 413 is fed with a simple elementary stream, from which it generates an audio signal that can be subsequently combined , by combiner 415, according to the specific audio preferences for the specific rendering, and specifically the output signal may be generated based on the substitutable audio data or alternative data, as required, for example, by the specific user.

[0282] In some embodiments, the apparatus may use a more complex audio decoder 1403 that may specifically receive one or more elementary audio streams and, from there, generate the output audio by manipulating, for example, the data before decoding the audio signal. For example, the audio decoder 1403 may, in such embodiments, select the appropriate data to be used before performing the actual decoding. Thus, audio data that is not used may not be decoded and, in fact, may not even be extracted from the data stream(s) provided to audio decoder 1403. For example, the data replaceable audio files that are replaced with alternative data may not be decoded or even extracted. Similarly, alternative audio data that is not used, or alternative audio data that is not used to replace replaceable audio data, may not be decoded or extracted.

[0283] Thus, in some embodiments, replacement of substitutable audio data with alternative audio data may not be accomplished by adapting a combination of decoded audio signals, but instead may be accomplished by selecting alternative audio data to decode, rather than selecting replaceable audio data. In this case, the output signal may be generated, for example, by decoding only the mandatory audio data and alternative audio data, and summing the resulting audio signals.

[0284] Such an approach can, in many scenarios, provide more efficient operation. However, it requires the decoder to be able to adapt to the specific requirements and preferences of the individual scenario. Furthermore, it requires the decoder to be able to process a plurality of different audio components and adapt to them in a flexible manner. Specifically, instead of the audio decoder merely rendering all received audio data, it needs to be able to flexibly combine different audio objects and, specifically, replace some audio objects with other objects.

[0285] However, in order to achieve this, it is necessary for the receiver to instruct the audio decoder 1403 on how to process the audio data provided thereto. One possibility is to implement a unique interface in which the receiver 1401 is arranged to generate multiple control signals that control the audio decoder 1403 to perform the necessary operations. However, such an approach tends to result in complex modalities and requires the use of a dedicated and exclusive decoder.

[0286] In the system of Figure 14, the audio decoder 1403 is arranged to be controlled by the receiver 1401 generating an audiovisual data stream that includes audio data, and that additionally includes an audio data processing descriptor that indicate how different audio data should be processed by audio decoder 1403. In the example, receiver 1401 specifically generates a single audiovisual data stream comprising all audio data that can potentially be used by audio decoder 1403 and possibly all audio data comprised in the input transport stream for the given content item. Furthermore, the same audiovisual data stream comprises decoding/processing control data that enables the audio decoder 1403 to adapt the decoding/processing and, specifically, the combination of audio objects to specific preferences for such rendering. Thus, in the system of Figure 14, the receiver 1401 adds a control descriptor in the form of an audio data processing descriptor to the elementary audio stream fed into the audiovisual data stream, and the audio decoder 1403 interprets this audio descriptor. audio data processing to control your decoding task.

[0287] In more detail, in the example of Figure 14, the receiver 1401 receives a transport stream comprising a plurality of elementary streams that includes multiple elementary audio streams for a given content item. Receiver 1401 extracts one or more of the elementary audio streams and generates one or more elementary output streams from that extraction. Specifically, based on the first audiovisual data stream, which may specifically be a first elementary stream comprising mandatory audio data and substitutable audio data for a first content item, the receiver 1401 may generate a second audiovisual data stream that It may specifically be a second elementary stream comprising the mandatory audio data and the replaceable audio data of the first audiovisual data stream. Furthermore, the second audiovisual data stream/elemental stream may comprise audio data from another input elementary stream, such as from an elementary stream comprising alternative audio data.

[0288] Additionally, the receiver 1401 generates an audio data processing descriptor that is indicative of how the audio decoder 1403 should generate the output audio signal from the audio data of the second elementary stream. The audio data processing descriptor is included in the second elementary stream. For example, the second elementary stream may be generated to comprise one or more fields comprising data describing how individual audio components are to be processed, and specifically, how (if) they are to be decoded and combined with other audio components. .

[0289] Then, the receiver 1401 feeds the second elementary stream into the audio decoder 1403. The audio decoder 1403 proceeds by extracting the audio data processing descriptor from the second audiovisual data stream and generating an audio signal in response. to the audio data processing descriptor.

[0290] The audio data processing descriptor may define which audio components should be combined with each other to generate the output signal, and accordingly, the audio decoder 1403 may specifically recover (only) the indicated audio components , decode them and finally add them up.

[0291] In some embodiments, the receiver 1401 may include a plurality of audio data processing descriptors, each of the audio data processing descriptors processing descriptors that provide processing indications to a subset of the audio components of the second elementary flow. In some embodiments, an audio data processing descriptor may be provided for each audio component, wherein the audio data processing descriptor specifically describes whether the corresponding audio component is mandatory audio data, replaceable audio data, or alternative audio data. Audio data processing descriptors may additionally describe whether or not an alternative audio component should actually replace a given replaceable audio component.

[0292] As a specific example, an audio data processing descriptor may be provided for a given audio component to indicate whether it should be treated as alternative audio data or additional data. Alternative audio data is the data that will replace some replaceable audio data, while additional data is the data that is added to the replaceable audio data.

[0293] For example, the input stream at receiver 1401 may comprise an elementary stream comprising first audio data for a given first audio component. In some rendering situations, you may want the first audio component to replace a second audio component of the input transport stream. Thus, in this case, the first audio data represents alternative audio data for an alternative audio component that is rendered instead of the second audio component, which is a replaceable audio component. However, in other scenarios, it may be desirable for the first audio component to be rendered together with the second audio component, that is, for both audio components to be rendered together.

[0294] In the first case, a data processing descriptor may be included in the second elementary stream by receiver 1401 to indicate whether the first audio data is, in fact, alternative audio data that should replace the second audio data. However, in the latter case, an audio data processing descriptor may be included by receiver 1401 to indicate that the first audio data is instead additional audio data that is to be added to the second audio data, i.e. , that both audio components must be extracted and rendered. Therefore, in this case, an audio data processing descriptor is inserted for the first audio data indicating whether it is designated as alternative audio data or additional audio data.

[0295] The audio decoder 1403 will process the data according to the received audio data processing descriptors. In the specific example, if the audio data processing descriptor indicates that the first audio data is designated as alternate audio data, the audio decoder 1403 proceeds to extract the first audio data and treat it as alternate audio data. . Furthermore, it replaces the substitutable audio data, in the form of the second audio data, with this extracted first audio data. Substitution can be achieved specifically by decoding only the first audio data, while the second audio data is neither extracted nor decoded. Therefore, the first audio component is generated and included in the output audio signal.

[0296] However, if the audio data processing descriptor indicates that the first audio data is designated as additional audio data, the audio decoder 1403 proceeds to extract both the first audio data and the second audio data. Furthermore, both the first audio data and the second audio data are decoded to generate the first audio component and the second audio component. Both are then combined into the output signal.

[0297] In such embodiments, the receiver 1401 may therefore control the operation of the audio decoder 1403, and specifically may control which audio data contributes to the output signal and which is ignored. The approach enables low complexity control and may enable an audio decoder 1403 to require only one input data stream to generate the desired audio output.

[0298] The approach may further enable, for example, all audio data from the transport stream (for a given content item) to be included in the second audiovisual data stream provided to the audio decoder 1403, while at the same time , enables this data to be used flexibly, and while still specifically enabling an output signal to be generated that includes only a subset of the audio components. Audio components may be flexibly selected by receiver 1401 via data included in the second audiovisual data stream itself.

[0299] Receiver 1401 may generate, for example, an audio data processing descriptor that is indicative of whether or not audio data is audio data among the group of: mandatory audio data; replaceable audio data; alternative audio data; and additional audio data;

[0300] In some embodiments, a subset or superset of this group may be used.

[0301] The audio data processing descriptor for a given content item may be generated, for example, in response to a rendering preference for the specific device, or, for example, in response to user-entered data.

[0302] As an example, a received transport stream may comprise a mandatory background audio component, an English language audio component, a German background audio component, and a secondary background component. The user can select which audio components are desired and audio data processing descriptors can be generated to reflect the selected audio components so that they are extracted and decoded by the audio decoder 1403.

[0303] In some embodiments, audio data processing descriptors may be generated by receiver 1401 without relying on any predetermined audio data processing descriptor, and specifically, without relying on any audio data processing descriptor of the stream of transportation received.

[0304] However, in other embodiments, audio data processing descriptors of the second elementary stream are generated from audio data processing descriptors as part of the data stream of the received audiovisual content item, i.e., as part of the received transport flow.

[0305] Specifically, receiver 1401 may be arranged to generate a first audio data processing descriptor from a second audio data processing descriptor received in the transport stream. Thus, for a given audio elementary stream of the input transport stream, the transport stream may include one or more audio data processing descriptors that indicate how the audio data of the audio elementary stream should be decoded by the decoder. audio 1403.

[0306] For example, a given elementary audio stream may comprise alternative audio data that is intended to replace replaceable audio data. For example, the elementary flow may comprise German dialogue that should replace the default dialogue audio component, such as an English dialogue audio component. Accordingly, the input transport stream may include an audio data processing descriptor that defines that such audio data is to be treated as replaceable audio data and that it is to replace other dialog audio data. The transport stream may include other elementary dialog audio streams, and for each of these, the transport stream may also include an audio data processing descriptor.

[0307] Thus, in this example, the device that generates the transport stream, such as, for example, the device in Figure 3, can also be arranged to generate audio data processing descriptors that indicate how the corresponding audio data must be, or can be, processed. For example, an audio data processing descriptor may be included for each dialog audio component that indicates that that track can be selected and that, if successful, it should override all other dialog audio components.

[0308] Audio data processing descriptors are included in the transport stream, for example, in the program map for the content item, or possibly in the individual data packets for the individual elementary stream.

[0309] In particular, in many embodiments, the audio data processing descriptor may, on the transport stream generation side, be included in the program map table for the content item. As a specific example, for each elementary audio stream, the program map table may include not only the packet identifier for the elementary stream but also one or more audio data processing descriptors that indicate how the audio data should be processed. be decoded. This can enable the source to control and describe how content can and/or should be decoded and rendered.

[0310] The receiver 1401 may, upon receiving such a transport stream, generate the second elementary stream and feed it to the audio decoder 1403, the second elementary stream comprising audio data processing descriptors that are generated from those of the input transport stream and that describe how specific audio data should be decoded and rendered in the specific situation.

[0311] The audio data processing descriptors of the second elementary stream are generated from the received audio data processing descriptors. This process may, in some embodiments, be a complex process that evaluates received audio data processing descriptors and then generates new audio data processing descriptors that reflect specific processing.

[0312] However, in many embodiments, the receiver 1401 may be arranged to simply copy the audio data processing descriptors from the transport stream to the second elementary stream.

[0313] Specifically, audio data processing descriptors from the input stream that describe a processing or specifically what audio data is desired for rendering may be copied from the input transport stream to the second stream elementary.

[0314] For example, if the audio data is a given audio component or elementary audio stream that one wishes to render, the receiver 1401 may simply copy the audio data processing descriptor for that audio component or elementary audio stream. audio for the second elementary stream. For example, if the German dialogue audio track is desired to be rendered, the receiver 1401 may copy the audio data processing descriptor for the elementary German dialogue stream (along with the audio data) to the second elementary flow. Accordingly, the second elementary stream will contain an audio data processing descriptor that causes the audio decoder 1403 to replace the substitutable audio data of the English dialogue with that of the German dialogue.

[0315] If it is not necessary for the audio data to be rendered, for example, if the user prefers English dialogue, the receiver 1401 may not copy the audio data processing descriptor for the elementary German dialogue stream to the second elementary flow. In such embodiments, the audio decoder 1403 may be arranged, for example, to ignore or discard audio data for which no audio data processing descriptor is included in the elementary stream input to the audio decoder 1403.

[0316] In some embodiments, the source may be arranged to generate a transport stream comprising a plurality of possible audio data processing descriptors for a given elementary audio stream. The plurality of possible audio data processing descriptors can describe, for example, the different possible ways the elementary stream audio data can be used. For example, if the audio component can be rendered either alongside or instead of another audio component, the transport stream may comprise an audio data processing descriptor indicating that the data is alternative audio data and an audio data processing descriptor indicating that the data is additional audio data. In this way, each of the possible audio data processing descriptors can describe a permitted processing of the corresponding audio data.

[0317] In such embodiments, the receiver 1401 may extract, for example, the plurality of possible audio data processing descriptors for the audio data. Based on the desired rendering (e.g., based on a user's language selections), the receiver 1401 may proceed to select one of the audio data processing descriptors that corresponds to the desired rendering. This audio data processing descriptor can then be copied to the second elementary stream, while the other audio data processing descriptors can be ignored.

[0318] In the described example, the audio data processing descriptor can be considered to correspond to the use of a token, found in the input transport stream, to be injected into the data stream for the audio decoder 1403, and used as a command to do so. The audio data processing descriptor may comprise an indication of the required processing of the audio data by the audio decoder 1403.

[0319] This approach can result in a more generic and open way of implementing an interface in an audio decoder 1403. It can also enable low-complexity processing. As a specific significant advantage, the receiver 1401 simply needs to select which rendering is desired and then copy the corresponding audio data processing descriptor from the input transport stream to the second elementary stream. It is not necessary to consider, or indeed have any knowledge of, how the actual processing is performed by the audio decoder 1403. Instead, the audio decoder 1403 interprets and performs the processing defined by the data processing descriptor audio.

[0320] This is very advantageous, as the alternative of interfacing with the audio decoder 1403 requires that the receiver 1401 understands not only the choice but also the concept of “substitution”, namely that the objects in this stream need to replace objects in the main flow. A receiver 1401 that only passes along copies of audio data processing descriptors has only to understand that a selection has been made. This implies that the action described by the audio data processing descriptors can be arbitrarily complex, and constantly changes and improves over time, because all the receiver 1401 has to do is copy the audio data processing descriptors relevant. The receiver 1401 does not need to understand what unique commands will be required to perform the complex action coupled with including a given stream in the total stream.

[0321] Specific examples of an operation using audio data processing descriptors are provided in Figures 15 and 16 for a (modified) MPEG transport stream. In the examples, the audio data processing descriptors are referred to as “3D AD” and “3D Audio Descriptor”. In the examples, the second audiovisual data stream/elementary stream is generated as an MPEG-H Audio Stream (MHAS).

[0322] The foregoing description provides a series of examples of approaches that address the efficient distribution of audio in audiovisual content item data streams, and specifically the distribution of audio content in such data streams.

[0323] When deciding on an approach to distributing audiovisual content, the designer is faced with many different and often conflicting technical challenges. Specifically, you want the data rate to be as low as possible while maintaining quality as high as possible. Another “compromise” is between the desire to provide a data representation that is very simple for a receiver to process and render, but that still provides a high degree of flexibility for the receiver to customize the rendering of the data. To sustain such flexibility, an increasingly complex data flow structure and organization may be required, but this also tends to require more complex processing on the receiver side.

[0324] Thus, the question of how to divide, structure, and organize data in audiovisual data streams is a critical and difficult technical problem that requires several technical trade-offs.

[0325] The previously described approaches address this issue and seek to provide an improved trade-off between different characteristics, requirements, and preferences for various applications.

[0326] The approaches utilize a more flexible approach than traditional MPEG data streams, with all audio for an audio representation generally being provided in a single elementary stream. In fact, a conventional MPEG data stream is organized as a transport stream comprising a plurality of elementary streams, where each elementary data stream comprises a video or audio component. Data in MPEG transport streams is structured into elementary streams and two possible elementary audio streams are defined. One is an elementary data stream that provides a complete audio representation for a content item, that is, an audio component corresponding to the entire sound platform is provided in a single elementary stream. The other option is for an elementary audio stream to include audio data for an additional sound component that can be combined with/added to a full audio component (full representation).

[0327] The MPEG approach has the significant advantage of being low complexity and content can be rendered based on low complexity processing. In fact, identifying and extracting the audio data required for a given content item generally requires only that an elementary data stream be identified and extracted from the transport stream, and generally only requires that one audio component be decoded.

[0328] MPEG provides some flexibility by enabling audio to be supplemented by additional audio components. These additional audio components are additional audio components that can be ignored or can be combined with/added to a complete audio component. However, no other option is possible, that is, audio components are additional audio components that cannot replace the audio data for the complete representation.

[0329] Thus, for a conventional MPEG transport stream, the audio data is either designated as complete audio data, provided individually and separately in a single elementary data stream, or is designated as additional audio data, also provided in a separate elementary stream. Consequently, each elementary stream provides an audio data type that can either be rendered directly or added to a complete audio component.

[0330] The MPEG approach provides a data stream with a compact data representation and low data overhead that enables low complexity processing by a receiver. In fact, the receiver only needs to consider a PID (Packet IDentifier) of data packets to find and extract only the audio data that is needed, that is, there is a direct correspondence between the audio data type and the stream ID elementary. However, a drawback of the approach lies in that the audio rendering flexibility is low, and is essentially limited to the possibility of optionally adding additional audios to another form of complete audio representation.

[0331] In recent years, it has been proposed to introduce audio objects. Rather than providing one audio component that corresponds to the entire sound platform/scene, the approach is based on instead providing a large number of individual audio objects that each correspond to a single sound source in the audio scene. Consequently, the final renderer can create a given sound scene by individually decoding each audio object, positioning the object in the sound scene, generating the appropriate spatial audio signals, combining the audio objects, and finally rendering the combined signals.

[0332] Such an approach provides a high degree of flexibility to the rendered audio and provides the receiver with a high degree of freedom. However, it requires very complex processing at the receiver. In fact, the processing required to render audio data is complex, and furthermore, the complexity and processing required to identify and extract the necessary and desired audio data, as well as determining how the audio data can be used in rendering, are very complex. The approach also results in a large data overhead in order to provide the information needed to identify the data distribution in the data stream.

[0333] The previously described approaches are intended to provide a different, and, for many scenarios, improved trade-off between complexity, data overhead, and flexibility in distributing content items, relative to the approaches provided by a conventional MPEG transport stream or an audio object transport stream.

[0334] The approaches are based on an innovative data stream that comprises different audio data types than those in the conventional transport stream, and that has a different organization of that data in the data stream.

[0335] Similar to traditional MPEG approaches, the approaches are based on an audiovisual content item data stream (transport stream) comprising a plurality of individual audiovisual data streams comprising audiovisual components (called elementary streams) to a audiovisual content item.

[0336] However, the approaches are based on elementary data streams that can each provide sufficient audio data for an audio representation for the content, but that comprise different types of audio data. In this way, the transport stream is arranged to include an elementary stream comprising different types of audio data. In fact, some of the audio data in this elementary stream is designed to be mandatory audio data for audio representation. Mandatory audio data needs to be rendered when the audio representation is presented. Other data in the elementary data stream is designated as replaceable audio data. This replaceable audio data is audio data for the audio representation, but it is audio data that the receiver is allowed to replace with other (alternative) data.

[0337] Thus, in the approaches, a single elementary stream with different data types is provided for a given audio representation, namely, with some data that is designated as mandatory audio data that needs to be rendered when the audio representation is displayed, and with other data that is referred to as replaceable audio data that can be replaced/or replaced by other data.

[0338] Accordingly, in the approaches, a transport stream provides an audio representation in which different types of audio data for the audio representation are provided within a single elementary data stream. Furthermore, data is specifically designated as data that needs to be presented/rendered (when the corresponding audio representation is presented/rendered) or data that can be exchanged/replaced with other data. Replaceable audio data is data that is designated as being exchangeable for other data, but cannot be erased without being overwritten.

[0339] This particular approach provides a data flow structure that is substantially more efficient and provides a better trade-off between data rate, complexity, and flexibility in many scenarios and applications. Additionally, it can provide these advantages while providing greater backwards compatibility with prior art approaches, such as MPEG transport streams.

[0340] Specifically, the approaches can provide a bit stream that does not require much overhead, yet provides a flexible approach. A low-complexity receiver can identify and extract data based on how the data is structured into elementary streams. However, it can also increase flexibility by designating that different audio data from an elementary stream have different characteristics with regard to audio representation.

[0341] The approaches may enable, for example, a low-complexity legacy receiver to still be able to render the audio representation of the new data stream based merely on elementary stream identification. For example, in many embodiments, an MPEG decoder may simply extract the first elementary stream and treat it as a standard MPEG elementary stream. This may result in the generation of a nominal audio presentation. However, in addition to such backward compatibility, a non-legacy receiver may process the received data stream and exchange the substitutable audio data for alternative data, thereby generating an alternative audio presentation. This can still be achieved with low complexity and does not require the overhead and complex processing associated with, for example, audio objects.

[0342] The approaches provide an advantageous solution to the technical problem of how to best distribute audio content to audiovisual content items, and specifically provide a data flow (transport) and data content structure that provides an improved trade-off between the rate of data, complexity and flexibility, and which can also provide improved backwards compatibility.

[0343] Mandatory audio data is data that is designated (and in many embodiments indicated) to be data that needs to be included in a rendering of the audio representation/audio scene. Therefore, the required audio data for an audio presentation is data that is included in the elementary data stream and that needs to be rendered if the audio representation/audio scene of the elementary data stream is selected. Thus, mandatory audio data is data that is designated to be compulsory, required, required, etc., for the rendering of the audio representation/audio scene associated with the elementary data stream. In many embodiments, the required audio data and the replaceable audio data may be comprised of an elementary data stream associated with only one audio representation/audio scene.

[0344] Mandatory audio data may be designated/indicated to be mandatory audio data, for example, through position in the elementary stream and/or by control data describing the data that will be mandatory. Thus, in some embodiments, the transport data stream, and possibly specifically the elementary data stream, may include data indicating which audio data of the elementary data stream is required audio data.

[0345] Replaceable data is data that is designed (and in many embodiments, indicated) to be data that can be included in a rendering of the audio representation/audio scene or that can be replaced by other audio data in the rendering of the audio representation/audio scene. Thus, substitutable audio data for an audio presentation is data that is included in the elementary data stream and that needs to be rendered if the audio representation/audio scene of the elementary data stream is selected, unless overridden by other audio data. Replaceable audio data is data that can be rendered or that can be swapped/replaced/exchanged with alternative data when rendering/presenting the audio representation/audio scene associated with the elementary data stream.

[0346] In the approaches described, content distribution is consequently based on a stream (transport) that includes a plurality of (elementary) data sub-streams, whereby an audio representation can be described by a single stream elemental data stream that comprises both data designated as mandatory data and data designated as replaceable audio data, that is, a single elementary data stream includes both audio data that a receiver needs to present when presenting the audio representation, and audio that the receiver needs to exchange for other data.

[0347] In this way, the receiver receives audio data that it knows needs to be rendered, as well as data that it knows it can exchange for other data. Furthermore, it receives this data in a single elementary stream, thus making a presentation of the audio representation possible simply by extracting the audio data from an elementary data stream and rendering it (thus providing backward compatibility).

[0348] In this way, the transport stream comprises data that is associated with specific constraints on how it needs to be used by a receiver/renderer. This specific structure of different types of data that are designed to be used differently, and that are associated with different restrictions on their use, can provide the generation of an efficient transport flow with improved tradeoffs, and, specifically, that still allows low complexity processing and backward compatibility while providing greater flexibility.

[0349] In fact, designating audio data for an audio representation within an elementary data stream as mandatory or replaceable audio data provides substantial benefits. The described structure, content and organization of the audiovisual content item data stream provides a transport stream that can be processed with low complexity, can be backwards compatible, providing even greater flexibility.

[0350] Consequently, the approach is not focused on deriving, determining or selecting any specific mixing or audio policy, but rather addresses the technical solution that can support the effective distribution of audio data so that one can comply with the guidelines. The approach enables the use of data streams that enable flexible (but not completely free) guidance to be distributed while maintaining low data overhead and reduced complexity. The approaches do not focus on how to select a mixing strategy, but on how to technically support the distribution of audio data. The approach enables some aspects of the rendering orientation to be determined by the source (e.g., the required audio data), while other aspects can be adapted at the receiving/rendering end (e.g., the receiver can select between different components alternative audio tracks).

[0351] It should also be noted that it is not the actual processing at the receiver that defines the data. Rather, it is the data in the data stream that is designated as mandatory or substitutable for an audio representation. Consequently, the data is defined by its intended use, i.e., mandatory audio data is the data that must be presented if the audio representation is selected, while substitutable audio data can be exchanged for other alternative data (but cannot be merely deleted).

[0352] It should be considered that, for the sake of clarity, the above description described embodiments of the invention with reference to different functional circuits, units and processors. However, it will be evident that any suitable distribution of functionality between the different functional circuits, units or processors can be used without departing from the invention. For example, functionality illustrated to be performed by separate processors or controllers may be performed by the same processor or controllers. Therefore, references to specific functional units or circuits should be considered only as references to means suitable for providing the described functionality and not as indicative of a physical structure or strict logical or physical organization.

[0353] The invention may be implemented in any suitable form, including hardware, software, firmware or any combination thereof. The invention may optionally be implemented, at least partially, as computer software that runs on one or more data processors and/or digital signal processors. The elements and components of an embodiment of the invention may be physically, functionally and logically implemented in any suitable manner. In fact, functionality can be implemented in a single unit, in a plurality of units, or as part of other functional units. Thus, the invention can be implemented in a single unit or can be physically and functionally distributed among different units, circuits and processors.

[0354] Although the present invention has been described in relation to some embodiments, it is not intended to limit it to the specific form presented here. Instead, the scope of the present invention is limited only by the appended claims. Additionally, although it may appear that a feature is described in relation to specific embodiments, one skilled in the art will recognize that various features of the described embodiments may be combined in accordance with the invention. In the claims, the term “comprising” does not exclude the presence of other elements or steps.

[0355] Furthermore, although individually mentioned, a plurality of means, elements, circuits or method steps may be implemented, for example, by means of a single circuit, unit or processor. Additionally, although individual features may be included in different claims, they may be advantageously combined, and their inclusion in different claims does not imply that a combination of features is not viable and/or advantageous. Furthermore, the inclusion of a feature in one category of claims does not imply a limitation to that category, but rather indicates that the feature is equally applicable to other categories of claims, as appropriate. Furthermore, the order of the features in the claims does not imply any specific order in which the features need to be worked on, and in particular, the order of the individual steps in a method claim does not imply that the steps need to be performed in that order. Rather, the steps can be performed in any suitable order. Furthermore, singular references do not exclude a plurality. Thus, references to “one”, “one”, “first”, “second”, etc., do not exclude a plurality. Reference marks in the claims are provided merely as clarifying examples and should not be construed as limiting the scope of the claims in any way.

Claims (12)

1. METHOD OF GENERATING AN AUDIO SIGNAL FOR AN AUDIOVISUAL CONTENT ITEM, characterized by comprising: receiving a data stream from the audiovisual content item comprising a plurality of individual audiovisual data streams, the plurality of streams Individual audiovisual data (streams) comprise audiovisual components for an audiovisual content item; wherein each audiovisual data stream comprises encoded data representing at least one audiovisual component and synchronization data; wherein the data stream of the audiovisual content item comprises a first individual audiovisual data stream comprising first audio data for an audio representation for the audiovisual content item, wherein the first audio data comprises audio data required for the audio representation and substitutable audio data for the audio representation, the mandatory audio data corresponding to an audio component that must be rendered when the audio representation is rendered, and the substitutable audio data corresponding to a substitutable audio component that may be replaced by an alternative audio component when the audio representation is rendered, the content item data stream further comprising a content item map table for the audiovisual content item, the content item map table comprising a list of audiovisual data stream packet identifiers for data packets for audiovisual data streams linked to the audiovisual content item, and the content item map table comprising a data stream packet identifier common to the audiovisual content item for a set of at least two of the plurality of individual audiovisual data streams; wherein the method further comprises: extracting the required audio data from the first audio data in response to the common data stream packet identifier; recover alternative audio data; replace the substitutable audio data with alternative audio data; generating the audio signal from the required audio data and alternative audio data; wherein the method further comprises: generating a new individual audiovisual data stream from the plurality of individual audiovisual data streams; generate an audio data processing descriptor for second audio data from at least one of the audiovisual data streams, insert the audio data processing descriptor into the new individual audiovisual data stream, and feed the new individual audiovisual data stream to an audio decoder; and wherein generating the audio signal comprises the audio decoder generating the audio signal from audio data of the new individual audiovisual data stream, extracting the audio data processing descriptor from the new individual audiovisual data stream ; and generating the audio signal in response to the audio data processing descriptor. 2. METHOD, according to claim 1, characterized by additionally comprising receiving a second data stream of the audiovisual content item comprising a plurality of audiovisual data streams, wherein the plurality of audiovisual data streams comprises audiovisual components; and that the recovery comprises recovering the alternative audio data of an audiovisual data stream from the second data stream of the audiovisual content item. 3. METHOD, according to claim 1, characterized by the step of generating the audio signal comprising: decoding the mandatory audio data in a first audio decoder to generate a first decoded audio signal; decoding the alternative audio data in a second audio decoder to generate a second decoded audio signal; and generating the audio signal in response to the combination of the at least first decoded audio signal and the second decoded audio signal. 4. METHOD, according to claim 1, characterized in that the content item map table comprises a plurality of secondary packet identifiers, each secondary packet identifier identifying an audiovisual data stream from the set of at least two among the plurality of individual audiovisual data streams; and by the step of extracting audio data for the content item comprising extracting audio data for different audio components in response to the secondary packet identifiers. 5. APPARATUS FOR GENERATING AN AUDIO SIGNAL FOR AN AUDIOVISUAL CONTENT ITEM, characterized by comprising: a receiver for receiving a data stream from the audiovisual content item comprising a plurality of individual audiovisual data streams, the plurality of streams ( streams) of individual audiovisual data comprises audiovisual components for an audiovisual content item; wherein each audiovisual data stream comprises encoded data representing at least one audiovisual component and synchronization data; wherein the content item data stream comprises a first audiovisual data stream comprising first audio data for an audio representation for the audiovisual content item, the first audio data comprising audio data required for the representation of audio and substitutable audio data for the audio representation, the mandatory audio data corresponding to an audio component that must be rendered when the audio representation is rendered, and the substitutable audio data corresponding to a substitutable audio component that can be replaced by an alternate audio component when the audio representation is rendered, the audiovisual content item data stream further comprising a content item map table for the audiovisual content item, the content item map table comprising a list of audiovisual data stream packet identifiers for data packets for audiovisual data streams linked to the audiovisual content item, and the content item map table comprising a common data stream packet identifier for the content item audiovisual for a set of at least two of the plurality of individual audiovisual data streams; wherein the apparatus further comprises: an extractor (403) for extracting the required audio data from the first audio data in response to the common data stream packet identifier; a retriever (407) for retrieving alternative audio data; a replacer (415) for replacing the replaceable audio data with the alternative audio data; an output (415) for generating the audio signal from the mandatory audio data and alternative audio data; an audio decoder for generating the audio signal from the audio data of a new individual audiovisual data stream, the audio decoder comprising the extractor (403), the recoverer (407), the replacer (415) and the output; and wherein the receiver (1401) is arranged to generate the new individual audiovisual data stream from the plurality of new individual audiovisual data streams and to generate an audio data processing descriptor for the second audio data of at least one of the audiovisual data streams and to insert the audio data processing descriptor into the new individual audiovisual data stream and to feed the new individual audiovisual data stream to the audio decoder; and the audio decoder (1403) further comprises an audio data processing descriptor extractor for extracting the audio data processing descriptor from the second audiovisual data stream; and the audio decoder (1403) is arranged to generate the audio signal in response to the audio data processing descriptor. 6. APPARATUS according to claim 5, characterized in that it comprises an audio decoder for generating the audio signal from audio data of the plurality of individual audiovisual data streams, the audio decoder comprising the extractor (403 ), the recuperator (407), the replacer (415) and the outlet; and: by the receiver (1401) being arranged to generate an audio data processing descriptor for the second audio data of at least one of the audiovisual data streams and insert the audio data processing descriptor into a second stream of audiovisual data and feeding the second audiovisual data stream into the audio decoder; and by the audio decoder (1403) further comprising an audio data processing descriptor extractor for extracting the audio data processing descriptor from the second audiovisual data stream; and by the audio decoder (1403) being arranged to generate the audio signal in response to the audio data processing descriptor. 7. APPARATUS according to claim 6, characterized in that the audio data processing descriptor is indicative of whether the second audio data replaces the replaceable data. 8. APPARATUS according to claim 6, characterized in that the audio data processing descriptor is indicative of the second audio data being designated as alternative audio data or additional audio data, and in that the audio decoder (1403) is arranged so as to recover the second audio data as alternative audio data, if the first audio data processing descriptor indicates that the second audio data is designated as alternative audio data, and to recover the second audio data as alternative audio data additional audio data, if the first data processing descriptor indicates that the second audio data is designated as additional audio data; and replacing the substitutable audio data with the second audio data, if the second audio data is designated as substitutable audio data, and the output is arranged to generate the audio signal by combining the required audio data, of the second audio data and the replaceable audio data, if the first audio data processing descriptor indicates that the second audio data is designated as additional audio data. 9. APPARATUS according to claim 6, characterized in that the receiver (1401) is arranged to generate the first audio data processing descriptor from a second audio data processing descriptor received in the data stream of the audiovisual content item. 10. APPARATUS according to claim 6, characterized in that the receiver (1401) is arranged to copy the second audio data processing descriptor to the second audiovisual data stream. 11. APPARATUS according to claim 6, characterized in that the second audio data processing descriptor is comprised in the content item map table. 12. APPARATUS according to claim 6, characterized in that the data stream of the audiovisual content item comprises a plurality of possible audio data processing descriptors for the second audio data; and by the receiver being arranged to select the first audio data processing descriptor from the plurality of possible audio data processing descriptors.