KR20070084442A - Supporting fidelity range extensions in advanced video codec file format - Google Patents

Abstract

A parameter set is created to specify chroma format, luma bit depth, and chroma bit depth for a portion of multimedia data. The parameter set is encoded into a metadata file that is associated with the multimedia data. The parameter set is extracted from the metadata file if a decoder configuration record contains fields corresponding to the parameter set. In another aspect, the decoder configuration record is created with fields corresponding to the parameter set.

Description

SUPPORTING FIDELITY RANGE EXTENSIONS IN ADVANCED VIDEO CODEC FILE FORMAT}

<Related application>

This application is filed on February 21, 2003, all assigned to the same assignee as the present application, US Patent Application Ser. Nos. 10 / 371,434, 10 / 371,438, 10 / 371,464, and 10 / 371,927, and US Patent Application Nos. 10 / 425,291 and 10 / 425,685, both filed April 28, 2003.

<Technology field>

The present invention relates generally to the storage and retrieval of audiovisual content in a multimedia file format, and more particularly to a file format compatible with the ISO media file format.

<Copyright Notice / Permission>

2003, Sony Electronics, Inc., All Rights Reserved. The disclosure portion of this patent document contains copyrighted material. The copyright holder, as indicated in the Patent Office patent file or record, allows anyone to facsimile reproduction of the patent document or patent disclosure, but otherwise reserves all copyright rights. The following notice applies to the software and data and their drawings as described below: Copyright

2003, Sony Electronics, Inc., All Rights Reserved.

파일 포맷이다. QuickTime 파일 포맷은 ISO(International Organization for Standardization) 멀티미디어 파일 포맷, ISO/IEC 14496-12, 정보 기술 - 시청각 객체의 코딩 - 12편:ISO 미디어 파일 포맷(또한 ISO 파일 포맷으로서 공지됨)을 생성하는 시작점으로서 사용되었다. 한편, ISO 파일 포맷은 2개의 표준 파일 포맷을 위한 템플렛으로서 사용되었다: (1) MP4(ISO/IEC 14496-14, 정보 기술 - 시청각 객체의 코딩 - 14편:MP4 파일 포맷)로서 공지된 MPEG(Moving Picture Experts Group)에 의해 개발된 MPEG-4 파일 포맷, 및 (2) JPEG(Joint Photographic Experts Group)에 의해 개발된, JPEG 2000(ISO/IEC 15444-1)을 위한 파일 포맷.Due to the rapidly growing demand for networks, multimedia, databases, and other digital functions, many multimedia coding and storage methods have evolved. One of the well known file formats for encoding and storing audiovisual data is QuickTime, developed by Apple Computer, Inc.

File format. The QuickTime file format is an international organization for standardization (ISO) multimedia file format, ISO / IEC 14496-12, Information Technology-Coding of Audiovisual Objects-Part 12: The Starting Point for Creating ISO Media File Formats (also known as ISO File Formats). Was used as. On the other hand, the ISO file format was used as a template for two standard file formats: (1) MPEG (known as MP4 (ISO / IEC 14496-14, Information Technology-Coding of Audiovisual Objects-Part 14: MP4 File Format)). MPEG-4 file format developed by Moving Picture Experts Group, and (2) File format for JPEG 2000 (ISO / IEC 15444-1), developed by Joint Photographic Experts Group (JPEG).

The ISO media file format is a hierarchical data structure. The data structure includes metadata that provides declarative, structural, and temporal information about the actual media data. The media data itself may be located within the data structure, within the same file, or in another file external. Each metadata stream is referred to as a track. The metadata in this track contains structure information that provides a reference to external framed media data.

The media data referenced by the metadata track can be of various types (eg, video data, audio data, binary format screen representation (BIFS), etc.). The outer framed media data is separated into samples (also known as access units or pictures). A sample is the smallest data entity that represents a unit of media data at a particular point in time and can be represented by timing, location, and other metadata information. Therefore, each metadata track contains various sample entries and descriptions that provide information about the media data types referenced, followed by their timing, position, and size information.

Subsequently, the Video Group of MPEG and the Video Coding Experts Group (VCEG) of the International Telecommunication Union (ITU) began working together as the Joint Video Team (JVT) to develop a new video coding / decoding (codec) standard. This new standard is referred to as ITU Recommendation H.264 or MPEG-4-10, Advanced Video Codec (AVC). The encapsulation method defined in the AVC file format can be used to store coded video data generated by these specifications.

The JVT codec design is divided into two different concept layers, the video coding layer (VCL) and the network abstraction layer (NAL). The VCL includes coding related portions of the codec, such as motion compensation, coefficient transform coding, and entropy coding. The output of the VCL is slices, each containing header information associated with a series of video microblocks. NAL abstracts the VCL from the details of the transport layer used to carry the VCL data. NAL defines a general, transport-independent representation of the information and defines the interface between the video codec itself and the outside world. The JVT codec design specifies a set of NAL units, each containing a different type of data.

In many existing video coding formats, coded stream data includes various kinds of headers that contain parameters that control the decoding process. For example, the MPEG-2 video standard includes a sequence header, an enhanced group of picture (GOP), and a picture header before video data corresponding to these items. In the JVT, the information needed to decode the VCL data is grouped into parameter sets, and the JVT defines the NAL units for transmitting the parameter set to the decoder. The parameter set NAL unit may be transmitted in the same stream (in-band) or in a separate stream (out-of-band) as the video NAL unit.

The originally adopted H.264 Recommendation / AVC specification defines three basic feature sets, called profiles, baseline, main, and extended. These profiles only support chroma samples YUV 4: 2: 0 and 8 bits per sample, which are used for consumer video such as television, DVD, streaming video and the like. Several new profiles, collectively referred to as fidelity range extensions (FRExt), are created sequentially to allow storage and management of professional video formats. FRExt specifies higher bit depth encoding, including 10 and 12 bit video samples, and additional chromatic sample formats, such as YUV 4: 2: 2 and 4: 4: 4. In addition, FRExt also adds to the previously supported YcbCr (yellow, chroma-blue, chroma-red) color space, such as the International Commission on Illumination (CIE) XYZ and RBG (red, green, blue) color spaces. Specifies an additional color space.

Although the JVT team has adopted FRExt in their specification, the H.264 Recommendation / AVC specification itself does not define how the existing AVC file format is modified to incorporate new parameters associated with the extension.

A parameter set is generated for specifying the chroma format, luma bit depth, and chroma bit depth for some of the multimedia data. The parameter set is encoded into a metadata file associated with the multimedia data. If the decoder configuration record contains fields corresponding to the parameter set, the parameter set is extracted from the metadata file. In another aspect, the decoder configuration record is generated with fields corresponding to the parameter set.

The present invention is shown by way of example in the accompanying drawings, wherein like reference numerals refer to like elements, but are not limited thereto.

1 is a block diagram of one embodiment of an encoding system.

2 is a block diagram of one embodiment of a decoding system.

3 is a block diagram of a computer environment suitable for practicing the present invention.

4 is a flowchart of a method for storing parameter set metadata in an encoding system.

5 is a flowchart of a method of using parameter set metadata in a decoding system.

DETAILED DESCRIPTION In the following detailed description of embodiments of the invention, like reference numerals designate like elements, and reference is made to the accompanying drawings, in which specific embodiments in which the invention may be practiced are shown. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is understood that other embodiments can be used and that logic, mechanical, electrical, functional, and other changes can be made without departing from the scope of the invention. Will be. Therefore, the following detailed description is not to be considered in a limiting sense, and the scope of the present invention is defined only by the appended claims.

To support the FRExt described in the AVC specification, the decoder configuration record of the AVC file format is extended to specify the chromaticity format, the luminance bit depth, and the chroma bit depth for the portion of the multimedia data. The parameter set associated with the FRExt profile is encoded into a metadata file associated with the multimedia data. If the decoder configuration record contains fields corresponding to the presence of FRExt data, the parameter set is extracted from the metadata file.

Beginning with an overview of the operation of the present invention, FIG. 1 illustrates one embodiment of an encoding system 100 for generating parameter set metadata. Encoding system 100 includes a media encoder 104, a metadata generator 106, and a file generator 108. The media encoder 104 may synthesize video data (eg, video objects and other external video objects generated from natural source video scenes), audio data (eg, audio objects and other external audio objects generated from natural source audio scenes), synthesis Receive media data, which may include an object, or any combination thereof. Media encoder 104 may be comprised of a number of individual encoders or may include a sub-encoder to process various types of media data. Media encoder 104 codes the media data and passes it to metadata generator 106. Metadata generator 106 generates metadata that provides information about media data. For AVC, metadata is formatted as parameter set NAL units.

The file generator 108 stores metadata in a file whose structure is defined by the media file format. The media file format may specify that metadata was stored in-band or in whole or in part out-of-band. The coded media data is linked (eg, via a URL) to out-of-band metadata by a reference included in the metadata file. The file generated by file generator 108 is available in channel 110 for storage or transmission.

2 illustrates one embodiment of a decoding system 200 for extracting parameter set metadata. The decoding system 200 includes a metadata extractor 204, a media data stream processor 206, a media decoder 210, a composer 212, and a renderer 214. The decoding system 200 can reside on the client device and can be used for local playback. In other cases, decoding system 200 may be used for data streaming, and the server portion and client portion may communicate with each other via network 208 (eg, the Internet). The server portion may include a metadata extractor 204 and a media data stream processor 206. The client portion may include a media decoder 210, a builder 212, and a renderer 214.

The metadata extractor 204 is responsible for extracting metadata from files stored in the database 216 or received over the network (eg, from the encoding system 100). The decoder configuration record specifies the metadata that the metadata extractor 204 can handle. Any additional metadata that is not recognized is ignored.

The extracted metadata is also passed to a media data stream processor 206 that receives the associated and coded media data. The media data stream processor 206 forms the media data stream using the metadata and sends it to the media decoder 210.

Once the media data stream is formed, it is sent to the media decoder 210 directly (eg, for local playback) for decoding or via the network 208 (eg, for data streaming). The builder 212 receives the output of the media decoder 210 and creates a scene that is rendered on the user display device by the renderer 214.

The metadata can change between the time it is created and the time it is used to decode the corresponding portion of the media data. If such a change occurs, the decoding system 200 receives a metadata update packet specifying this change. The status of the metadata before and after the update is applied is managed in the metadata.

The following description of FIG. 3 is intended to provide an overview of computer hardware and other operating components suitable for implementing the present invention. 3 illustrates a computer system suitable for use as the metadata generator 106 and / or file generator 108 of FIG. 1, or the metadata extractor 204 and / or media data stream processor 206 of FIG. 2. One embodiment is shown.

Computer system 340 includes an input / output function 360 connected to processor 350, memory 355, and system bus 365. The memory 355 is configured to store instructions when executed by the processor 350 to perform the methods described herein. Input / output 360 also includes various types of machine-readable media, including any type of storage device accessible by processor 350. Those skilled in the art will immediately recognize that the term "machine-readable media" further includes a carrier wave that encodes the data signal. It will also be appreciated that system 340 is controlled by operating system software running on memory 355. Input / output and associated media 360 store computer-executable instructions for the operating system and method of the present invention. Each of the metadata generator 106, file generator 108, metadata extractor 204, and media data stream processor 206, shown in FIGS. 1 and 2, may be separate components connected to the processor 350. Or may be implemented as computer-executable instructions executed by the processor 350. In one embodiment, computer system 340 may be part of, or connected to, an Internet Service Provider (ISP) via input / output 360 to transmit or receive media data via the Internet. It is evident that the invention is also contemplated as a private network directly connected without being limited to Internet access and Internet web-based sites.

It will be appreciated that computer system 340 is one example of a number of possible computer systems having different architectures. A typical computer system will usually include at least a processor, memory, and a bus connecting memory to the processor. Those skilled in the art will readily appreciate that the present invention may be practiced in other computer system configurations, including multiprocessor systems, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

4 and 5 illustrate a process of storing and retrieving parameter set metadata performed by each of encoding system 100 and decoding system 200. The process may be performed by processing logic that may include hardware (eg, circuitry, dedicated logic, etc.), software (eg, software running on a general purpose computer system or dedicated machine), or a combination thereof. have. In the case of a software implemented process, the description of the flow diagram illustrates that a person of ordinary skill in the art would develop such a program containing instructions to execute such instructions on an appropriately configured computer (processor of a computer that executes instructions from a computer-readable medium containing memory). To run the process. The computer-executable instructions may be written in a computer programming language or may be implemented in firmware logic. When written in a programming language that conforms to recognized standards, such instructions can be executed for interfacing to various operating systems on various hardware platforms. In addition, embodiments of the present invention are not described with reference to any particular programming language. It will be understood that various programming languages may be used to implement the teachings described herein. In addition, in this field, reference to software, in one or another form (eg, program, procedure, process, application, module, logic, etc.), such as taking an action or causing a result, It is common. Such expressions are merely a shorthand way of saying that the execution of software by a computer causes the computer's processor to perform an action or produce a result. More or fewer operations may be incorporated into the process shown in FIGS. 4 and 5 without departing from the scope of the present invention, and no specific order is implied in the arrangement of the blocks shown and described herein.

4 is a flow diagram of one embodiment of a method 400 for generating parameter set metadata in an encoding system 100. The processing logic of block 402 receives a file with encoded media data that includes a set of encoding parameters that specify how to decode portions of media data. The processing logic examines the relationship between the set of encoding parameters and the corresponding portions of the media data (block 404), and generates metadata that defines the parameter set and its association with the media data portions (block 404). (406)).

In one embodiment, parameter set metadata is organized into sets of predefined data structures. The set of predefined data structures may include a data structure that includes descriptive information about the parameter set, and a data structure that includes information defining an association between the media data portions and the corresponding parameter set.

In one embodiment, processing logic determines whether any parameter set data structure includes a sequence of repeated data (block 408). If this determination is positive, processing logic converts each repeated sequence of data into a reference to the sequence occurrence and the number of times that sequence occurred (block 410). This type of parameter set is referred to as a sequence parameter set.

At block 412, the processing logic merges the parameter set metadata into a file associated with the media data using a particular media file format (eg, AVC file format). Depending on the media file format, parameter set metadata can be in-band or out-of-band.

5 is a flowchart of an embodiment of a method 500 of using parameter set metadata in a decoding system 200. In block 502 processing logic receives a file associated with encoded media data. The file may be received from a database (local or external database), encoding system 100, or any other device on the network. The file contains parameter set metadata that defines a parameter set for the corresponding media data. The processing logic of block 504 extracts parameter set metadata from the file.

In block 506 the processing logic uses the extracted metadata to determine which parameter set is associated with the particular media data portion. The information in the parameter set controls the decoding and transmission time of the parameter set corresponding to the media data portions.

In response to the adoption of the JVT FRExt profile, the chroma format and bit depth parameters are generated by the JVT team to merge FRExt into the existing AVC sequence parameter set. If the video sample is in one of the extended chromaticity formats, such as YUV 4: 2: 2 or 4: 4: 4, then the metadata generator of FIG. 1 when executing blocks 406 through 410 of the method 400. In 106, the corresponding sequence parameter set includes a chromaticity format indicator, "chroma_format_idc". The chroma_format_idc parameter specifies chromaticity (hue and saturation) sampling associated with luma (luminosity) sampling and has a value ranging from 0 to 3. The presence of 10 and 12 bit video samples is indicated by two additional parameters, bit_depth_luma_minus8 specifies the bit depth of the luminance sample, and bit_depth_chroma_minus8 specifies the bit depth of the chroma sample. The values of the bit_depth_luma_minus8 and bit_depth_chroma_minus8 parameters range from 0 to 4 according to the following formula:

BitDepth = 8+ bit_depth_luma_minus8

BitDepth = 8+ bit_depth_chroma_minus8

Therefore, a value of 0 corresponds to a bit depth of 8 bits, while a value of 4 corresponds to a bit depth of 12 bits.

A change corresponding to the AVC decoder configuration record of the AVC file format for a decoder capable of processing the media format specified by FRExt is required. In one embodiment, the class AVCDecoderConfigurationRecord is modified by adding the following fields.

bit (6) reserved = '111lll'b;

unsigned int (2) chroma_format;

bit (5) reserved = '111ll'b;

unsigned int (3) bit_depth_luma_minus8;

bit (5) reserved = '11111'b;

unsigned int (3) bit_depth_chroma_minus8;

Here, the chroma_format field includes a chromaticity format indicator defined by the chroma_format_idc parameter. The other two fields contain corresponding luminance and chromaticity parameter values.

Assuming the decoder 210 of FIG. 2 can decode the video in the extended format, the modified decoder configuration record is modified as the metadata extractor 204 executes block 505 of the method 500. Controls the extraction of FRExt parameters.

Storage and retrieval of audiovisual metadata has been described. Although specific embodiments have been described and described herein in the AVC file format, those skilled in the art will understand that any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. This application is intended to cover any adaptations or variations of the present invention.

Claims (22)

As a computerized method, Generating a parameter set for the portion of the multimedia data, wherein the parameter set includes, for the portion of the multimedia data, a chroma format, a luma bit depth, and a chroma bit depth; Contains a parameter specifying a; And Encoding the parameter set into a metadata file associated with the multimedia data How to include. The method of claim 1, wherein the portion of the multimedia data comprises video samples encoded with the chromaticity format and bit depth. The method of claim 1, wherein generating the parameter set comprises: Generating a first data structure including descriptive information about the parameter set and information defining an association between the parameter set and a portion of the multimedia data. The method of claim 1, Receiving the metadata file; And Extracting the parameter set from the metadata file, If the decoder configuration record does not include corresponding fields, the chroma format and the bit depth parameter are ignored. As a computerized method, Receiving a metadata file associated with the portion of the multimedia data, the metadata file comprising a parameter set for the portion of the multimedia data that specifies a chromaticity format, a luminance bit depth, and a chroma bit depth; And Extracting the parameter set from the metadata file, If the decoder configuration record does not include corresponding fields, the chroma format and the bit depth parameter are ignored. 6. The method of claim 5, wherein the portion of the multimedia data comprises video samples encoded with the chromaticity format and bit depth. As a computerized method, For the multimedia data, generating a decoder configuration record comprising metadata entries corresponding to the parameters for the chromaticity format, the luminance bit depth, and the chroma bit depth. 8. The method of claim 7, further comprising inserting the decoder configuration record into a decoder that processes multimedia data encoded with the chromaticity format and bit depth specified by the parameters. Generating a parameter set for the portion of the multimedia data, the parameter set including parameters specifying, for the portion of the multimedia data, a chromaticity format, a luminance bit depth, and a chroma bit depth; And Encoding the parameter set into a metadata file associated with the multimedia data A machine-readable medium having executable instructions for causing a processor to perform a method comprising a. 10. The machine-readable medium of claim 9, wherein the portion of the multimedia data comprises video samples encoded with the chromaticity format and bit depth. 10. The method of claim 9, wherein generating the parameter set comprises a first data structure comprising descriptive information about the parameter set and information defining an association between the parameter set and a portion of the multimedia data. Generating a second data structure. The method of claim 9, wherein Receiving the metadata file; And Extracting the parameter set from the metadata file, If the decoder configuration record does not include corresponding fields, the chroma format and the bit depth parameter are ignored. Receiving a metadata file associated with the portion of the multimedia data, the metadata file comprising a parameter set for the portion of the multimedia data that specifies a chromaticity format, a luminance bit depth, and a chroma bit depth; And Extracting the parameter set from the metadata file, And executable instructions that cause a processor to perform a method in which the chromaticity format and bit depth parameters are ignored if the decoder configuration record does not include corresponding fields. 14. The machine-readable medium of claim 13, wherein the portion of the multimedia data comprises video samples encoded with the chromaticity format and bit depth. Generating executable decoder configuration records comprising metadata entries corresponding to the parameters of chromaticity format, luminance bit depth, and chroma bit depth, for the multimedia data; Machine-readable media. A processor connected to the memory via a bus; And A process executed from the memory by the processor to cause the processor to generate a parameter set for the portion of multimedia data and to encode the parameter set into a metadata file associated with the multimedia data, the parameter set being the multimedia For a portion of data, including parameters specifying a chromaticity format, a luminance bit depth, and a chroma bit depth System comprising a. 17. The system of claim 16, wherein the portion of the multimedia data comprises video samples encoded with the chromaticity format and bit depth. 17. The method of claim 16, wherein generating the parameter set includes a first data structure that includes descriptive information about the parameter set and information defining an association between the parameter set and a portion of the multimedia data. Generating a second data structure. The chromaticity format of claim 16, wherein the process further causes the processor to receive the metadata file, extract the parameter set from the metadata file, and if the decoder configuration record does not include corresponding fields. And bit depth parameters are ignored. A processor connected to the memory via a bus; And A process executed from the memory by the processor to cause the processor to receive a metadata file associated with the portion of multimedia data and to extract the parameter set from the metadata file, the metadata file being the multimedia data For a portion of the parameter including a chromaticity format, a luminance bit depth, and a chroma bit depth, and if the decoder configuration record does not include corresponding fields, the chromaticity format and bit depth parameters are ignored. System comprising a. 21. The system of claim 20, wherein the portion of the multimedia data comprises video samples encoded with the chromaticity format and bit depth. A processor connected to the memory via a bus; And Executed by the processor to generate a decoder configuration record for the multimedia data, the decoder configuration record comprising metadata entries corresponding to the parameters for the chromaticity format, the luminance bit depth, and the chroma bit depth. process System comprising a.

Images