JP5591932B2 - Media extractor track for file format track selection - Google Patents

Description

  This application is a US provisional application 61 / 243,030 filed on September 16, 2009, filed September 22, 2009, the entire contents of each of which are incorporated herein by reference. Provisional application 61 / 244,827, US provisional application 61 / 293,961 filed on January 11, 2010, and US provisional application 61/295, filed January 15, 2010, Claim the profit of 261.

  The present disclosure relates to transport of encoded video data.

  Digital video functions include digital television, digital direct broadcast system, wireless broadcast system, personal digital assistant (PDA), laptop or desktop computer, digital camera, digital recording device, digital media player, video game console, video game console, It can be incorporated into a wide range of devices, including cellular or satellite radiotelephones, video teleconferencing devices, and the like. Digital video devices are MPEG-2, MPEG-4, ITU-T H.264, and so on. 263 or ITU-T H.264. Implement video compression techniques such as H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), and video compression techniques described in standards defined by extensions of such standards, and Transmit and receive video information more efficiently.

  Video compression techniques perform spatial prediction and / or temporal prediction to reduce or remove redundancy inherent in video sequences. For block-based video coding, a video frame or slice may be partitioned into macroblocks. Each macroblock may be further partitioned. Macroblocks in an intra-coded (I) frame or slice are encoded using spatial prediction on neighboring macroblocks. Macroblocks in an inter-coded (P or B) frame or slice may use spatial prediction for neighboring macroblocks in the same frame or slice, or temporal prediction for other reference frames.

  After the video data is encoded, the video data can be packetized by a multiplexer for transmission or storage. MPEG-2 includes a “system” section that defines transport levels for many video coding standards. The MPEG-2 transport level system can be used by MPEG-2 video encoders, or other video encoders that conform to different video encoding standards. For example, MPEG-4 defines a different encoding and decoding method than the MPEG-2 encoding and decoding method, but video encoders that implement the techniques of the MPEG-4 standard are still MPEG-2 transport. A level method can be used. In general, reference to “MPEG-2 system” refers to the transport level of video data defined by MPEG-2. In the present disclosure, the transport level defined by MPEG-2 is also referred to as “MPEG-2 transport stream” or simply “transport stream”. Similarly, the transport level of the MPEG-2 system also includes a program stream. The transport stream and program stream generally include different formats for delivering similar data, and the transport stream comprises one or more “programs” that include both audio and video data, and the program stream Includes one program that includes both audio and video data.

  H. Efforts are being made to develop new video coding standards based on H.264 / AVC. One such standard is H.264. It is a scalable video coding (SVC) standard that is a scalable extension of H.264 / AVC. Another standard is H.264. H.264 / AVC multi-view video coding (MVC). The MPEG-2 system specification describes how a compressed multimedia (video and audio) data stream can be multiplexed with other data to form a single data stream suitable for digital transmission or storage. Have been described. The latest specification of MPEG-2 system is `` Information Technology-Generic Coding of Moving Pictures and Associated Audio: Systems, Recommendation H.222.0; International Organization for Standardisation, ISO / IEC JTC1 / SC29 / WG11; Coding of Moving Pictures and Associated Audio ", May 2006.". MPEG has recently designed a transport standard for MVC over the MPEG-2 system. The latest version of this specification is "Study of ISO / IEC 13818-1: 2007 / FPDAM4 Transport of MVC", MPEG doc. . N10572, MPEG of ISO / IEC JTC1 / SC29 / WG11, Maui, Hawaii, USA, April 2009.

  The latest joint draft of MVC is JVT-AB204, “Joint Draft 8.0 on Multiview Video Coding, available at http://wftp3.itu.int/av-arch/jvt-site/2008_07_Hannover/JVT-AB204.zip ", 28th JVT Conference, Hannover, Germany, July 2008." Later versions incorporated into the AVC standard are JVT-AD007, “Editors' draft revision, available at http://wftp3.itu.int/av-arch/jvt-site/2009_01_Geneva/JVT-AD007.zip to ITU-T Rec. H.264 | ISO / IEC 14496-10 Advanced Video Coding-in preparation for ITU-T SG 16 AAP Consent (in integrated form) ”, 30th JVT Conference, Geneva, Switzerland, 2009 2 Listed in the month.

  In general, this disclosure describes techniques for using a media extractor in a multi-track video data format to form a media extractor track. The present disclosure modifies the International Standards Organization (ISO) base media format to utilize an extractor that can reference one or more potentially non-contiguous network access layer (NAL) units. . Such an extractor may be present in any track of the ISO base media format file. The present disclosure also describes a third generation partnership project (3GPP) file format change to include a frame rate value as an attribute of the track selection box. This disclosure further describes the use of extractors that support efficient extraction of MVC operating points for multi-view video coding (MVC) extensions to the ISO base media format.

  In one example, a method for encoding video data constructs a first track that includes video samples comprising a plurality of network access layer (NAL) units based on video data encoded by a source video device. At least one of a plurality of NAL units in the video samples of the first track by constructing a first track in which the video samples are included in the access unit and by the source video device Constructing a second track that includes an extractor identifying at least one of the plurality of NAL units comprising a first identified NAL unit, wherein the extractor is a second NAL of the access unit Identify the unit, the first identified NAL unit and the second Constructing a second track that is non-contiguous with the separated NAL units, and at least partially compliant with the International Organization for Standardization (ISO) base media file format for the first track and the second track Including in the video file to be output and outputting the video file.

  In another example, an apparatus for encoding video data includes an encoder configured to encode video data and a plurality of network access layer (NAL) units based on the encoded video data. Constructing a first track comprising video samples comprising constructing a first track, wherein the video samples are included in an access unit, and a plurality of NAL units in the video samples of the first track Constructing a second track that includes an extractor that identifies at least one of the at least one of the plurality of NAL units comprising the first identified NAL unit, wherein the extractor has access Identifies the second NAL unit of the unit, the first identified NAL unit and the second identification Building a second track that is non-contiguous with the recorded NAL units, and at least partially compliant with the International Organization for Standardization (ISO) base media file format for the first track and the second track A multiplexer configured to include in the video file and an output interface configured to output the video file.

  In another example, an apparatus for encoding video data is for constructing a first track that includes video samples comprising a plurality of network access layer (NAL) units based on the encoded video data. Means for constructing a first track, wherein the video sample is included in the access unit, and an extra identifying at least one of the plurality of NAL units in the video sample of the first track Means for constructing a second track comprising a Kuta, wherein at least one of the plurality of NAL units comprises a first identified NAL unit, and the extractor comprises a second NAL unit of the access unit. Identify and the first identified NAL unit and the second identified NAL unit are non-contiguous Means for constructing a second track; means for including the first track and the second track in a video file at least partially compliant with an International Organization for Standardization (ISO) base media file format; And means for outputting a video file.

  In another example, a computer-readable storage medium, when executed, constructs a first track that includes video samples comprising a plurality of network access layer (NAL) units based on encoded video data. A first sample including an extractor for identifying at least one of a plurality of NAL units in the video sample of the first track, and constructing a first track in which the video sample is included in the access unit; Constructing two tracks, wherein at least one of the plurality of NAL units comprises a first identified NAL unit, an extractor identifies a second NAL unit of the access unit, A second track in which the identified NAL unit and the second identified NAL unit are discontinuous. , Including a first track and a second track in a video file that is at least partially compliant with the International Organization for Standardization (ISO) base media file format, and outputting the video file Includes an instruction for causing the processor of the source device to execute.

  In another example, a method for decoding video data is to receive, by a demultiplexer of a destination device, a video file that conforms at least in part to an International Organization for Standardization (ISO) base media file format, The file comprises a first track and a second track, the first track comprising a video sample comprising a plurality of network access layer (NAL) units corresponding to the encoded video data, the video sample being accessed Included in the unit, the second track includes an extractor that identifies at least one of the plurality of NAL units of the first track, and at least one of the plurality of NAL units is first identified. NAL unit and extractor is an access unit Identifying a second NAL unit and receiving the first identified NAL unit and the second identified NAL unit are non-contiguous and selecting a second track to be decoded And sending the encoded video data of the first NAL unit and the second NAL unit identified by the extractor of the second track to the video decoder of the destination device.

  In another example, an apparatus for decoding video data receives a video decoder configured to decode video data and a video file that is at least partially compliant with an International Organization for Standardization (ISO) base media file format. A video sample comprising a first track and a second track, wherein the first track comprises a plurality of network access layer (NAL) units corresponding to the encoded video data. Wherein the video sample is included in the access unit, the second track includes an extractor identifying at least one of the plurality of NAL units of the first track, and at least one of the plurality of NAL units One with the first identified NAL unit and the extractor A second NAL unit of the access unit is identified, the first identified NAL unit and the second identified NAL unit are non-contiguous, and receiving a second track to be decoded A demultiplexer configured to select and send the encoded video data of the first NAL unit and the second NAL unit identified by the extractor of the second track to the video decoder Including.

  In another example, an apparatus for decoding video data is a means for receiving a video file at least partially compliant with an International Organization for Standardization (ISO) base media file format by a demultiplexer of a destination device. The video file comprises a first track and a second track, the first track comprising a video sample comprising a plurality of network access layer (NAL) units corresponding to the encoded video data; Is included in the access unit, the second track includes an extractor identifying at least one of the plurality of NAL units of the first track, and at least one of the plurality of NAL units is the first With the identified NAL unit, the extractor can access Means for receiving, wherein the first identified NAL unit and the second identified NAL unit are non-contiguous, and the second NAL unit to be decoded Means for selecting a track; means for sending the encoded video data of the first NAL unit and the second NAL unit identified by the extractor of the second track to a video decoder of the destination device; including.

  In another example, a computer-readable storage medium, when executed, selects a second track to be decoded when receiving a video file that conforms at least in part to the International Organization for Standardization (ISO) base media file format. A video sample comprising a first track and a second track, wherein the first track comprises a plurality of network access layer (NAL) units corresponding to the encoded video data. Wherein the video sample is included in the access unit, the second track includes an extractor identifying at least one of the plurality of NAL units of the first track, and at least one of the plurality of NAL units One with the first identified NAL unit and the extractor has access Identifying a second NAL unit of the unit, selecting a second track in which the first identified NAL unit and the second identified NAL unit are non-contiguous; and Encoded with instructions that cause the destination device processor to send the encoded video data of the first NAL unit and the second NAL unit identified by the extractor to the video decoder.

  The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims.

1 is a block diagram illustrating an example system in which an audio / video (A / V) source device transports audio and video data to an A / V destination device. The block diagram which shows the example structure of the component of a multiplexer. FIG. 3 is a block diagram illustrating an example file that includes a first track having a set of video samples and a second track having an extractor that references a subset of the video samples of the first track. FIG. 4 is a block diagram illustrating another example file that includes two separate extractor tracks. FIG. 4 is a block diagram illustrating another example file that includes a subset track and two media extractor tracks. FIG. 3 is a block diagram illustrating an example media data box of a file that includes examples of media extractors for various media extractor tracks. FIG. 3 is a block diagram illustrating an example media data box of a file that includes examples of media extractors for various media extractor tracks. FIG. 3 is a block diagram illustrating an example media data box of a file that includes examples of media extractors for various media extractor tracks. The conceptual diagram which shows an example MVC prediction pattern. FIG. 7 is a block diagram illustrating various examples of data structures for media extractors and other supporting data structures that may be used in accordance with the techniques of this disclosure. FIG. 7 is a block diagram illustrating various examples of data structures for media extractors and other supporting data structures that may be used in accordance with the techniques of this disclosure. FIG. 7 is a block diagram illustrating various examples of data structures for media extractors and other supporting data structures that may be used in accordance with the techniques of this disclosure. FIG. 7 is a block diagram illustrating various examples of data structures for media extractors and other supporting data structures that may be used in accordance with the techniques of this disclosure. FIG. 7 is a block diagram illustrating various examples of data structures for media extractors and other supporting data structures that may be used in accordance with the techniques of this disclosure. FIG. 7 is a block diagram illustrating various examples of data structures for media extractors and other supporting data structures that may be used in accordance with the techniques of this disclosure. FIG. 7 is a block diagram illustrating various examples of data structures for media extractors and other supporting data structures that may be used in accordance with the techniques of this disclosure. FIG. 7 is a block diagram illustrating various examples of data structures for media extractors and other supporting data structures that may be used in accordance with the techniques of this disclosure. FIG. 7 is a block diagram illustrating various examples of data structures for media extractors and other supporting data structures that may be used in accordance with the techniques of this disclosure. FIG. 7 is a block diagram illustrating various examples of data structures for media extractors and other supporting data structures that may be used in accordance with the techniques of this disclosure. FIG. 7 is a block diagram illustrating various examples of data structures for media extractors and other supporting data structures that may be used in accordance with the techniques of this disclosure. FIG. 7 is a block diagram illustrating various examples of data structures for media extractors and other supporting data structures that may be used in accordance with the techniques of this disclosure. FIG. 7 is a block diagram illustrating various examples of data structures for media extractors and other supporting data structures that may be used in accordance with the techniques of this disclosure. FIG. 7 is a block diagram illustrating various examples of data structures for media extractors and other supporting data structures that may be used in accordance with the techniques of this disclosure. FIG. 4 is a block diagram illustrating an exemplary modified third generation partnership project (3GPP) track selection box that signals additional attributes of the track selection box. 6 is a flowchart illustrating an example method for using a media extractor in accordance with the techniques of this disclosure.

  The techniques of this disclosure are generally directed to improving the International Organization for Standardization (ISO) base media file format and extensions of the ISO base media file format. ISO base media file format extensions include, for example, advanced video coding (AVC), scalable video coding (SVC), multiview video coding (MVC), and third generation partnership project (3GPP: Third Generation Partnership Project) file format included. In general, the techniques of this disclosure may be used to generate a media extractor track with an ISO base media file format and / or an extension of the ISO base media file format. As described in more detail below, such media extractor tracks may be used in some examples to support adaptation in Hypertext Transport Protocol (HTTP) video streaming. In some examples, the media extractor may extend the ISO base media file format and / or the ISO base media file format to extract a whole sample of another track to form a new media extractor track (eg, , AVC, SVC, MVC, and 3GPP).

  These techniques may be used by a Motion Picture Experts Group (MPEG-2) system, ie, a system that conforms to MPEG-2 with respect to transport level details. MPEG-4, for example, provides a standard for video coding, but in general it is assumed that video encoders compliant with the MPEG-4 standard utilize the MPEG-2 transport level system. Accordingly, the techniques of this disclosure are MPEG-2, MPEG-4, ITU-T H.264. 263, ITU-TH. The present invention is applicable to video encoders conforming to H.264 / MPEG-4, or any other video encoding standard that utilizes MPEG-2 transport streams and / or program streams.

  The ISO base media file format defines a file that includes one or more tracks. The ISO base media file format standard defines a track as a timed sequence of related samples. The ISO base media file format standard defines a sample as data related to a single time stamp, and samples as a video individual frame, a series of video frames in decoding order, or a compressed section of audio in decoding order. An example is given. Special tracks called hint tracks do not contain media data, but instead contain instructions for packaging one or more tracks into a streaming channel. The ISO base media file format standard mentions that in a hint track, a sample defines the formation of one or more streaming packets.

  The techniques of this disclosure allow for the generation of media extractor tracks. A media extractor track may generally include one or more extractors. An extractor in a media extractor track is used to identify and extract another track sample. In this way, media extractors in a media extractor track can be thought of as pointers that retrieve samples from another track when dereferenced. Unlike an SVC extractor, for example, an extractor of the present disclosure may reference one or more potential non-contiguous network access layer (NAL) units on another track. According to the techniques of this disclosure, a media extractor track, a track that includes one or more media extractors, and other tracks that do not include a media extractor are grouped together to form an alternate group. obtain.

  In this disclosure, the term “consecutive” is used with respect to NAL units to describe two or more NAL units that occur in succession in the same track. That is, when two NAL units are consecutive, the last byte of data in one of the NAL units comes immediately before the first byte of data of another NAL unit in the same track. Two NAL units in the same access unit generally have two NAL units separated by a certain amount of data in the same track, or one NAL unit occurs in one track and the other NAL In any case where the units occur in different tracks, they are considered “discontinuous”. The techniques of this disclosure provide an extractor that can identify two or more non-contiguous NAL units of an access unit.

  Moreover, the extractors of the present disclosure are not limited to SVC, but may generally be included in ISO base media file formats or other extensions of ISO base media file formats such as, for example, AVC, SVC, or MVC. The extractors of the present disclosure may also be included in a third generation partnership project (3GPP) file format. The present disclosure further allows the 3GPP file format to be changed to explicitly signal the frame rate as an attribute of the track selection box.

  The media extractor track can be used, for example, in the MVC file format to support operating point extraction. The server device may provide various operating points in the MPEG-2 transport layer bitstream, each of which corresponds to a respective subset of a particular view of multiview video coding video data. That is, the operating point generally corresponds to a subset of the view of the bitstream. In some examples, each view of the operating point includes video data at the same frame rate. In accordance with the techniques of this disclosure, an operating point is a media extractor that includes one or more extractors that reference video data of other tracks and potentially additional samples not included in other tracks. It can be represented using a track.

  In this way, each operating point may contain only the necessary NAL units required to decode the operating point to output a subset of views with a common frame rate. The combination of the extractor track and the full representation of the MVC video may form a playlist of MVC representations. The use of the media extractor track of the present disclosure may support operating point selection and switching, for example, for operating points where various bit rates result from temporal scalability.

  The media extractor tracks of the present disclosure can also be used to form alternative groups or switch groups. That is, in the ISO base media file format, tracks can be grouped together to form alternative groups. In the example of an ISO base media file format, alternate group tracks generally form a viable alternative to each other so that only one of the alternate group tracks is played or streamed at any given time. The tracks in the substitution group should be distinguishable from other tracks in the substitution group, for example via attributes such as bit rate, codec, language, packet size, or other characteristics. The techniques of this disclosure allow media extractor tracks, tracks containing media extractors, and / or other regular video tracks to be grouped to form alternative groups. In an example conforming to MVC, each track may correspond to a respective operating point. That is, each operating point in MVC may be represented by a particular one of the tracks, eg, either a media extractor track or a track that does not include a media extractor. One track in the same alternate group is generally selected for progressive download to accommodate available bandwidth.

  Similarly, media extractor tracks and other tracks can be grouped together to form a switch group in 3GPP file format, for track selection to adapt bandwidth and decoder capabilities in HTTP streaming applications. Can be used. The 3GPP file format provides a definition of a track switch group. The tracks in the switch group belong to the same alternative group. That is, according to the 3GPP file format, tracks in the same switch group are available for switching during a session, but tracks in different switch groups are not available for switching.

  FIG. 1 is a block diagram illustrating an exemplary system 10 in which an audio / video (A / V) source device 20 transports audio and video data to an A / V destination device 40. The A / V source device 20 may be referred to as a “source video device”. The system 10 of FIG. 1 is a video communication conferencing system, server / client system, broadcaster / receiver system, or video data from a source device such as an A / V source device 20 to a destination device such as an A / V destination device 40. Can correspond to any other system in which is sent. The A / V destination device 40 may be referred to as a “destination video device” or a “client device”. In some examples, A / V source device 20 and A / V destination device 40 may perform a bidirectional information exchange. That is, A / V source device 20 and A / V destination device 40 may be capable of both encoding and decoding (and transmitting and receiving) audio and video data. In some examples, the audio encoder 26 may comprise a voice encoder, also called a vocoder.

  In the example of FIG. 1, the A / V source device 20 includes an audio source 22 and a video source 24. Audio source 22 may comprise, for example, a microphone that generates an electrical signal representing captured audio data to be encoded by audio encoder 26. Alternatively, audio source 22 may comprise a storage medium that stores previously recorded audio data, an audio data generator such as a computer synthesizer, or any other source of audio data. Video source 24 may be a video camera that generates video data to be encoded by video encoder 28, a storage medium encoded with previously recorded video data, a video data generation unit, or any other video data May have a source.

  Raw audio and video data may comprise analog or digital data. The analog data may be digitized before being encoded by audio encoder 26 and / or video encoder 28. The audio source 22 may obtain audio data from the call participant while the call participant is speaking, and at the same time, the video source 24 may obtain the call participant's video data. In other examples, audio source 22 may comprise a computer readable storage medium comprising stored audio data and video source 24 may comprise a computer readable storage medium comprising stored video data. In this way, the techniques described in this disclosure may be applied to live, streaming, real-time audio and video data, or archived, pre-recorded audio and video data.

  An audio frame corresponding to a video frame is generally an audio frame that includes audio data captured by the audio source 22 at the same time as the video data captured by the video source 24 that is contained within the video frame. For example, the audio source 22 captures audio data while the call participant generally generates audio data by speaking, and at the same time, ie, while the audio source 22 is capturing audio data, the video source 24 is the call participant. Capture video data. Thus, an audio frame may correspond temporally to one or more specific video frames. Thus, an audio frame corresponding to a video frame is generally a situation where audio data and video data are captured simultaneously, and a situation where the audio frame and video frame comprise simultaneously captured audio data and video data, respectively. Corresponding to

  In some examples, audio encoder 26 may encode a time stamp in each encoded audio frame that represents the time at which the audio data of the encoded audio frame was recorded, and similarly, video encoder 28 may encode encoded video. A time stamp in each encoded video frame that represents the time at which the video data of the frame was recorded may be encoded. In such an example, the audio frame corresponding to the video frame may comprise a video frame with the same time stamp as the audio frame with the time stamp. A / V source device 20 may be used by audio encoder 26 and / or video encoder 28 to generate a time stamp therefrom or by audio source 22 and video source 24 to associate audio and video data with the time stamp, respectively. An internal clock may be included.

  In some examples, audio source 22 may send data corresponding to the time at which audio data was recorded to audio encoder 26, and video source 24 sends data corresponding to the time at which video data was recorded to video encoder 28. Can be sent to. In some examples, the audio encoder 26 may be included in the encoded audio data to indicate the relative time order of the encoded audio data without necessarily indicating the absolute time at which the audio data was recorded. The sequence identifier may be encoded, and similarly, video encoder 28 may use the sequence identifier to indicate the relative time order of the encoded video data. Similarly, in some examples, the sequence identifier may be mapped to a timestamp or possibly correlated with a timestamp.

  The techniques of this disclosure are generally directed to transporting encoded multimedia (eg, audio and video) data and receiving and subsequent interpretation and decoding of the transported multimedia data. The techniques of this disclosure may be used in various ways, for example, scalable video coding (SVC), advanced video coding (AVC), OSI base layer, or multi-view video coding (MVC) data, or other video data comprising multiple views. It can be applied to standard and extended video data transport. As shown in the example of FIG. 1, video source 24 may provide multiple views of a scene to video encoder 28. Multiple views of video data may be useful for generating 3D video data to be used by a 3D display, such as a stereoscopic or autostereoscopic 3D display.

  The A / V source device 20 may provide a “service” to the A / V destination device 40. A service generally corresponds to a subset of the available views of MVC data. For example, multi-view video data may be available for 8 views ordered from 0 to 7. One service may correspond to stereo video with two views, while another service may correspond to four views, and yet another service may correspond to all eight views. In general, a service corresponds to any combination of available views (ie, any subset). The service may also correspond to available views as well as combinations of audio data.

  A / V source device 20 may provide services corresponding to a subset of views in accordance with the techniques of this disclosure. Generally, a view is represented by a view identifier, also called “view_id”. A view identifier generally comprises a syntax element that can be used to identify a view. When a view is encoded, the MVC encoder gives the view's view_id. The view_id may be used for inter-view prediction by the MVC decoder, or may be used for other purposes, eg, rendering, by other units.

  Inter-view prediction is a technique for encoding MVC video data of a frame as encoded frames of different views with reference to one or more frames at a common temporal location. FIG. 7, described in further detail below, provides an exemplary coding scheme for inter-view prediction. In general, an encoded frame of MVC video data may be predictively encoded spatially, temporally, and / or with reference to frames of other views at a common temporal location. Thus, a reference view from which other views are predicted is generally prior to the view in which the reference view serves as a reference, so that when the reference view is decoded, the decoded reference view can be used for reference. Is decrypted. The decoding order does not necessarily correspond to the order of view_id. Therefore, the decoding order of views is described using the view order index. The view order index is an index indicating the decoding order of corresponding view components in the access unit.

  Each individual data stream (whether audio or video) is called an elementary stream. An elementary stream is a single component of a digitally encoded (possibly compressed) program. For example, the coded video or audio portion of the program can be an elementary stream. The elementary stream may be converted to a packetized elementary stream (PES) before being multiplexed into the program stream or transport stream. Within the same program, the stream ID is used to distinguish PES packets belonging to one elementary stream from others. The basic data unit of the elementary stream is a packetized elementary stream (PES) packet. Therefore, each view of the MVC video data corresponds to each elementary stream. Similarly, the audio data corresponds to one or more respective elementary streams.

  The MVC coded video sequence can be separated into several sub-bitstreams, each of which is an elementary stream. Each sub-bitstream may be identified using the MVC view_id subset. Based on the concept of each MVC view_id subset, an MVC video sub-bitstream is defined. The MVC video sub-bitstream includes NAL units of views described in the MVC view_id subset. A program stream generally includes only NAL units that are of elementary streams. It is also designed so that no two elementary streams can contain the same view.

  In the example of FIG. 1, the multiplexer 30 receives an elementary stream including video data from the video encoder 28 and receives an elementary stream including audio data from the audio encoder 26. In some examples, video encoder 28 and audio encoder 26 may each include a packetizer for forming a PES packet from the encoded data. In other examples, video encoder 28 and audio encoder 26 may each interface with a packetizer for forming PES packets from encoded data. In yet another example, multiplexer 30 may include a packetizer for forming PES packets from encoded audio data and encoded video data.

  A “program” as used in this disclosure may comprise a combination of audio data and video data, eg, an audio elementary stream delivered by a service of the A / V source device 20 and a subset of available views. Each PES packet includes a stream_id that identifies the elementary stream to which the PES packet belongs. Multiplexer 30 may assemble the elementary stream into a constituent program stream or a transport stream. A program stream and a transport stream are two alternative multiplexes that target different applications.

  In general, a program stream may include data for one program and a transport stream may include data for one or more programs. Multiplexer 30 may encode either or both of the program stream and / or transport stream based on the service provided, the medium on which the stream is passed, the number of programs to be sent, or other considerations. For example, when video data in a storage medium is to be encoded, multiplexer 30 may be more likely to form a program stream and the video data is streamed or broadcast over a network. Or when to be sent as part of video telephony, the multiplexer 30 may be more likely to use a transport stream.

  Multiplexer 30 may be biased in favor of using a program stream for storage and display of a single program from a digital storage service. Because program streams are rather error prone, program streams are intended for use in error-free or less error-prone environments. The program stream only comprises elementary streams belonging to it, and usually includes variable length packets. In the program stream, PES packets derived from the contributing elementary stream are organized into “packs”. A pack comprises a pack header, an optional system header, and any number of PES packets taken from any of the contributing elementary streams in any order. The system header contains a summary of the characteristics of the program stream, such as the maximum data rate of the program stream, the number of contributing video and audio elementary streams, additional timing information, or other information. The decoder may use the information contained in the system header to determine whether the decoder can decode the program stream.

  Multiplexer 30 may use the transport stream for simultaneous delivery of multiple programs over a potentially error-prone channel. A transport stream is a multiplex designed for multi-program applications such as broadcast, so that a single transport stream can be adapted to many independent programs. The transport stream comprises a sequence of transport packets, each of which is 188 bytes long. The use of short, fixed-length packets means that the transport stream is less error prone than the program stream. In addition, each 188 byte long transport packet may be given additional error protection by processing the packet through a standard error prevention process such as Reed-Solomon coding. Improved transport stream error resilience means that, for example, it is more likely to overcome error-prone channels that are to be discovered in a broadcast environment.

  A transport stream may appear to be a better multiplex of the two multiplexes, with its improved error resilience and the ability to carry many simultaneous programs. However, transport streams are much more multiplexed than program streams and are therefore more difficult to create and demultiplex. The first byte of the transport packet is a synchronization byte having a value of 0x47 (hexadecimal value 47, binary value “01000111”, decimal value 71). A single transport stream can carry many different programs, each program comprising many packetized elementary streams. Multiplexer 30 may use a 13-bit packet identifier (PID) field to distinguish transport packets that contain data from one elementary stream from those that carry data from other elementary streams. . It is the responsibility of the multiplexer to ensure that each elementary stream is given a unique PID value. The last byte of the transport packet is a continuity count field. The multiplexer 30 increments the value of the continuity count field between consecutive transport packets belonging to the same elementary stream. This allows a decoder or other unit of the destination device, such as A / V destination device 40, to detect the loss or gain of the transport packet and otherwise conceal errors that may arise from such events. To.

  The multiplexer 30 receives the PES packet of the elementary stream of the program from the audio encoder 26 and the video encoder 28, and forms a corresponding network abstraction layer (NAL) unit from the PES packet. H. In the H.264 / AVC (Advanced Video Coding) example, the coded video segments are organized into NAL units that provide a “network friendly” video representation that addresses applications such as video telephony, storage, broadcast, or streaming. NAL units may be categorized into Video Coding Layer (VCL) NAL units and non-VCL NAL units. The VCL unit includes a core compression engine and may comprise block, macroblock, and / or slice levels. Other NAL units are non-VCL NAL units.

  Multiplexer 30 may form a NAL unit comprising a header identifying the program to which the NAL belongs and a payload, eg, audio data, video data, or data describing the transport or program stream to which the NAL unit corresponds. For example, H.M. In H.264 / AVC, a NAL unit may include a 1-byte header and a variable size payload. In one example, the NAL unit header includes a priority_id element, a temporal_id element, an anchor_pic_flag element, a view_id element, a non_idr_flag element, and an inter_view_flag element. In conventional MVC, except for prefix NAL units and MVC coded slice NAL units, which include a 4-byte MVC NAL unit header and a NAL unit payload, NAL units defined by H.264 are retained.

  The priority_id element of the NAL header can be used for a simple one-path bitstream adaptation process. The temporal_id element may be used to specify the time level of the corresponding NAL unit when different time levels correspond to different frame rates.

  The anchor_pic_flag element may indicate whether the picture is an anchor picture or a non-anchor picture. The anchor picture and all pictures that follow it in the output order (ie display order) can be decoded correctly without decoding the previous picture in decoding order (ie bitstream order) and are therefore used as random access points Can be done. Anchor pictures and non-anchor pictures can have different dependencies, both of which are signaled in the sequence parameter set. Other flags are described and used in the following sections of this chapter. Such anchor pictures may also be referred to as open GOP (Group Of Pictures) access points, and closed GOP access points are also supported when the non_idr_flag element is equal to 0. The non_idr_flag element indicates whether the picture is an instantaneous decoder refresh (IDR) or a view IDR (V-IDR) picture. In general, an IDR picture and all pictures that follow it in output order or bitstream order can be correctly decoded without decoding the previous picture in decoding order or display order.

  The view_id element is a syntax that can be used to identify a view that can be used for data interactivity within the MVC decoder, eg, for inter-view prediction, and outside the decoder, eg, for rendering. Provide information. The inter_view_flag element may specify whether the corresponding NAL unit is used for inter-view prediction by other views. Prefix NAL units are defined in MVC to carry base view 4-byte NAL unit header information, which may be AVC compliant. In the context of MVC, the base view access unit includes a VCL NAL unit for the current time instance of the view, as well as a prefix NAL unit that contains only the NAL unit head. H. The H.264 / AVC decoder may ignore the prefix NAL unit.

  A NAL unit that includes video data in its payload may comprise video data of various granularity levels. For example, a NAL unit may comprise a block of video data, a macroblock, a plurality of macroblocks, a slice of video data, or an entire frame of video data.

  In general, an access unit may comprise one or more NAL units for representing a frame of video data, as well as such audio data when audio data corresponding to that frame is available. An access unit generally includes all NAL units over one output time instance, eg, all audio and video data over one time instance. H. In an example corresponding to H.264 / AVC, an access unit may comprise a coded picture in one time instance that may be presented as a primary coded picture. Thus, an access unit may comprise all view components corresponding to all video frames of a common time instance, eg, time X.

  This disclosure also refers to the coded picture of a particular view as a “view component”. That is, the view component comprises a coded picture (or frame) of a particular view at a particular time. Thus, an access unit may comprise all view components of a common time instance in some examples. The decoding order of access units does not necessarily have to be the same as the output or display order. A set of consecutive access units may form a coded video sequence that may correspond to a picture group (GOP) or other independently decodable unit of a NAL unit bitstream or sub-bitstream.

  As with many video coding standards, H.264 / AVC defines error-free bitstream syntax, semantics, and decoding processes, either of which conform to a specific profile or level. H. H.264 / AVC does not specify an encoder, but the encoder is required to ensure that the generated bitstream conforms to the decoder standard. In the context of a video coding standard, a “profile” corresponds to a subset of algorithms, functions, or tools and constraints that apply to them. For example, H.M. The “profile” defined by the H.264 standard is H.264. A subset of the entire bitstream syntax specified by the H.264 standard. A “level” corresponds to a limit on decoder resource consumption, eg, decoder memory and computations related to picture resolution, bit rate, and macroblock (MB) processing rate.

  H. The H.264 standard, for example, depends on the values taken by syntax elements in the bitstream, such as the specified size of the decoded picture, within the limits imposed by the syntax of a given profile. We recognize that there may still be a need for large fluctuations in performance. H. The H.264 standard further recognizes that in many applications it is neither practical nor economical to implement a decoder that can handle all hypothetical uses of syntax within a particular profile. Yes. Therefore, H.H. The H.264 standard defines a “level” as a specified set of constraints imposed on the values of syntax elements in a bitstream. These constraints can be simple limits on values. Alternatively, these constraints may take the form of constraints on a combination of value operations (eg, picture width × picture height × number of pictures decoded per second). H. The H.264 standard further defines that individual implementations may support different levels for each supported profile.

  Profile compliant decoders typically support all functions defined in the profile. For example, as a coding function, B-picture coding is H.264. H.264 / AVC baseline profile is not supported. Supported in other H.264 / AVC profiles. A level compliant decoder needs to be able to decode any bitstream that does not require resources beyond the limits defined in the level. Profile and level definitions can be useful for accountability. For example, during video transmission, a profile definition and level definition pair may be negotiated and agreed upon for all transmission sessions. More particularly, In H.264 / AVC, levels are, for example, limits on the number of macroblocks that need to be processed, decoded picture buffer (DPB) size, coded picture buffer (CPB) size, vertical motion vector range, One can define the maximum number of motion vectors per two consecutive MBs and whether a B block can have sub-macroblock partitions of less than 8 × 8 pixels. In this way, the decoder may determine whether the decoder can properly decode the bitstream.

  The parameter set generally includes sequence layer header information in the sequence parameter set (SPS) and rarely changing picture layer header information in the picture parameter set (PPS). If there is a parameter set, this infrequently changing information does not need to be repeated for each sequence or picture, thus coding efficiency can be improved. In addition, the use of parameter sets may allow out-of-band transmission of header information, avoiding the need for redundant transmissions to achieve error resilience. In out-of-band transmission, a parameter set NAL unit is transmitted on a different channel than other NAL units.

  The techniques of this disclosure involve including an extractor in a media extractor track. An extractor of the present disclosure may reference two or more NAL units on different tracks in a common file. That is, the file may include a first track having a plurality of NAL units and a second track including an extractor that identifies two or more of the plurality of NAL units of the first track. In general, when the demultiplexer 38 encounters an extractor, the extractor 38 can retrieve the NAL units identified by the extractor from the first track and send those NAL units to the video decoder 48. Can act as a pointer. A track that includes an extractor may be referred to as a media extractor track. The extractors of the present disclosure include various file formats, such as ISO base media file format, scalable video coding (SVC) file format, advanced video coding (AVC) file format, third generation partnership project (3GPP) file format, and And / or may be included in a file that conforms to a multi-view video coding (MVC) file format.

  In general, various tracks of a video file can be used as switch tracks. That is, multiplexer 30 may include various tracks to support various frame rates, display capabilities, and / or decoding capabilities. For example, when a video file conforms to the MVC file format, each track may represent a different MVC operating point. Accordingly, the demultiplexer 38 is configured to select one of the tracks for which the NAL unit is to be searched and discard data of other tracks other than the NAL unit identified by the extractor of the selected track. obtain. That is, when the selected track includes an extractor that references another track's NAL unit, the demultiplexer 38 may discard the unreferenced NAL unit of the other track while extracting the referenced NAL unit. The demultiplexer 38 may send the extracted NAL unit to the video decoder 48.

  By using extractors in the media extractor tracks, the techniques of this disclosure can be used to achieve temporal scalability between the various tracks of the video file. In MPEG-1 and MPEG-2, for example, B-coded pictures provide natural time scalability. The first track of a video file compliant with MPEG-1 or MPEG-2 may include a complete set of I-coded pictures, P-coded pictures, and B-coded pictures. The second track of the video file may include one or more extractors that reference only the I and P encoded pictures of the first track, omitting references to the B encoded pictures. By missing the B-encoded picture, the video file can achieve confirming a half-resolution video representation. MPEG-1 and MPEG-2 also provide a base layer and enhancement layer concept for coding two temporal layers, and enhancement layer pictures refer to pictures from either the base layer or the enhancement layer for each prediction direction. Can be selected.

  As another example, H.C. H.264 / AVC uses layer B coded pictures to support temporal scalability. H. The first picture of a video sequence in H.264 / AVC is sometimes referred to as an instantaneous decoder refresh (IDR) picture and is also known as a key picture. Key pictures are generally coded at regular or irregular intervals and are coded either intra-coded or inter-coded using the previous key picture as a reference for motion compensated prediction. A picture group (GOP) generally includes a key picture and all pictures located in time between the key picture and the previous key picture. A GOP may be divided into two parts, one is a key picture and the other contains non-key pictures. Non-key pictures are predicted hierarchically by two reference pictures, which are the closest pictures at lower time levels from the past and the future. A time identifier value may be assigned to each picture to indicate the hierarchical position of the picture. Thus, a picture with a time identifier value up to N may form a video segment with a frame rate twice that of the video segment formed by a picture with a time identifier value up to N-1. Thus, the techniques of this disclosure also refer to a first track that includes all NAL units with time identifier values up to N, and a NAL unit of the first track that has time identifier values up to N-1. Having a second track including one or more extractors. It can be used to achieve temporal scalability in H.264 / AVC.

  As described above, the techniques of this disclosure may include an ISO base media file format, a scalable video coding (SVC) file format, an advanced video coding (AVC) file format, a third generation partnership project (3GPP) file format, and / or multiple It can be applied to video files that conform to any of the view video coding (MVC) file formats. The ISO base media file format is designed to include timed media information for presentation in a flexible, extensible format that allows media exchange, management, editing, and presentation. The ISO base media file format (ISO / IEC 14496-12: 2004) is specified in MPEG-4 Part-12, which defines a general structure for time-based media files. It is H.264 / MPEG-4 AVC file format defined for support of AVC video compression (ISO / IEC 14496-15), 3GPP file format, SVC file format, and other file formats in the family, such as MVC file format Has been used as a basis for. The 3GPP file format and the MVC file format are extensions of the AVC file format. The ISO base media file format includes timing, structure, and media information for timed sequences of media data, such as audiovisual presentations. The file structure is object oriented. Files can be decomposed into basic objects very easily and the structure of the objects is implied from their type.

  Files conforming to the ISO base media file format are formed as a series of objects called “boxes”. Data in the ISO base media file format is contained in a box and there is no other data in the file. This includes the initial signature required by the particular file format. A “box” is an object-oriented building block defined by a unique type identifier and length. In general, the presentation is contained in one file, and the media presentation is self-contained. A movie container (movie box) contains media metadata, and video and audio frames are contained in the media data container and may be in other files.

  Presentations (motion sequences) can be included in several files. All timing and framing (position and size) information is generally in ISO base media files, and auxiliary files can use essentially any format. This presentation may be “local” to the system containing the presentation or may be via a network or other stream delivery mechanism.

  A file may have a logical structure, a time structure, and a physical structure, and these structures need not be combined. The logical structure of the file can be a movie that in turn contains a set of time-parallel tracks. The temporal structure of the file can be that the track contains a sequence of samples in time, which are mapped to the overall movie timeline by an optional edit list. The physical structure of the file may separate the data needed for logical, time and structural decomposition from the media data sample itself. This structural information is concentrated in the movie box and in some cases can be extended in time by the movie fragment box. Movie boxes may record the logic and timing relationships of samples and may include pointers to where they are located. These pointers can be, for example, pointers to the same file or another file referenced by the URL.

  Each media stream may be included in a special track for that media type (audio, video, etc.) and may further be parameterized by a sample entry. The sample entry may include the "name" of the correct media type (the type of decoder required to decode the stream) and any parameter indication of that decoder that is required. The name can also take the form of a four-letter code, for example, “moov” or “trak”. There are sample entry formats defined not only for MPEG-4 media, but also for media types used by other organizations that use this file format family.

  Metadata support generally takes two forms. First, the timed metadata is stored on an appropriate track and can be synchronized with the media data described by the timed metadata as needed. Second, there can be general support for non-timed metadata attached to movies or individual tracks. Structural support is common and allows storage of metadata resources to that file or elsewhere in another file, as is the case with media data. In addition, these resources can be named and protected.

  In the ISO base media file format, sample grouping is to assign each of the samples in a track to be a member of one sample group. Samples in a sample group need not be contiguous. For example, in the AVC file format, When presenting H.264 / AVC, video samples at one time level may be sampled into one sample group. A sample group can be represented by two data structures: a SampleToGroup box (sbdp) and a SampleGroupDescription box. The SampleToGroup box represents the assignment of samples to sample groups. There can be one instance of the SampleGroupDescription box for each sample group entry to describe the properties of the corresponding group.

  An optional metadata track to tag each track with its “interesting characteristics” (eg, its bit rate, screen size, or language) that it may have different values than other members of the group Can be used. Some samples in the track may have special characteristics or may be individually identified. An example of a characteristic is a synchronization point (often a video I frame). These points can be identified by special tables in each track. More generally, the nature of dependencies between track samples can also be recorded using metadata. The metadata can be structured as a series of file format samples, just like a video track. Such a track may be referred to as a metadata track. Each metadata sample can be structured as a metadata statement. There are various types of statements that correspond to various questions that can be asked about the corresponding file format sample or its constituent sample.

  When media is delivered via a streaming protocol, the media may need to be converted from the form it is represented in the file. An example of this is when the media is transmitted via a real time protocol (RTP). In the file, for example, each frame of the video is stored continuously as a file format sample. In RTP, in order to place these frames in RTP packets, the packetization rules specific to the codec used must be observed. A streaming server may be configured to compute such packetization at runtime. However, there is support for supporting streaming servers. Special tracks called hint tracks can be placed in the file.

  The hint track contains general instructions for the streaming server on how to form a packet stream from the media track for a particular protocol. Because the form of these instructions is media independent, it may not be necessary to modify the server when a new codec is introduced. Furthermore, the encoding and editing software may not be aware of the streaming server. Once editing is complete on the file, a piece of software called hinter can be used to place the hint track on the file before placing the file on the streaming server. One example is the hint track format defined for RTP streams in the MP4 file format specification.

  3GP (3GPP file format) is a multimedia container format defined by the 3rd Generation Partnership Project (3GPP) for 3G UMTS multimedia services. It is commonly used on 3G mobile phones and other 3G-enabled devices, but can also be played on some 2G and 4G phones and devices. The 3GPP file format is based on the ISO base media file format. The latest 3GP is defined in 3GPP TS 26.244, “Transparent end-to-end packet switched streaming service (PSS); 3GPP file format (3GP)”. The 3GPP file format is MPEG-4 Part 2 or H.264. The video stream is stored as H.263 or MPEG-4 Part 10 (AVC / H.264). Since 3GPP stipulates the use of sample entries and template fields in the ISO base media file format (MPEG-4 Part 12) and the definition of new boxes that codecs refer to, 3GPP is the ISO base media file format. AMR and H. Enables use of H.263 codec. For storage of MPEG-4 media specific information in 3GP files, the 3GP specification refers to MP4 and AVC file formats, which are also based on the ISO base media file format. The MP4 and AVC file format specifications describe the use of MPEG-4 content in the ISO base media file format.

  The SVC file format has a new structure of extractors and tiers as an extension of the AVC file format. An extractor is a pointer that gives information about the position and size of video coding data in a sample with equal decoding time in another track. This makes it possible to build the track hierarchy directly in the coding area. Extractor tracks in SVC are linked to one or more basic tracks from which data is extracted at runtime. The extractor is a dereferenceable pointer having a NAL unit header in the case of SVC extension. If the track used for extraction contains video coding data of different frame rates, the extractor also contains a decoding time offset to ensure synchrony between the tracks. At run time, before the stream is passed to the video decoder, the extractor must be exchanged with the data it points to.

  Since extractor tracks in SVC are structured like video coding tracks, the subsets required by that extractor track may be represented differently. The SVC extractor track contains only instructions on how to extract data from another track. In the SVC file format, there is also an aggregator that can aggregate NAL units in a sample as one NAL unit, including aggregating NAL units in one layer to an aggregator. Extractor in SVC to extract a range of bytes from a sample or aggregator, or to extract just one NAL unit but not multiple NAL units, especially non-contiguous ones in a sample Designed. There can be many video operating points in the SVC file format. A tier is designed to group samples in one or more tracks for an operating point.

  The MVC file format also supports extractor tracks, which extract NAL units from different views to form operating points that are a subset of the view at a certain frame rate. The design of the MVC extractor track is the same as the extractor in the SVC file format. However, using MVC extractor tracks to form alternate groups is not supported. To support track selection, the following MPEG proposal, P.I. Frojdh, A.M. Norkin, and C.I. PRIDELE, “File format sub-track selection and switching”, ISO / IEC JTC1 / SC29 / WG11 MPEG M16665, UK and London are proposed for MPEG. This proposal attempts to enable an alternative / switch group concept at the subtrack level.

  The map sample group is an extension to the sample group. In a map sample group, each group entry (in the sample) may have its own “groupID”, which is actually a map to view_id after aggregating the NAL units in the view into one NAL unit. Have a description. In other words, each sample group entry describes the view it contains in the ScalableNALUMapEntry value. The grouping_type of this sample group entry is “scnm”.

  Progressive download is a term used to describe the transfer of digital media files from a server to a client, typically using the HTTP protocol. When booted from the computer, the consumer can begin playing the media before the download is complete. The main difference between streaming media and progressive download is in how the digital media data is received and stored by the end user device accessing the digital media. A media player capable of progressive download playback uses the original metadata in the file header and the local buffer of the digital media file when downloaded from the web server. At the point where the specified amount of data becomes available to the local playback device, the media begins to play. This specified buffer amount is embedded in the file by the content producer in the encoder settings and is augmented by additional buffer settings imposed by the media player.

  3GPP supports HTTP / TCP / IP transport for 3GP files for download and progressive download. In addition, using HTTP for video streaming has several advantages, and video streaming services based on HTTP have become widespread. Some advantages of HTTP streaming include that existing Internet components and protocols can be used, thereby eliminating the need for new efforts to develop new techniques for transporting video data over the network. Other transport protocols, such as the RTP payload format, require an intermediate network device, such as an intermediate box, to be aware of the media format and signaling context. HTTP streaming can also be client driven to avoid many control problems. For example, to take advantage of all the features for optimal performance, the server may monitor the size and content of packets that have not yet been confirmed. The server can also reconfigure the client buffer state to analyze the file structure and make RD optimal switching / thinning decisions. In addition, constraints on the bitstream variants may be met to remain compliant with the negotiated profile. HTTP does not necessarily require a new hardware or software implementation on a web server where HTTP 1.1 is implemented. HTTP streaming also provides TCP affinity and firewall traversal. The techniques of this disclosure may improve HTTP streaming of video data, for example, by providing bit rate adaptation to overcome bandwidth related problems.

  ITU-TH. 261, H.H. 262, H.C. H.263, MPEG-1, MPEG-2 and H.264. Video compression standards such as H.264 / MPEG-4 part 10 utilize motion compensated temporal prediction to reduce temporal redundancy. The encoder uses motion compensated prediction from several previous coded pictures (also referred to herein as frames) to predict the current coded picture according to the motion vector. There are three main picture types in typical video coding. They are an intra-coded picture (“I picture” or “I frame”), a predictive picture (“P picture” or “P frame”), and a bi-predictive picture (“B picture” or “B frame”). It is. A block of P pictures may be intra-coded or predicted with respect to one other picture. For B pictures, a block can be predicted from one or two reference pictures or can be intra-coded. These reference pictures may be located before or after the current picture in temporal order.

  H. According to the H.264 coding standard, by way of example, a B picture uses two lists of previously coded reference pictures: list 0 and list 1. Each of these two lists may include past and / or future coded pictures in time order. Blocks in B pictures can be moved in several ways: motion compensated prediction from a list 0 reference picture, motion compensated prediction from a list 1 reference picture, or a combination of both a list 0 reference picture and a list 1 reference picture. It can be predicted with one of the compensated predictions. To obtain a combination of both the list 0 reference picture and the list 1 reference picture, two motion compensation reference areas are obtained from the list 0 reference picture and the list 1 reference picture, respectively. Their combination is used to predict the current block.

  Smaller video blocks can give better resolution and can be used for the location of video frames containing high levels of detail. In general, macroblocks and various partitions may be referred to as sub-blocks and may be considered video blocks. Further, a slice may be considered as multiple video blocks such as macroblocks and / or sub-blocks. Each slice may be a single decodable unit of a video frame. Alternatively, the frame itself can be a decodable unit, or other part of the frame can be defined as a decodable unit. The term “coding unit” or “coding unit” refers to any unit that can be decoded independently of a video frame, such as an entire frame, a slice of a frame, a picture group (GOP), also referred to as a sequence, or an applicable coding technique. It may refer to another independently decodable unit defined.

  The term macroblock refers to a data structure for encoding picture and / or video data according to a two-dimensional pixel array comprising 16 × 16 pixels. Each pixel has a chrominance component and a luminance component. Thus, a macroblock consists of four luminance blocks each comprising a two-dimensional array of 8 × 8 pixels, two chrominance blocks each comprising a two-dimensional array of 16 × 16 pixels, and a coded block pattern (CBP), Coding mode (eg, intra (I) or inter (P or B) coding mode), partition size of intra coding block partition (eg, 16 × 16, 16 × 8, 8 × 16, 8 × 8, 8 × 4, 4 × 8, or 4 × 4), or a header with syntax information, such as one or more motion vectors for inter-coded macroblocks.

  Video encoder 28, video decoder 48, audio encoder 26, audio decoder 46, multiplexer 30, and demultiplexer 38 are each one or more microprocessors, digital signal processors (DSPs), application specific, as applicable. It may be implemented as any of a variety of suitable encoder or decoder circuits, such as an integrated circuit (ASIC), field programmable gate array (FPGA), discrete logic, software, hardware, firmware, or any combination thereof. Each of video encoder 28 and video decoder 48 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined video encoder / decoder (CODEC). Similarly, each of audio encoder 26 and audio decoder 46 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined audio encoder / decoder (CODEC). An apparatus that includes video encoder 28, video decoder 48, audio encoder 26, audio decoder 46, multiplexer 30, and / or demultiplexer 38 may comprise an integrated circuit, a microprocessor, and / or a wireless communication device such as a cellular telephone.

  In accordance with the techniques of this disclosure, multiplexer 30 assembles a NAL unit into a track of a video file that conforms to an ISO base media file format or a derivative thereof (eg, SVC, AVC, MVC, or 3GPP) A media extractor track that identifies one or more potential non-contiguous NAL units may be passed and the video file may be passed to the output interface 32. The output interface 32 is a device for writing data to a computer readable medium, such as a transmitter, transceiver, eg, optical drive, magnetic media drive (eg, floppy drive), universal serial bus (USB) port, for example. A network interface or other output interface. The output interface 32 outputs the NAL unit or access unit to a computer readable medium 34, eg, a temporary medium such as a transmission signal or carrier wave, or a computer readable storage medium such as magnetic media, optical media, memory, or flash drive.

  Input interface 36 retrieves data from computer readable medium 34. The input interface 36 may comprise, for example, an optical drive, magnetic media drive, USB port, receiver, transceiver, or other computer readable media interface. Input interface 36 may provide a NAL unit or access unit to demultiplexer 38. The demultiplexer 38 demultiplexes the transport stream or program stream into constituent PES streams, depacketizes the PES stream to retrieve encoded data, and encodes, for example, as indicated by the PES packet header of the stream Depending on whether the data is part of an audio or video stream, the encoded data may be sent to an audio decoder 46 or a video decoder 48. The demultiplexer 38 first selects one of the tracks contained in the received video file, then the data for the selected track and other tracks referenced by the selected track extractor. And the data of the other tracks that are not referenced by the extractor of the selected track are discarded. The audio decoder 46 decodes the encoded audio data and sends the decoded audio data to the audio output 42, and the video decoder 48 decodes the encoded video data and may contain decoded views of the stream. Send data to video output 44. Video output 44 may comprise a display using a stereoscopic or autostereoscopic display that presents multiple views of the scene simultaneously, eg, each view of the scene.

  FIG. 2 is a block diagram illustrating an exemplary configuration of components of multiplexer 30 (FIG. 1). In the example of FIG. 2, the multiplexer 30 includes a stream management unit 60, a video input interface 80, an audio input interface 82, a multiplexed stream output interface 84, and a program specific information table 88. The stream management unit 60 includes a NAL unit constructor 62, a stream identifier (stream ID) lookup unit 66, a track generation unit 64, and an extractor generation unit 68.

  In the example of FIG. 2, video input interface 80 and audio input interface 82 include respective packetizers to form a PES unit from encoded video data and encoded audio data. In other examples, the video and / or audio packetizer may be external to the multiplexer 30. With respect to the example of FIG. 2, video input interface 80 may form PES packets from encoded video data received from video encoder 28, and audio input interface 82 may detect PES from encoded audio data received from audio encoder 26. Packets can be formed.

  After the NAL unit constructor 62 constructs the NAL unit, the NAL unit constructor 62 sends the NAL unit to the track generation unit 64. A track generation unit 64 receives the NAL unit and assembles a video file that includes the NAL unit in one or more tracks of the video file. Track generation unit 64 may further execute extractor generation unit 68 to generate extractors for one or more media extractor tracks constructed by track generation unit 64. When it is determined that one or more NAL units belong to multiple tracks, the extractor generation unit 68 does not duplicate the NAL units between the tracks, but the extractor generation unit 68 selects an extractor for the track that references the NAL unit. Can be built. In this way, multiplexer 30 can avoid duplication of data between tracks, thereby reducing bandwidth consumption when transmitting video files.

  Various examples of data structures and components for extractors are described below. In general, an extractor may include a track identifier value that references a track that includes a referenced NAL unit, and one or more NAL unit identifiers that identify the NAL unit referenced by the extractor. In some examples, the NAL unit identifier may refer to a bit or byte range in the track referenced by the track identifier value corresponding to the identified NAL unit. In some examples, for example, to identify non-contiguous NAL units, the NAL unit identifier may individually reference each NAL unit identified by the extractor. In some examples, the NAL unit identifier may reference a NAL unit based on an offset from the extractor's time or spatial location in the media extractor track.

  Track generation unit 64 may include additional NAL units in the media extractor track in some examples. That is, the media extractor track may include a NAL unit and an extractor. Thus, in some examples, the track generation unit 64 includes a first track that includes only NAL units, and a first track that includes one or more extractors that reference all or a subset of the NAL units of the first track. A video file having two tracks can be constructed. Moreover, in some examples, the track generation unit 64 may include additional NAL units in the second track that are not included in the first track. Similarly, the techniques of this disclosure may be extended to multiple tracks. For example, the track generation unit 64 may build a third track that may refer to the NAL unit of the first track and / or the NAL unit of the second track and is not included in the first or second track. It may further include a NAL unit.

  FIG. 3 is a block diagram illustrating an example file 100 that includes a first track having a set of video samples and a second track having an extractor that references a subset of the video samples of the first track. . In the example of FIG. 3, the file 100 includes a MOOV box 102 and a media data (MDAT) box 110. The MOOV box 102 corresponds to a movie box, and the ISO base media file format defines the movie box as a container box in which sub-boxes define metadata for presentation. The MDAT box 104 corresponds to a media data box, and the ISO base media file format defines the media data box as a box that can hold actual data for presentation.

  In the example of FIG. 3, the MOOV box 102 includes a complete subset track 104 and a media extractor track 106. The ISO base media file format defines a “track” as a timed sequence of related samples in an ISO base media file. The ISO base media file format further mentions that for media data, a track corresponds to a series of images or sampled audio.

  In the example of FIG. 3, the MDAT box 110 includes an I encoded sample 112, a P encoded sample 114, a B encoded sample 116, and a B encoded sample 118. B coded sample 116 and B coded sample 118 are considered to be at different hierarchical coding levels. In the example of FIG. 3, B coded sample 116 may be used as a reference for B coded sample 118, and therefore B coded sample 118 is a hierarchy lower than the hierarchical coding level of B coded sample 116. It can be an encoding level. The display order of the samples may be different from the hierarchical order (also called decoding order) and the order in which the samples are included in the MDAT box 110. For example, the I encoded sample 112 may have a display order value 0 and a decoding order value 0, the P encoded sample 114 may have a display order value 2 and a decoding order value 1, and the B encoded sample 116 may be The display order value 1 and the decoding order value 2 may be included, and the B encoded sample 118 may have the display order value 4 and the decoding order value 3. Track 1 may include additional samples, eg, samples with display order value 3 and decoding order value 4.

  Each of I-coded sample 112, P-coded sample 114, B-coded sample 116, and B-coded sample 118 may correspond to various NAL units or access units. The ISO base media file format takes a “sample” as a compressed section of all data associated with a single time stamp, eg, individual frames of video, a series of video frames in decoding order, or audio in decoding order. Defined. The complete subset track 104 includes metadata referring to the I encoded sample 112, the P encoded sample 114, the B encoded sample 116, and the B encoded sample 118 in the example of FIG.

  The MDAT box 110 further includes an extractor 120, an extractor 122, and an extractor 124. Accordingly, the extractors 120-124 are generally included in a movie data box that will contain a sample of data. In the example of FIG. 3, the extractor 120 refers to the I encoded sample 112, the extractor 122 refers to the P encoded sample 114, and the extractor 124 refers to the B encoded sample 118. There may be two or more NAL units corresponding to the I encoded sample 112, the P encoded sample 114, and / or the B encoded sample 118, and the NAL units may be non-contiguous. According to the techniques of this disclosure, even though there may be more than one non-contiguous NAL unit in the corresponding sample, the extractors 120-124 nevertheless identify each of the corresponding sample NAL units. obtain. In the example of FIG. 3, the media extractor track 106 includes metadata that refers to the extractor 120, the extractor 122, and the extractor 124.

  In addition, each of the extractors 120 to 124 may include a display order value and a decoding order value. For example, the extractor 120 may have a display order value 0 and a decoding order value 0, the extractor 122 may have a display order value 1 and a decoding order value 1, and the extractor 124 may have a display order value 2 And a decoding order value of 2. In some examples, the displayed and / or decoded values may skip some values, for example, to match the values of the identified samples.

  The complete subset track 104 and the media extractor track 106 may form an alternate group so that the demultiplexer 38 (FIG. 1) is to be decoded by the video decoder 48 to the complete subset track 104 or media extractor. Any of the tracks 106 may be selected. For the MVC example, the complete subset track 104 may correspond to a first operating point and the media extractor track 106 may correspond to a second operating point. For the 3GPP example, the complete subset track 104 and media extractor track 106 may form a switch group. In this way, the complete subset track 104 and media extractor track 106 can be used, for example, to adapt bandwidth availability and decoder capabilities in HTTP streaming applications.

  When the complete subset track 104 is selected, the demultiplexer 38 selects samples corresponding to the complete subset track 104 (eg, I-coded samples 112, P-coded samples 114, B-coded samples 116, and B-coded samples). Sample 118) may be sent to video decoder 48. When the media extractor track 106 is selected, the demultiplexer 38 sends samples corresponding to the media extractor track 106 to the video decoder 48, including samples identified by the media extractor corresponding to the media extractor track 106. obtain. Thus, when the media extractor track 106 is selected, the demultiplexer 38 can retrieve from the complete subset track 104 by dereferencing the extractor 120, extractor 122, and extractor 124. The I encoded sample 112, the P encoded sample 114, and the B encoded sample 118 may be sent to the video decoder 48.

  FIG. 4 is a block diagram illustrating another exemplary file 140 that includes two separate extractor tracks 146, 148. In the example of FIG. 4, two extractor tracks are shown, but in general, a file may include any number of extractor tracks. In the example of FIG. 4, the file 140 includes a MOOV box 142 and an MDAT box 150. The MOOV box 142 includes a complete subset track 144 and media extractor tracks 146, 148. The MDAT box 150 includes data samples and extractors for various tracks, eg, I-coded samples 152, P-coded samples 154, B-coded samples 156, B-coded samples 158, and extractors 160-168. including.

  In the example of FIG. 4, the extractors 160 to 164 correspond to the media extractor track 146, while the extractors 166 to 168 correspond to the media extractor track 148. In this example, extractor 160 of media extractor track 146 identifies I encoded sample 152, extractor 162 identifies P encoded sample 154, and extractor 164 identifies B encoded sample 156. To do. In this example, extractor 166 identifies I encoded sample 152, while extractor 162 identifies P encoded sample 154. The example of FIG. 4 shows an example where two or more extractors of various media extractor tracks reference the same sample of a complete subset track.

  The media extractor track can be used to represent a temporal subset of a video stream that is decodable and that contains the original full temporal resolution bitstream, eg, a video stream that is an alternative / switch track of the full subset track 144 . The complete subset track 144 may represent, for example, a 30 frame per second (FPS) video stream. In some examples, by not including certain hierarchical level B pictures in the sub-bitstream, the frame rate of the sub-bitstream may be halved or reduced by some other part. For example, the media extractor track 146 may have a frame rate that is halved relative to the full subset track 144 by not including the B encoded samples 158. For example, the media extractor track 146 may have a frame rate of 15 FPS. Similarly, media extractor track 148 has a frame rate that is halved relative to media extractor track 146 by omitting both B encoded sample 156 and B encoded sample 158, and thus frame May have a rate of 7.5 FPS.

  FIG. 5 is a block diagram illustrating another exemplary file 180 that includes a subset track 188 and two media extractor tracks 184, 186. The MOOV box 182 of the file 180 includes a subset track 188 and media extractor tracks 184 and 186, while the MDAT box 190 includes an I encoded sample 192, a P encoded sample 194, and a B encoded sample 202. , B encoded samples 208 and extractors 198, 200, 204, 206 and 210.

  As explained above, a media extractor track may include an extractor that references a sample of another track. Further, the media extractor track may further include additional video samples that are not included in another track. In the example of FIG. 5, subset track 188 includes I encoded samples 192 and P encoded samples 194. Media extractor track 186 includes extractors 198, 200 and further includes B encoded samples 202. Similarly, media extractor track 184 includes extractors 204, 206, 210 and B encoded samples 208.

  In the example of FIG. 5, media extractor track 186 includes encoded samples of video data (B encoded sample 202), and media extractor track 184 refers to a sample of media extractor track 186 that includes encoded samples. Including an extractor 210. That is, in the example of FIG. 5, the extractor 210 refers to the B encoded sample 202. Thus, media extractor track 184 may represent the full time resolution of the bitstream, while media extractor track 186 and subset track 188 may represent a subset of the full time resolution bitstream. That is, media extractor track 186 and subset track 188 may have a lower temporal resolution (eg, a lower frame rate) than the full temporal resolution represented by media extractor track 184.

  According to the techniques of this disclosure, H.264. The H.264 / AVC file format can be modified to include extractor tracks that can be extracted as any compliant time subset of the track that contains the original full time resolution bitstream. H.264 that supports layer B (or P) picture coding. For H.264 / AVC, assuming there are N time levels, each sub-bitstream containing samples from time levels 0 to k (k <N) can be extracted by defining the corresponding extractor track . Thus, for the same video, there can be N tracks (including N-1 extractor tracks) forming an alternate / switch group. The extractor can be associated with a time hierarchy level that corresponds to the time hierarchy level of the sample identified by the extractor. Also, for example, a time identifier value that specifies the time level of a sample may be signaled in the extractor.

  6A-6C are block diagrams illustrating an example of a file MDAT box 220 that includes examples of media extractors for various media extractor tracks. Each of FIGS. 6A-6C includes an anchor sample 222 including a view 0 sample 224A, a view 2 sample 226A, a view 1 sample 228A, a view 4 sample 230A, and a view 3 sample 232A, a view 0 sample 224B, and a view 2 sample 226B. , View 1 sample 228B, view 4 sample 230B, and non-anchor sample 223 including view 3 sample 232B. An ellipse beside the non-anchor sample 223 indicates that additional samples may be included in the MDAT box 220. Each of the anchor and non-anchor samples may collectively form the first track of the file. In one example, according to the techniques of this disclosure, the media extractor tracks for each set of file extractors shown in FIGS. 6A-6C may correspond to different operating points for video files that conform to the MVC file format. . In this way, the techniques of this disclosure may be used to generate one or more media extractor tracks that correspond to the operating point of a video file that conforms to the MVC file format.

  FIGS. 6A-6C illustrate various media extractor track extractors 240, 244, 250, each included in MDAT box 220, but separate views for clarity. Shown in That is, when fully assembled, MDAT box 220 may include each set of extractors 240, 244, and 250.

  FIGS. 6A-6C give examples of files containing media extractors as well as tracks containing real video samples. Various samples can be included separately in different tracks according to different time levels. For each time level, a particular track may include all video samples as well as extractors to tracks with lower time levels. Video samples (NAL units) can be separated into different tracks, but tracks with higher frame rates can have extractors pointing to other tracks. In this way, it is possible to have a movie fragment that contains samples of only one time level, and the movie fragment may in some cases include an extractor that points to another fragment. In this case, movie fragments of different tracks can be interleaved in ascending order of time level if they do not have the same time period.

  FIG. 6A provides an example of an extractor 240 that includes extractors 242A-242N corresponding to media extractor tracks. In this example, extractor 242A references view 0 sample 224A of anchor sample 222. Extractor 242N references view 0 sample 224B of non-anchor sample 223. In general, in the example of FIG. 6A, the extractors in extractor set 240 reference the corresponding view 0 samples. Each of the extractors 242A-242N corresponds to a common media extractor track that may belong to a switch group and / or an alternate group. The media extractor track may further correspond to individual operating points, eg, operating points including view 0.

  In some examples, for stereo video coded using MVC, one operating point that supports outputting two views, and only one view (eg, view 0 or view 1 only) There can be three operating points including a second operating point that supports output. The third operating point can be an operating point that outputs view 1. Depending on the prediction relationship, the third operating point is the VCL NAL unit in view 1 and the associated non-VCL NAL unit only, all NAL units in view 0 and view 1, or the NAL unit in view 1 and the anchor NAL. Units (ie, NAL units of anchor view components) may be included. In the case of such stereo, the example technique disclosed may provide that the other two operating points can be represented by two extractor tracks. These two extractor tracks can form a switch group, and together with the original video track, these three tracks can form an alternate group.

  The present disclosure provides techniques for changing the MVC file format to include MVC media extractor tracks. In general, MVC video tracks, including MVC media extractor tracks, with the same number of views for output, can be characterized as a switch group. All operating points represented by a track of a file can belong to one alternative group of MVC video presentations. Each view of anchor sample 222 and non-anchor sample 223 may form an operating point that includes a complete subset track, eg, all of the available views.

  The extractor may refer to a continuous portion of the sample as shown, for example, with respect to extractors 246A-246N in FIG. 6B. In the example of FIG. 6B, extractor 246A refers to view 0 sample 224A and view 2 sample 226A. The data structure representing extractor 246A can be a byte range for the identified view, a starting and ending view, a number of starting and subsequent views, or other representations of a series of consecutive views identified by the extractor. Can be specified. The set of extractors 244 may correspond to different media extractor tracks, which may in turn correspond to different MVC operating points.

  Also, the two extractors may reference two parts of the sample (eg, two non-contiguous views), for example, as shown with respect to extractors 254A, 256A in FIG. 6C. For example, extractor sample 252A includes an extractor 254A that references view 0 sample 224A and view 2 sample 226A, and an extractor 254B that references view 4 sample 230A. Thus, the sample represented by extractor sample 252A may correspond to an extractor sample that references a discontinuous view sample. Similarly, the extractor sample 252N includes an extractor 256A that references the view 0 sample 224B and the view 2 sample 226B and an extractor 256B that references the view 4 sample 230B in the example of FIG. 6C.

  An extractor may also be defined with respect to anchor or non-anchor samples, and an extractor defined with respect to anchor samples may reference a different view than an extractor defined with respect to non-anchor samples.

  The MVC media extractor track described above in the ISO base media file format or MVC file format can be implemented with similar extraction functions and can be used to represent regular video track alternatives and / or switch tracks. It can be an instance of a data track.

  In an example using the MVC file format, a complete bitstream may be included in one track, and all other possible operating points may be represented by extractor tracks, each of which is for example for output The number of views, the view identifier value of the view for output, the bandwidth required for transmission, and the frame rate may be signaled.

  FIG. 7 is a conceptual diagram illustrating an exemplary MVC prediction pattern. In the example of FIG. 7, eight views (with view IDs “S0” to “S7”) are shown, and twelve temporal locations (“T0” to “T11”) are shown for each view. That is, each row in FIG. 7 corresponds to a view, and each column indicates a temporal location.

  MVC is H.264. Although it has a so-called base view that can be decoded by an H.264 / AVC decoder, and stereo view pairs can be supported by MVC, the advantage of MVC is that MVC uses more than two views as 3D video input. It can support the example of decoding this 3D video represented by multiple views. A renderer of a client with an MVC decoder can use a plurality of views to predict 3D video content. Anchor view components and non-anchor view components in a view can have different view dependencies. For example, the anchor view component in view S2 depends on the view component in view S0. However, the non-anchor view components in view S2 do not depend on the view components in other views.

  The frame in FIG. 7 is shown for each row and each column in FIG. 7 using a shaded block containing characters, and the indication is that the corresponding frame is intra-coded (ie, an I frame). Or inter-coded in one direction (ie, as a P frame) or inter-coded in multiple directions (ie, as a B frame). In general, the prediction is indicated by an arrow, where the arrow end frame uses the arrow start object for prediction reference. For example, the P frame of view S2 at time location T0 is predicted from the I frame of view S0 at time location T0.

As with single view video encoding, frames of a multi-view video coding video sequence may be predictively encoded with respect to frames at different time locations. For example, the b frame of view S0 at time location T1 has an arrow pointing from the I frame of view S0 at time location T0 to that b frame, which indicates that the b frame is predicted from the I frame. Show. In addition, however, in the context of multi-view video coding, frames can be inter-view predicted. That is, view components can use view components in other views for reference. In MVC, for example, inter-view prediction is realized as if the view component in another view is an inter prediction reference. Potential inter-view references are signaled in a sequence parameter set (SPS) MVC extension and may be modified by a reference picture list construction process that allows flexible ordering of inter-prediction or inter-view prediction references. Table 1 below provides an exemplary definition of the MVC extended sequence parameter set.

  FIG. 7 gives various examples of inter-view prediction. In the example of FIG. 7, the frame of view S1 is assumed to be predicted from frames at different time locations of view S1, and is assumed to be inter-view predicted from the frames of views S0 and S2 at the same time location. It is shown. For example, the b frame of view S1 at time location T1 is predicted from each of the B frames of view S1 at time locations T0 and T2 and the b frames of views S0 and S2 at time location T1.

  In the example of FIG. 7, the uppercase “B” and the lowercase “b” indicate different hierarchical relationships between frames, not different encoding methods. In general, uppercase “B” frames have a relatively higher prediction hierarchy than lowercase “b” frames. That is, in the example of FIG. 7, the “b” frame is encoded with respect to the “B” frame. Additional hierarchical levels can be added with additional bi-coded frames that can refer to the “b” frame of FIG. FIG. 7 also illustrates a variant of the prediction hierarchy using different levels of shading, with a larger amount of shading (ie, relatively darker) frames having less shading (ie, relatively lighter). ) The prediction hierarchy is higher than those frames. For example, all I frames in FIG. 7 are shown with full shading, while P frames have somewhat brighter shading, and B frames (and lowercase b frames) have different levels of each other. Has shading, but is always brighter than the shading of P and I frames.

  In general, a frame with a relatively high prediction hierarchy is a frame with a relatively lower hierarchy so that those frames with a higher hierarchy can be used as reference frames during decoding of a frame with a lower hierarchy. The prediction hierarchy is related to the view order index in that it should be decoded before decoding. The view order index is an index indicating the decoding order of the view components in the access unit. H. The view order index is implied in the SPS MVC extension, as specified in Annex H of H.264 / AVC (MVC Addendum). In SPS, for each index i, the corresponding view_id is signaled. Decoding of view components follows the ascending order of the view order index. If all views are presented, the view order index is a sequential order from 0 to num_views_minus_1.

  In this way, a frame used as a reference frame can be decoded before decoding a frame encoded with reference to the reference frame. The view order index is an index indicating the decoding order of the view components in the access unit. For each view order index i, the corresponding view_id is signaled. Decoding of view components follows the ascending order of the view order index. If all views are presented, the set of view order indices comprises a continuous ordered set from 0 to one less than the total number of views.

  For some frames at equal levels in the hierarchy, the decoding order may not be important to each other. For example, the I frame of view S0 at time location T0 is used as a reference frame for the P frame of view S2 at time location T0, which in turn is for the P frame of view S4 at time location T0. Used as a reference frame. Thus, the I frame of view S0 at time location T0 should be decoded before the P frame of view S2 at time location T0, and the P frame is decoded before the P frame of view S4 at time location T0. It should be. However, views S1 and S3 do not rely on each other for prediction, but instead, the decoding order is not important between views S1 and S3 because the prediction hierarchy is only predicted from higher views. Moreover, view S1 can be decoded before view S4 as long as view S1 is decoded after views S0 and S2.

In this way, a hierarchical order can be used to describe views S0-S7. The notation SA> SB means that the view SA should be decoded before the view SB. Using this notation, in the example of FIG. 7, S0>S2>S4>S6> S7. For the example of FIG. 7, S0> S1, S2> S1, S2> S3, S4> S3, S4> S5, and S6> S5. Any decoding order for views that do not violate these requirements is possible. Thus, many different decoding orders are possible with only some limitations. Two exemplary decoding orders are presented below, but it should be understood that many other decoding orders are possible. In the example shown in Table 2 below, the view is decoded as soon as possible.

  In the example of Table 2, view S1 may be decoded immediately after views S0 and S2 are decoded, view S3 may be decoded immediately after views S2 and S4 are decoded, and view S5 is viewed as views S4 and S6. Recognizes that it can be decoded immediately after it is decoded.

In Table 3 below, another example where any view used as a reference for another view is in decoding order such that it is decoded before a view that is not used as a reference for another view. Give the correct decoding order.

  The example in Table 3 shows that the frames of views S1, S3, S5, and S7 do not serve as reference frames for the frames of other views, so views S1, S3, S5, and S7 are examples of FIG. In view, ie, views S0, S2, S4, and S6 can be decoded after the frame used as the reference frame. For each other, views S1, S3, S5, and S7 may be decoded in any order. Thus, in the example of Table 3, view S7 is decoded before each of views S1, S3, and S5.

  For clarity, there may be a hierarchical relationship between the frames of each view as well as between the time locations of the frames of each view. With respect to the example of FIG. 7, the frame at temporal location T0 is intra predicted from the frames of other views at temporal location T0 or is inter-view predicted. Similarly, the frame at temporal location T8 is intra predicted from the frames of other views at temporal location T8 or inter-view predicted. Thus, with respect to the time hierarchy, time locations T0 and T8 are at the top of the time hierarchy.

  In the example of FIG. 7, the frame at time location T4 is B-coded with reference to the frames at time locations T0 and T8, so that the frame at time location T4 has a higher time hierarchy than the frames at time locations T0 and T8. Low. Frames at time locations T2 and T6 are lower in time hierarchy than frames at time location T4. Finally, the frames at time locations T1, T3, T5, and T7 are lower in time hierarchy than the frames at time locations T2 and T6.

  In MVC, a subset of the entire bitstream can be extracted to form a sub-bitstream that is still compliant with MVC. For example, many applications may require a particular application based on the service provided by the server, the capacity, support and capabilities of one or more client decoders, and / or one or more client preferences. There are possible sub-bitstreams. For example, a client may need only three views and there may be two scenarios. In one example, one client may require a smooth browsing experience and may prefer views with view_id values S0, S1, and S2, while another client may require view scalability, and view_id values S0, S2, and You may prefer the view of S4. If the view_id was originally ordered with respect to the example in Table 9, the view order index values in these two examples are {0, 1, 2} and {0, 1, 4}, respectively. Note that both of these sub-bitstreams can be decoded and supported simultaneously as independent MVC bitstreams.

  There can be many MVC sub-bitstreams that can be decoded by an MVC decoder. Theoretically, (1) the view components in each access unit are ordered in ascending order of the view order index, and (2) for each view in any combination of views, that view's dependent view is also the above combination The above combination that satisfies the two properties of being included in can be decoded by an MVC decoder that conforms to a certain profile or level.

  With respect to the techniques of this disclosure, various MVC sub-bitstreams may be represented using media extractor tracks and / or pure video sample tracks. Each of these tracks may correspond to an MVC operating point.

  8-21 are block diagrams illustrating various examples of data structures for media extractors and other supporting data structures that may be used in accordance with the techniques of this disclosure. The various media extractors of FIGS. 8-22 include various features described in detail below. In general, any of the media extractors of FIGS. 8-21 is in a media extractor track of a file that conforms to the ISO base media file format or an extension to the ISO base media file format to identify a coded sample of the file. Can be included. In general, a media extractor can be used to extract one or more whole samples from a referenced track. FIGS. 8-12 are examples of media extractors that can identify one video sample box of another track. As shown in FIG. 13, another way to implement an extractor is to allow sample grouping of samples from different tracks. To provide more specific support for time scalability, a time identifier may be signaled as shown in FIG. 16-22 are examples of media extractors for MVC, which can extract one or more potential non-contiguous NAL units from each video sample box (access unit). Various examples of extractors are based on offsets and lengths of bytes in the file or access unit, but other examples can be based purely on the index of all NAL units, thus requiring byte range signaling. Not necessary. Also, the mechanism of the signaling extractor with the index of all NAL units can be extended to the SVC file format.

  8 to 21 can be directly applied to the 3GPP file format as an extension to the 3GPP file format. Also, one or more elements and concepts of FIGS. 8-21 may be combined with elements of other figures of FIGS. 8-22 to form other extractors. Although some of the figures in FIGS. 8-21 have been described with respect to particular file formats, in general, the examples of FIGS. 8-21 are arbitrary file formats with similar characteristics, eg, ISO-based It can be used in connection with an extension of the media file format or the ISO base media file format. To enable the use of the extractor proposed in 3GPP, the 3GPP track selection box includes a time identifier, the number of views to be displayed, and the number of views to be decoded, as shown in the example of FIG. Etc. and can be extended to include more properties for each of the alternative tracks (extracted).

  FIG. 8 is a block diagram illustrating an exemplary media extractor 300 showing the format of the media extractor. In the example of FIG. 8, the media extractor 300 includes a track reference index 302 and a sample offset value 304. In accordance with the techniques of this disclosure, media extractor 300 may correspond to a definition of a data structure that can be instantiated within a media extractor track. Multiplexer 30 may be configured to include an extractor that conforms to the example of media extractor 300 in the media extractor track of the video file to identify NAL units of different tracks of the video file. Demultiplexer 38 may be configured to search for identified NAL units using an extractor that is compliant with media extractor 300.

  The track reference index 302 may correspond to the identifier of the track in which the identified NAL unit is present. In order to distinguish the tracks of the video file, each track of the video file may be assigned a unique index. The track reference index 302 may specify the index of the track reference used to find the track from which data is to be extracted. The sample in the track from which the data is extracted can be accurately time aligned to the sample containing the extractor (specified by the sample offset value 304 using the time sample table in the media decoding timeline. Only the offset is adjusted). In some examples, the first track of the video file has an index value “1”, and therefore the multiplexer 30 sets the value “1” to the track reference index value 302 that references the first track of the video file. Can be assigned. The value “0” of the track reference index value may be reserved for future use.

  The sample offset value 304 defines an offset value from the time location of the media extractor 300 in the media extractor track to the identified NAL unit of the track referenced by the track reference index 302. That is, the sample offset value 304 gives the relative index of the sample in the linked track that is used as an information source. A sample offset value 304 of 0 refers to the sample with the same or closest preceding decoding time as the sample containing the extractor. Sample 1 is the next sample, Sample-1 is the previous sample, and so on. For example, a media extractor conforming to the media extractor 300 is H.264. When used in H.263 or MPEG-4 part 2, the media extractor can be used to extract a temporal subset of the video track referenced by the track reference index 302.

The following pseudo code provides an exemplary definition of a media extractor class similar to media extractor 300.

  Multiplexer 30 and demultiplexer 38 may instantiate a media extractor data object using the media extractor defined in the example pseudocode above. Thus, when demultiplexer 38 retrieves data from a selected track, for example, to retrieve identified data from another track referenced by the instantiated media extractor, instantiated media extractor You can refer to

  In the exemplary pseudo code, the class MediaExtractor () is byte aligned. That is, when the extractor is instantiated from the MediaExtractor () class, the extractor is aligned on an 8-byte boundary. The variable “track_ref_index” corresponds to the track reference index value 302, and in this exemplary pseudo code corresponds to an unsigned 8-byte integer value. The variable “sample_offset” corresponds to the sample offset value 304, and in this example corresponds to a signed 8-byte integer value.

  FIG. 9 is a block diagram illustrating another example of the media extractor 310. The media extractor 310 includes a track reference index 314 and a sample offset value 316, and further includes a sample header 312. The track reference index 314 and sample offset value 316 may include data that is generally similar to the track reference index 302 and sample offset value 304 (FIG. 8).

  The sample header 312 is an H.264 file. In an example corresponding to H.264 / AVC, it may be constructed according to the NAL unit header of the video sample referenced by the media extractor 310. The sample header 312 may include one byte of data with three syntax elements: forbidden_zero_bit, nal_ref_idc (which may comprise 3 bits), and nal_unit_type (which may comprise 5 bits). The value of “nal_unit_type” may be 29 (or any other reserved number), and the other two syntax elements may be similar to those syntax elements in the identified video sample. For an example compliant with MPEG-4 part-2 visual, the sample header 312 may include a start code prefix “0x 00 00 01” and a start code “0x C5” (or any other reserved number) 4 A byte code may be provided, and “0x” indicates that the value following “0x” is a hexadecimal value. H. In the case of H.263, the sample header 312 may also include a byte aligned start code that is different from the start code of a normal video sample. The sample header 312 can be used by the demultiplexer 38 for synchronization purposes so that the extractor can be considered a normal video sample.

The following pseudo code provides an exemplary definition of a media extractor class similar to media extractor 310.

  Multiplexer 30 and demultiplexer 38 may instantiate a media extractor data object using the media extractor defined in the example pseudocode above. Thus, when demultiplexer 38 retrieves data from a selected track, for example, to retrieve identified data from another track referenced by the instantiated media extractor, instantiated media extractor You can refer to

  FIG. 10 is a block diagram illustrating an example media extractor 320 that identifies a NAL unit by signaling the byte range of the NAL unit identified in the extractor. The media extractor 320 includes a sample header 322 that can be similar to the sample header 312 and a track reference index 324 that can be similar to the track reference index 302. However, instead of the sample offset value, the example of the media extractor 320 includes a data offset value 326 and a data length value 328.

  Data offset value 326 may describe the starting point of the data identified by media extractor 320. That is, the data offset value 326 may comprise a value representing the offset to the first byte in the track identified by the track index value 324 to be copied. The data length value 328 may describe the number of bytes to be copied and thus may be equivalent to the length of the referenced sample (or multiple samples when referring to multiple NAL units).

The following pseudo code gives an exemplary definition of a media extractor class similar to media extractor 320.

  Multiplexer 30 and demultiplexer 38 may instantiate a media extractor data object using the media extractor defined in the example pseudocode above. Thus, when demultiplexer 38 retrieves data from a selected track, for example, to retrieve identified data from another track referenced by the instantiated media extractor, instantiated media extractor You can refer to

FIG. 11 is a block diagram illustrating an example media extractor 340 that includes reserved bits for future scalability. Media extractor 340 includes track reference index 342 and sample offset value 346, which may be similar to media extractor 302 and sample offset value 304, respectively. In addition, media extractor 340 includes reserved bits 344 that may comprise reserved bits used for future extensions to the media extractor. The following pseudo code provides an example class definition for a media extractor class similar to media extractor 340.

  Multiplexer 30 and demultiplexer 38 may instantiate a media extractor data object using the media extractor defined in the example pseudocode above. Thus, when demultiplexer 38 retrieves data from a selected track, for example, to retrieve identified data from another track referenced by the instantiated media extractor, instantiated media extractor You can refer to

  FIG. 12 is a block diagram illustrating an example media extractor 350 that uses track identifier values rather than track reference index values. Use of the track identifier value to identify a track may refer to a presentation of a track reference box in the ISO base media file format. An example media extractor 350 includes a track identifier 352, a reserved bit 354, and a sample offset value 356. Reserved bit 354 is optional as indicated by the dashed line around reserved bit 354. That is, some examples may include reserved bits 354, while other examples may omit reserved bits 354. Sample offset value 356 may be similar to sample offset value 304.

  The track identifier 352 specifies the track ID of the track from which data is to be extracted. The sample in the track from which the data is extracted can be accurately time aligned to the sample containing the media extractor 350 (specified by the sample offset 356 using the time sample table in the media decoding timeline. Only the offset is adjusted). The first track reference may be assigned an identifier value of 1. The value 0 may be reserved for future use and expansion.

The following pseudo code provides an exemplary definition of a media extractor class similar to media extractor 350.

  Multiplexer 30 and demultiplexer 38 may instantiate a media extractor data object using the media extractor defined in the example pseudocode above. Thus, when demultiplexer 38 retrieves data from a selected track, for example, to retrieve identified data from another track referenced by the instantiated media extractor, instantiated media extractor You can refer to

  FIG. 13 is a block diagram illustrating an exemplary media extractor sample group 360. Multiplexer 30 may include a media extractor sample group 360 in a message type box (with type identifier “MESG”) in the sample table box container. Multiplexer 30 may be configured to include zero or one media extractor sample group 360 objects in the message box. In the example of FIG. 13, the media extractor sample group 360 includes a track reference index 362, a group type 364, a group count 366, a reserved bit 368, and a group description index 370.

  The track reference index 362 specifies the index of the track reference used to find the track from which data is to be extracted from the sample group under certain criteria. That is, the track reference index 362 identifies the track from which the data identified by the media extractor is to be extracted in the same manner as the track reference index 302.

  The group type value 364 identifies the type of sample group to which the media extractor sample group 360 corresponds. The group type value 364 generally identifies the criteria used to form the sample group of the sampling group, and the criteria in the sample group description table having the same value for the group type in the track identified by the track reference index 362. Link. Group type value 364 may comprise an integer value. In this way, the group type value of the media extractor sample group 360 may be similar to the group type of the track referenced by the track reference index 362. Alternatively, for a video time subset, the group type value 364 may be defined as “vtst”, the media extractor sample group may be defined only for that group type, and the syntax table is a “grouping_type” syntax. You will not need the element.

  The group number count value 366 may describe the number of sample groups in the media extractor track that includes the media extractor sample groups 360. A group count value 366 value of 0 may represent that all sample groups under the criteria referenced by the group type value 364 are used to form a media extractor track. The group description index 368 defines the index of the sample group entry that is used to form the media extractor track in the sample group description table.

  In accordance with the techniques of this disclosure, a sample is temporal in time such that sample A following sample B in the media extractor track indicates that sample A follows sample B in the track referenced by track reference index 362. An assembly process may be used to place all samples in the sample group entry so that they are ordered.

The following pseudo code gives an exemplary definition of a media extractor sample group class similar to the media extractor sample group 360.

  Multiplexer 30 and demultiplexer 38 may instantiate a media extractor data object using the media extractor defined in the example pseudocode above. Thus, when demultiplexer 38 retrieves data from a selected track, for example, to retrieve identified data from another track referenced by the instantiated media extractor, instantiated media extractor You can refer to

  FIG. 14 is a block diagram illustrating an example media extractor 380 that may be used in the context of a video file that conforms to the AVC file format. An example media extractor 380 includes a track reference index 382, a time identifier value 384, a reserved bit 386, and a sample offset value 388. The track reference index 382 and sample offset value 388 may be used in a similar manner as the track reference index 302 and sample offset value 304, respectively. Reserved bit 386 may be reserved for future use and is not assigned a semantic value at this time.

  The time identifier value 384 specifies the time level of the sample extracted by the media extractor 380. In one example, the time level is in the range of 0 to 7. As explained above, an encoded picture may correspond to a temporal level, which generally describes the encoding hierarchy between frames. For example, key frames (also referred to as anchor frames) can be assigned the highest time level, and frames not used as reference frames can be assigned a relatively lower time level. In this way, the media extractor 380 identifies samples extracted from the track referenced by the track reference index 382 by referring to the time level of the sample rather than explicitly identifying the sample itself. obtain. Media extractor tracks with media extractors up to a value defined by the time identifier value 384 may correspond to operating points with higher frame rates.

The following pseudo code gives an exemplary definition of a media extractor class similar to media extractor 380.

  Multiplexer 30 and demultiplexer 38 may instantiate a media extractor data object using the media extractor defined in the example pseudocode above. Thus, when demultiplexer 38 retrieves data from a selected track, for example, to retrieve identified data from another track referenced by the instantiated media extractor, instantiated media extractor You can refer to

  FIG. 15 is a block diagram illustrating an example MVC media extractor 420 that may be used to modify the MVC to include a media extractor track. An example media extractor 420 includes an optional NAL unit header 422, a track reference index 424, a sample offset 426, a continuous byte set count 428, and a loop of values that includes a data offset value 430 and a data length value 432. . The MVC media extractor 420 can be used to extract several NAL units of a subset of view components from a particular track. The example MVC media extractor 420 can skip view components in a track when extracting data from a sample of the referenced track.

  When present, NAL unit header 422 may mirror the NAL unit header of the NAL unit identified by MVC media extractor 420. That is, the syntax element of the NAL unit header 422 may be generated according to the NAL unit header syntax in the extractor or aggregator generation process defined in the MVC file format. In some examples, the extractor may not require the NAL unit header 422, for example, when a series of extractors are generated to include the relevant NAL unit header.

  The track reference index value 424 specifies the index of the track reference used to find the track from which data is to be extracted. The samples in the track from which data is extracted can be temporally aligned with the samples containing the MVC media extractor 420 in the media decoding timeline, adjusted by the offset specified by the sample offset value 426. The first track reference may be designated to receive an index value of 1, and a track reference index value of 0 may be reserved.

  Sample offset value 426 defines the offset of the sample to be extracted in the track referenced by track reference index value 424 relative to the time location of MVC media extractor 420. A sample offset value 426 value of 0 indicates that the sample to be extracted is at the same time location, -1 indicates the previous sample, +1 indicates the next sample, and so on.

  Consecutive byte set count 428 describes the number of consecutive byte sets of samples of the track from which data is to be extracted. If the continuous byte set count 428 has the value 0, the entire referenced sample in the track will be extracted. A contiguous byte set can also be referred to as a separate part of the sample.

  Data offset value 430 and data length value 432 occur in the loop. In general, the number of loop iterations, ie, the number of data offset values 430 and data length values 432, is related to the number of portions of the sample to be extracted (eg, the number of consecutive byte sets). In this way, more than one part of the sample can be extracted using the MVC media extractor 420. For each portion of the sample to be extracted, a corresponding one of the data offset values 430 indicates the start of the portion (eg, the first byte of the portion relative to the first byte of the sample) and The corresponding one indicates the length to be copied, for example the number of bytes. In some examples, a value 0 of one of the data length values 432 should copy all remaining bytes in the sample, i.e., the portion corresponds to the corresponding 1 of the data offset value 430. May correspond to the byte indicated by one and all other consecutive bytes up to the end of the sample.

The following pseudo code gives an exemplary definition of a media extractor class similar to MVC media extractor 420.

  Multiplexer 30 and demultiplexer 38 may instantiate a media extractor data object using the media extractor defined in the example pseudocode above. Thus, when demultiplexer 38 retrieves data from a selected track, for example, to retrieve identified data from another track referenced by the instantiated media extractor, instantiated media extractor You can refer to

  FIG. 16 is a block diagram illustrating another example MVC media extractor 440 that may be used to modify the MVC to include a media extractor track. The MVC media extractor 440 example identifies a specific NAL unit for extraction, as opposed to the sample specific bytes described with respect to the example of FIG. In the example of FIG. 16, the MVC media extractor 440 includes an optional NAL unit header 442, a track reference index 444, a sample offset 446, a continuous NALU (NAL unit) set count 448, a NALU offset value 450, and a continuous NAL unit. And 452 loops. NAL unit header 442, track reference index 444, and sample offset value 446 are generally defined similarly to NAL unit header 422, track reference index 424, and sample offset value 426, respectively.

  The continuous NALU set count 448 describes the number of consecutive NAL units of the sample of the track from which data is to be extracted. In some examples, if this value is set to 0, the entire referenced sample in the track is extracted.

  The NALU offset value 450 and the number of consecutive NALUs 452 occur in the loop. In general, there are as many instances of NALU offset values and consecutive NALUs as there are consecutive NALU sets, as defined by the consecutive NALU set count 448. Each NALU offset value describes the offset of the corresponding NAL unit in the sample of the track from which data is to be extracted. NAL units starting from this offset of NAL units can be extracted using this extractor. Each number of consecutive NALU values describes the total number of single referenced NAL units that should be copied for the corresponding set of NAL units.

The following pseudo code provides an exemplary definition of a media extractor class similar to MVC media extractor 440.

  Multiplexer 30 and demultiplexer 38 may instantiate a media extractor data object using the media extractor defined in the example pseudocode above. Thus, when demultiplexer 38 retrieves data from a selected track, for example, to retrieve identified data from another track referenced by the instantiated media extractor, instantiated media extractor You can refer to

  FIG. 17 is a block diagram illustrating another example MVC media extractor 460 that aggregates NAL units in the same view component when there are two or more NAL units for the view component. In that case, MVC media extractor 460 may be used to extract the identified view components. In the example of FIG. 17, the MVC media extractor 460 includes an optional NAL unit header 462, a track reference index 464, a sample offset 466, a continuous view set count 468, a view component offset value 470, and a view component count 472. Loops. NAL unit header 462, track reference index 464, and sample offset value 466 are generally defined similarly to NAL unit header 422, track reference index 424, and sample offset value 426, respectively.

  The continuous view set count 468 defines the number of consecutive view components of the identified sample in the track identified by the track reference index 464 from which data is to be extracted. Multiplexer 30 may set the continuous view set count 468 value to 0 to indicate that the entire referenced sample in the track should be extracted.

  View component offset value 470 and view component count 472 occur in a loop. In general, there are as many loop iterations as the value of the continuous view set count 468, with each loop corresponding to one of the continuous view sets. Each view component offset value 470 indicates the offset of the first view component in the sample of the track from which data for the corresponding continuous view set is to be extracted. The view component starting from this offset of the view components can then be extracted using the MVC media extractor 460. Each view component count 472 describes the total number of referenced view components in the sample to be copied for the corresponding continuous view set.

The following pseudo code gives an exemplary definition of a media extractor class similar to MVC media extractor 460.

  Multiplexer 30 and demultiplexer 38 may instantiate a media extractor data object using the media extractor defined in the example pseudocode above. Thus, when demultiplexer 38 retrieves data from a selected track, for example, to retrieve identified data from another track referenced by the instantiated media extractor, instantiated media extractor You can refer to

  FIG. 18 is a block diagram illustrating another example of an MVC media extractor 480 that may be used to reference various tracks. In the example of FIG. 18, the MVC media extractor 480 includes an optional NAL unit header 482, a continuous view set count 484, a sample offset value 486, a track reference index value 488, a view component offset value 490, and a view component count. 492 loops. The NAL unit header 482 may be defined similarly to the NAL unit header 422 and may be omitted in some examples.

  The continuous view set count 484 gives the number of continuous view components of the sample of the media extractor track from which data is to be extracted with a track reference index of track_ref_index. track_ref_index may specify the index of the track reference that should be used to find the track from which to extract data. The view component in the track from which the data is extracted is (in the media decoding timeline, adjusted using the time sample table, adjusted by the offset specified by the corresponding one of the sample offset values 486) It can be time aligned to the sample containing the MediaExtractorMVC. The first track reference may have an index value of 1, and a value of 0 may be reserved for future use.

  The example MVC media extractor 480 includes a sample offset value 486, a track reference index value 488, a view component offset value 490, and a view component count 492 in the loop. Each iteration of the loop corresponds to a particular track from which data for the sample corresponding to MVC media extractor 480 is to be extracted.

  Sample offset value 486 defines the relative index of the sample in the track that can be used as an information source, referenced by a corresponding one of the track reference index values 488. Sample 0 is the sample in the track identified by the corresponding one of the track reference index values 488 having the same or closest preceding decoding time as the sample containing MVC media extractor 480; Sample 1 is the next sample, Sample-1 is the previous sample, and so on.

  Each of the track reference index values 488 specifies the index of the track reference that should be used to find the track from which data is to be extracted for the corresponding iteration of the loop. By using multiple track reference index values, the MVC media extractor 480 may extract data from multiple different tracks.

  Each view component offset value 490 has a track reference index corresponding to a corresponding one of the track reference index values 488 in this iteration of the loop, the first view configuration in the sample of the track from which data is to be extracted. Describes the offset of the element. View components starting from this offset of the view components can be extracted using the MVC media extractor 480. In some examples, the outer loop has a nested loop structure that repeats across the track from which the sample is to be extracted, and the inner loop repeats across the sample from which the sample is to be extracted, FIG. A media extractor similar to the media extractor of FIG. 17 can be constructed. Each view component count 492 describes the number of referenced view components in a sample of tracks having a track reference index corresponding to the current value of the track reference index value 488 in this iteration of the loop.

The following pseudo code provides an exemplary definition of a media extractor class similar to MVC media extractor 480.

  Multiplexer 30 and demultiplexer 38 may instantiate a media extractor data object using the media extractor defined in the example pseudocode above. Thus, when demultiplexer 38 retrieves data from a selected track, for example, to retrieve identified data from another track referenced by the instantiated media extractor, instantiated media extractor You can refer to

  FIG. 19 is a block diagram illustrating another example MVC media extractor 500 that signals the duration of the extractor. The MVC media extractor 500 may provide one or more advantages when different samples in the media extractor track share the same syntax elements of the extractor. In the example of FIG. 19, the MVC media extractor 500 includes a sample count 502, a continuous view set count 504, a sample offset value 506, a track reference index 508, a view component offset 510, and a view component count 512. Including.

  Continuous view set count 504, sample offset value 506, track reference index 508, view component offset 510, and view component count 512 generally include continuous view set count 484, sample offset value 486, track reference index 488, view component offset. 490 and a corresponding one of view component counts 492 may be defined. Sample count 502 may define the number of consecutive samples in MVC media extractor 500 that include media extractor tracks that use the same media extractor.

The following pseudo code gives an exemplary definition of a media extractor class similar to the MVC media extractor 500.

  Multiplexer 30 and demultiplexer 38 may instantiate a media extractor data object using the media extractor defined in the example pseudocode above. Thus, when demultiplexer 38 retrieves data from a selected track, for example, to retrieve identified data from another track referenced by the instantiated media extractor, instantiated media extractor You can refer to

  FIG. 20 is a block diagram illustrating another example MVC media extractor 520 that defines a different set of extractors. For each sample in the media extractor track, the sample can use either one or more of the set of extractors, or a reference to the extractor. That is, a set of media extractors similar to the MVC media extractor 520 can be defined, each sample being one or more of the set of extractors, or an extractor, to identify a sample of another track. Any of the references to can be used.

  An example of an MVC media extractor 520 includes an extractor identifier value 522, a sample offset value 524, a track reference index value 526, a continuous view set count 528, a loop that includes a view component offset 530 and a view component count 532. including. Sample offset value 524, continuous view set count 528, view component offset 530, and view component count 532 correspond to continuous view set count 484, sample offset value 486, view component offset 490, and view component count 492. Can be defined according to one The track reference index value 526 may be defined according to the track reference index 464, for example.

  The extractor identifier value 522 defines the identifier of the extractor, ie, the MVC media extractor 520. Extractors in the same media extractor track are assigned different extractor identifier values so that samples in the media extractor track can reference extractor identifier values to use the media extractor. The reference extractor box may also be defined to include the number of extractors and the reference extractor identifier. The number of extractors value may give the number of extractors that are used to copy the data for the samples in the extractor track. When the value of the number of extractors is equal to 0, an extractor having a predetermined extractor identifier, eg, an extractor identifier equal to 0, may be used. The reference extractor identifier may give the extractor identifier of the extractor used to copy the data for the samples in the extractor track. This box can be included in a sample of a media extractor track.

The following pseudo code gives an exemplary definition of a media extractor class similar to MVC media extractor 520.

  Multiplexer 30 and demultiplexer 38 may instantiate a media extractor data object using the media extractor defined in the example pseudocode above. Thus, when demultiplexer 38 retrieves data from a selected track, for example, to retrieve identified data from another track referenced by the instantiated media extractor, instantiated media extractor You can refer to

The following pseudo code gives an exemplary definition of a reference extractor box class for the reference extractor box described above.

  FIG. 21 is a block diagram illustrating an example MVC media extractor 550 that may be formed using map sample groups. The example MVC media extractor 550 specifies a NAL unit group from a series of sample entries, each giving a contiguous NAL unit in a map sample group. In the example of FIG. 22, the MVC media extractor 550 includes a NALU group count 552 and a loop that includes a track index 554, a group description index 556, a NALU start map sample 558, and a NALU view count 560.

  The NALU group count 552 specifies the number of NAL unit groups from the map sample group entry in the reference track. The track reference index value 554 specifies the index of the track reference to be used to find the track from which data for each corresponding iteration of the loop is to be extracted. The group description index 556 specifies the index of the map sample group entry that is used to form the NAL unit group for each corresponding iteration of the loop. The NALU start map sample 558 specifies the offset of the NAL unit in the map sample group, each having a corresponding one map sample entry index of the group description index 556 in the corresponding iteration of the loop. The NALU view count 560 specifies the number of consecutive NAL units to be extracted into the media extractor in the map sample group with the corresponding one map sample entry index of the group description index 556 in the corresponding iteration of the loop. To do.

The following pseudo code provides an exemplary definition of a media extractor class similar to MVC media extractor 550.

  Multiplexer 30 and demultiplexer 38 may instantiate a media extractor data object using the media extractor defined in the example pseudocode above. Thus, when demultiplexer 38 retrieves data from a selected track, for example, to retrieve identified data from another track referenced by the instantiated media extractor, instantiated media extractor You can refer to

  The techniques of this disclosure may include an assembly process for constructing sample view components in a sample group. The view component in the sample of the sample group entry is the sample in the media extractor track if the sample A follows sample B in the original track (with the index of the track reference index). Following the view component in B, if sample A has a decoding time before sample B, the view component in sample A follows the view component in sample B in the media extractor track, and Two view components in the same sample follow the order of presentation in the syntax table of the media extractor map sample group, and if two view components in the same sample of a track belong to the same group of NAL units, ie They are When extracted by the same loop syntax elements in a tractor map sample group, they follow the original order and if two view components are extracted from samples in different tracks but have the same timestamp, they Will order in a timely manner to follow the order of the order index specified in the view identifier box.

  FIG. 22 is a block diagram illustrating an exemplary modified 3GPP track selection box 390 that signals additional attributes of the track selection box. At the time of this writing, the most recent 3GPP standard specifies an AttributeList containing attributes describing language, bandwidth, codec, screen size, maximum packet size, and media type. The attribute list 392 of the 3GPP track selection box 390 includes a language value 394, a bandwidth value 396, a codec value 398, and a screen size value 400, and signals these attributes according to existing 3GPP standards. Further, the techniques of this disclosure may modify an existing 3GPP track selection box to include a frame rate value 406, a time identifier value 408, possibly a display view value 410, and an output view list value 412.

  The language value 394 defines the value of the group type LANG of the “alt group” attribute in the session level SDP, which is defined in Chapter 5.3.3.4 of the existing 3GPP standard. The bandwidth value 396 defines the value of the “b = AS” attribute in the media level SDP. Codec value 398 defines the SampleEntry value in the sample description box of the media track. Screen size value 400 defines the width and height fields of the MP4VisualSampleEntry value and the H263SampleEntry value in the media track. The maximum packet size value 402 defines the value of the MaxPacketSize field in RTPHintSampleEntry, eg, in the RTP hint track. Media type value 404 describes the HandlerType in the handler box of the media track. In general, these values correspond to existing 3GPP standards.

  Frame rate value 406 describes the frame rate of the video track or media extractor track corresponding to 3GPP track selection box 390. The time identifier value 408 corresponds to the time identifier of the video track corresponding to the 3gPP track selection box 390 and may depend on the track with the lower time identifier value. In some examples, the multiplexer 30 may indicate that the value is not specified by setting the value of the time identifier value 408 to a preconfigured “unspecified” value, eg, 8. In general, multiplexer 30 may indicate that the value of time identifier value 408 for a non-video track is not specified. In some examples, multiplexer 30 also indicates that the value of time identifier value 408 is not specified when the corresponding video track does not include a media extractor and / or is not referenced by other tracks as a time subset. Can show.

  In an example where MVC is considered in 3GPP, multiplexer 30 may include additional attributes of display view number value 410 and output view list value 412. In such an example, multiplexer 30 may omit time identifier value 408. The display view number value 410 describes the number of views to be output for the corresponding track. For example, when a view to be displayed is encoded with reference to a view that is not displayed, the number of views to be output and the number of views to be decoded are not necessarily the same. The output view list value 412 may define a list of N view identifiers that identify the N views to be output.

  FIG. 23 is a flowchart illustrating an exemplary method for using a media extractor according to the techniques of this disclosure. Initially, a source device, such as A / V source device 20 (FIG. 1), constructs a video track for a file that conforms to the file format in accordance with the techniques of this disclosure. That is, multiplexer 30 assembles the encoded video data in the track so that the video track includes encoded video samples that include one or more NAL units (600). Multiplexer 30 also builds an extractor that references some or all of one or more NAL units of the video track (602) and builds an extractor track that includes the extractor (604). In addition, multiplexer 30 may include encoded video samples in media extractor tracks and in additional tracks that include encoded video samples and / or media extractors.

  Next, the multiplexer 30 outputs the file (606). The file can be output to a signal via a transmitter, transceiver, network interface, modem, or other signal output means, or the file can be a USB interface, magnetic media recorder, optical recorder, or other hardware interface Or the like via a hardware interface.

  A / V destination device 40 ultimately receives the file (608), for example, by receiving a signal or reading a storage medium. Demultiplexer 38 selects one of the two (or more) tracks to be decoded (610). The demultiplexer 38 may select one of the tracks based on the decoding function of the video decoder 48, the rendering function of the video output 44, or other criteria. When an extractor track is selected, the demultiplexer 38 may retrieve the NAL unit referenced by the extractor in the extractor track from the track where the encoded video sample identified by the extractor is stored.

  The demultiplexer 38 may discard encoded video samples (or other NAL units) that are not in the selected track and are not identified by at least one extractor in the selected track. That is, demultiplexer 38 may avoid sending such encoded video samples to video decoder 48 so that the task of decoding unused video data need not be provided to video decoder 48.

  In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media may include computer-readable storage media such as data storage media or communication media including any medium that enables transfer of a computer program from one place to another. A data storage medium may be any available that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementation of the techniques described in this disclosure. It can be a medium. By way of example, and not limitation, such computer readable storage media may be in the form of RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage device, flash memory, or instruction or data structure. Any other medium that can be used to store the desired program code and that can be accessed by a computer can be provided. Any connection is also properly termed a computer-readable medium. For example, instructions may be sent from a website, server, or other remote source using coaxial technology, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, wireless, and microwave. When transmitted, coaxial technologies, fiber optic cables, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the media definition. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other temporary media. Discs and discs used in this specification are compact discs (CD), laser discs, optical discs, digital versatile discs (DVD), floppy discs. (Disk) and Blu-ray (registered trademark) disc, in which case the disc typically reproduces data magnetically and the disc optically reproduces data with a laser. Combinations of the above should also be included within the scope of computer-readable media.

  The instructions encoded in a computer readable medium may include one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs). ) Or other equivalent integrated or discrete logic circuit. Thus, as used herein, the term “processor” may refer to either the structure described above or other structure suitable for implementation of the techniques described herein. Further, in some aspects, the functionality described herein may be provided in dedicated hardware and / or software modules configured for encoding and decoding, or may be incorporated into a composite codec. The techniques may also be fully implemented in one or more circuits or logic elements.

  The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (eg, a chip set). Although this disclosure has described various components, modules, or units in order to highlight the functional aspects of a device that is configured to perform the disclosed techniques, It is not necessarily realized by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit, including one or more processors, as described above, with suitable software and / or firmware, or mutually. It can be given by a set of operating hardware units.

Various examples have been described. These and other examples are within the scope of the following claims.
Hereinafter, the invention described in the scope of claims of the present application will be appended.
[1]
A method for encoding video data, said method comprising:
Constructing, by a source video device, a first track comprising video samples comprising a plurality of network access layer (NAL) units based on the encoded video data, wherein the video samples are in the access units Building a first track included;
Constructing, by the source video device, a second track that includes an extractor that identifies at least one of the plurality of NAL units in the video samples of the first track, the plurality of the plurality of NAL units; The at least one of the NAL units comprises a first identified NAL unit, wherein the extractor identifies a second NAL unit of the access unit, and the first identified NAL unit and the second Constructing a second track that is discontinuous with the identified NAL units;
Including the first track and the second track in a video file that conforms at least in part to an International Organization for Standardization (ISO) base media file format;
Outputting the video file;
A method comprising:
[2]
The method according to [1], wherein the video file conforms to the ISO base media file format.
[3]
The video file includes a scalable video coding (SVC) file format, an advanced video coding (AVC) file format, a third generation partnership project (3GPP) file format, and a multi-view video coding (MVC) file format. The method according to [1], which conforms to at least one.
[4]
Constructing the second track may include one or more additional NAL units not included in the plurality of NAL units of the first track based on the encoded data. The method according to [1], further comprising inclusion in two tracks.
[5]
Identifying a first extractor identifying one or more of the plurality of NAL units of the first track and at least one of the one or more NAL units of the second track; The method according to [4], further comprising: constructing a third track including the second extractor.
[6]
Building the third track further comprises including in the third track one or more NAL units not included in the first track and the second track; [5].
[7]
Constructing the second track comprises constructing the extractor to identify each of the plurality of NAL units of the video samples of the first track, wherein the extractor includes a destination device The method according to [1], wherein each of the plurality of NAL units of the video sample is extracted as a whole.
[8]
Building the second track includes specifying the one or more byte ranges of the plurality of NAL units of the plurality of NAL units of the video sample in the first track of the video file. The method of [1], comprising constructing the extractor to identify the one or more of the plurality of NAL units of a sample.
[9]
The plurality of NAL units of the video sample in the first track include a slice of a common picture, a non-video coding layer (VCL) NAL unit, a supplemental enhancement information (SEI) message NAL unit, and the access unit The method of [1], comprising at least one of a video layer, a different view component of the access unit, and a NUL unit aggregated from a plurality of NAL units.
[10]
The plurality of NAL units comprises a first plurality of NAL units, and the method further comprises constructing a third track including a second plurality of NAL units based on the encoded video data. The second plurality of NAL units form part of the access unit, the second plurality of NAL units comprising the second identified NAL unit identified by the extractor; [1].
[11]
The video sample comprises a first video sample, the plurality of NAL units comprises a first plurality of NAL units, and the first track further comprises a second sample comprising a second plurality of NAL units. The method of [1], wherein the access unit comprises the second sample, and the second plurality of NAL units comprises the second NAL unit identified by the extractor.
[12]
Among the plurality of NAL units of the video sample of the first track, wherein the second NAL unit is separated from the first identified NAL unit in the video sample by at least one byte of data. The method according to [1], comprising the second NAL unit.
[13]
Based on the characteristics of each track, the first track and the second track are such that either the first track or the second track can be selected for decoding by a destination device. The method according to [1], wherein a switch group is formed.
[14]
Building the second track,
Signaling the frame rate of the second track;
Signaling a time identifier of the video sample of the first track for the second track;
Building the second track when the second track comprises more than one view;
Signaling a value representing the number of views to be displayed after decoding the second track;
Signaling one or more view identifier values representing a view to be displayed for the second track;
Signaling a value representing the number of views to be decoded for the second track;
The method according to [13], further comprising:
[15]
An apparatus for encoding video data, the apparatus comprising:
An encoder configured to encode video data;
Constructing a first track comprising video samples comprising a plurality of network access layer (NAL) units based on the encoded video data, wherein the video samples are included in an access unit; Building one track and building a second track including an extractor identifying at least one of the plurality of NAL units in the video samples of the first track, The at least one of the plurality of NAL units comprises a first identified NAL unit, the extractor identifying a second NAL unit of the access unit, and the first identified NAL unit; Construct a second track that is discontinuous with the second identified NAL unit And a multiplexer configured to include the first track and the second track in a video file that is at least partially compliant with an International Organization for Standardization (ISO) base media file format; ,
An output interface configured to output the video file;
An apparatus comprising:
[16]
The video file includes an ISO base media file format, a scalable video coding (SVC) file format, an advanced video coding (AVC) file format, a third generation partnership project (3GPP) file format, and a multi-view video coding (MVC). The device according to [15], which conforms to at least one of a file format.
[17]
The multiplexer is configured to include one or more NAL units in the second track that are not included in the first track based on the encoded video data. ] The apparatus of description.
[18]
A first extractor that identifies one or more of the plurality of NAL units of the first track; and one or more of the plurality of NAL units of the second track. The apparatus according to [17], wherein the apparatus is configured to construct a third track including a second extractor that identifies
[19]
The extractor comprises a first extractor, and the multiplexer is configured to construct a third extractor track including a plurality of NAL units based on the encoded video data; [15], wherein the second track is configured to include a second extractor that identifies one or more of the plurality of NAL units of the third track. Equipment.
[20]
The device is
An integrated circuit;
A microprocessor;
A wireless communication device including the video encoder and the multiplexer;
The apparatus according to [16], comprising at least one of the following.
[21]
An apparatus for encoding video data, the apparatus comprising:
Means for constructing a first track comprising video samples comprising a plurality of network access layer (NAL) units based on the encoded video data, wherein the video samples are included in the access units; Means for constructing the first track;
Means for constructing a second track that includes an extractor that identifies at least one of the plurality of NAL units in the video sample of the first track, the method comprising: The at least one of the plurality comprises a first identified NAL unit, and the extractor identifies a second NAL unit of the access unit, the first identified NAL unit and the second identified Means for constructing a second track that is non-contiguous with the NAL unit;
Means for including the first track and the second track in a video file that conforms at least in part to an International Organization for Standardization (ISO) base media file format;
Means for outputting the video file;
An apparatus comprising:
[22]
The video file includes an ISO base media file format, a scalable video coding (SVC) file format, an advanced video coding (AVC) file format, a third generation partnership project (3GPP) file format, and a multi-view video coding (MVC). The device according to [21], which conforms to at least one of a file format.
[23]
The method of [21], further comprising means for including, in the second track, one or more NAL units that are not included in the first track based on the encoded data. apparatus.
[24]
Identifying a first extractor identifying one or more of the plurality of NAL units of the first track and at least one of the one or more NAL units of the second track; The apparatus according to [23], further comprising means for constructing a third track including the second extractor.
[25]
The extractor comprises a first extractor, and the apparatus further comprises means for constructing a third extractor track comprising a plurality of NAL units based on the encoded video data; Building the second track such that the means for building a second track includes a second extractor that identifies one or more of the plurality of NAL units of the third track. The device according to [21], comprising means for
[26]
When executed
Constructing a first track including video samples comprising a plurality of network access layer (NAL) units based on the encoded video data, wherein the video samples are included in the access units; Building a track for
Constructing a second track including an extractor that identifies at least one of the plurality of NAL units in the video samples of the first track, wherein the plurality of NAL units At least one comprising a first identified NAL unit, wherein the extractor identifies a second NAL unit of the access unit, the first identified NAL unit and the second identified NAL unit Constructing a second track, wherein and are discontinuous;
Including the first track and the second track in a video file that conforms at least in part to an International Organization for Standardization (ISO) base media file format;
Outputting the video file;
A computer-readable storage medium comprising instructions for causing a processor to execute.
[27]
The video file includes an ISO base media file format, a scalable video coding (SVC) file format, an advanced video coding (AVC) file format, a third generation partnership project (3GPP) file format, and a multi-view video coding (MVC). The computer-readable storage medium according to [26], which conforms to at least one of file formats.
[28]
Further comprising instructions that cause the processor to include in the second track one or more NAL units not included in the first track based on the encoded data. 26].
[29]
Identifying a first extractor identifying one or more of the plurality of NAL units of the first track and at least one of the one or more NAL units of the second track; The computer-readable storage medium according to [28], further comprising instructions for causing the processor to construct a third track including the second extractor.
[30]
In the processor, the extractor comprises a first extractor, and the computer readable storage medium constructs a third extractor track including a plurality of NAL units based on the encoded video data. A second instruction that identifies one or more of the plurality of NAL units of the third track, wherein the instruction causes the processor to perform the building of the second track. [26] The computer-readable storage medium according to [26], comprising instructions for causing the processor to construct the second track so as to include an extractor.
[31]
A method for decoding video data, said method comprising:
Receiving a video file at least partially compliant with an International Organization for Standardization (ISO) base media file format by a demultiplexer of a destination device, the video file comprising a first track and a second track; The first track includes a video sample comprising a plurality of network access layer (NAL) units corresponding to the encoded video data, the video sample is included in the access unit, and the second track is An extractor that identifies at least one of the plurality of NAL units of the first track, wherein the at least one of the plurality of NAL units comprises a first identified NAL unit; An extractor is connected to the second of the access unit Identifies the AL unit, the first and identified NAL unit and the second identified NAL units are non-consecutive, and receiving,
Selecting the second track to be decoded;
Sending the encoded video data of the first NAL unit and the second NAL unit identified by the extractor of the second track to a video decoder of the destination device;
A method comprising:
[32]
The method of [31], further comprising discarding each of the plurality of NAL units of the first track that is not identified by the extractor of the second track.
[33]
The second track further comprises one or more NAL units not included in the first track, and the method encodes the one or more NAL units of the second track. The method according to [31], further comprising: sending the processed video data to the video decoder.
[34]
The video file further comprises a third track including a plurality of NAL units corresponding to the encoded video data, and the method includes the encoded video data of the plurality of NAL units of the third track. The method of [31], further comprising: sending to the video decoder.
[35]
An apparatus for decoding video data, the apparatus comprising:
A video decoder configured to decode the video data;
Receiving a video file that is at least partially compliant with an International Organization for Standardization (ISO) base media file format, the video file comprising a first track and a second track, wherein the first track is , Comprising a video sample comprising a plurality of network access layer (NAL) units corresponding to the encoded video data, wherein the video sample is included in the access unit, and wherein the second track is the first track An extractor that identifies at least one of the plurality of NAL units, wherein the at least one of the plurality of NAL units comprises a first identified NAL unit, wherein the extractor includes the access unit A second NAL unit is identified and said first identified Receiving a non-contiguous NAL unit and the second identified NAL unit, selecting the second track to be decoded, and by the extractor of the second track And a demultiplexer configured to send the encoded video data of the identified first NAL unit and the second NAL unit to the video decoder.
[36]
The apparatus of [35], wherein the demultiplexer is configured to discard each of the plurality of NAL units of the first track that is not identified by the extractor of the second track.
[37]
The second track further comprises one or more NAL units not included in the first track, and the demultiplexer includes a code of the one or more NAL units of the second track. [35] The apparatus according to [35], configured to send the converted video data to the video decoder.
[38]
The video file further comprises a third track including a plurality of NAL units corresponding to encoded video data, and the demultiplexer encodes the plurality of NAL units of the third track. The apparatus of [35], configured to send data to the video decoder.
[39]
An apparatus for decoding video data, the apparatus comprising:
Means for receiving a video file that conforms at least in part to an International Organization for Standardization (ISO) base media file format, the video file comprising a first track and a second track, wherein the first track A track includes a video sample comprising a plurality of network access layer (NAL) units corresponding to encoded video data, the video sample is included in the access unit, and the second track is the first An extractor that identifies at least one of the plurality of NAL units of a track, wherein the at least one of the plurality of NAL units comprises a first identified NAL unit, wherein the extractor includes the access A second NAL unit of the unit is identified and said first NAL unit Another has been NAL unit and the second identified NAL units are non-consecutive, and means for receiving,
Means for selecting the second track to be decoded;
Means for sending the encoded video data of the first NAL unit and the second NAL unit identified by the extractor of the second track to a video decoder of the device;
An apparatus comprising:
[40]
The apparatus of [39], further comprising means for discarding each of the plurality of NAL units of the first track that is not identified by the extractor of the second track.
[41]
The second track further comprises one or more NAL units not included in the first track, and the apparatus encodes the one or more NAL units of the second track. [39] The apparatus of [39], further comprising means for sending the processed video data to the video decoder.
[42]
The video file further comprises a third track including a plurality of NAL units corresponding to the encoded video data, and the apparatus encodes the encoded video data of the plurality of NAL units of the third track. The apparatus of [39], further comprising means for sending to the video decoder.
[43]
When executed
When receiving a video file that conforms at least in part to an International Organization for Standardization (ISO) base media file format, selecting the second track to be decoded, wherein the video file is A second track, wherein the first track comprises a video sample comprising a plurality of network access layer (NAL) units corresponding to the encoded video data, the video sample being included in the access unit The second track includes an extractor that identifies at least one of the plurality of NAL units of the first track, wherein the at least one of the plurality of NAL units is first identified. The NAL unit, and the extractor is the access unit And to identify a second NAL unit, and the first identified NAL unit and the second identified NAL units are non-consecutive, selecting the second track,
Sending the encoded video data of the first NAL unit and the second NAL unit identified by the extractor of the second track to a video decoder;
A computer-readable storage medium comprising instructions for causing a processor to execute.
[44]
[43] The computer readable storage medium of [43], further comprising discarding each of the plurality of NAL units of the first track not identified by the extractor of the second track.
[45]
The second track further comprises one or more NAL units not included in the first track, and the method encodes the one or more NAL units of the second track. The computer-readable storage medium according to [43], further comprising sending the processed video data to the video decoder.
[46]
The video file further comprises a third track including a plurality of NAL units corresponding to the encoded video data, and the method includes the encoded video data of the plurality of NAL units of the third track. The computer-readable storage medium according to [43], further comprising: sending to the video decoder.

Claims (46)

A method for encoding video data, said method comprising:
Constructing, by a source video device, a first track comprising video samples comprising a plurality of network access layer (NAL) units based on the encoded video data, wherein the video samples are in the access units Building a first track included;
Constructing a second track including a plurality of extractors by the source video device, the plurality of extractors including an extractor identifying a plurality of NAL units of the first track; The plurality of identified NAL units includes a first identified NAL unit of the NAL units in the video sample of the first track and a second identified NAL unit of the access unit. , wherein Ri said first identified NAL unit and the second identified NAL units and non-contiguous der, the extractor identifies the first NAL unit and the second NAL unit and constructing a second track,
Including the first track and the second track in a video file that conforms at least in part to an International Organization for Standardization (ISO) base media file format;
Outputting the video file.   The method of claim 1, wherein the video file is compliant with the ISO base media file format.   The video file includes a scalable video coding (SVC) file format, an advanced video coding (AVC) file format, a third generation partnership project (3GPP) file format, and a multi-view video coding (MVC) file format. The method of claim 1, wherein the method is compliant with at least one.   Constructing the second track may include one or more additional NAL units not included in the plurality of NAL units of the first track based on the encoded data. The method of claim 1, further comprising including in two tracks.   Identifying a first extractor identifying one or more of the plurality of NAL units of the first track and at least one of the one or more NAL units of the second track; 5. The method of claim 4, further comprising constructing a third track that includes a second extractor.   The building of the third track further comprises including in the third track one or more NAL units not included in the first track and the second track. Item 6. The method according to Item 5.   Constructing the second track comprises constructing the extractor to identify each of the plurality of NAL units of the video samples of the first track, wherein the extractor includes a destination device The method of claim 1, further comprising: extracting each of the plurality of NAL units of the video sample as a whole.   Building the second track includes specifying the one or more byte ranges of the plurality of NAL units of the plurality of NAL units of the video sample in the first track of the video file. The method of claim 1, comprising constructing the extractor to identify the one or more of the plurality of NAL units of a sample.   The plurality of NAL units of the video samples in the first track include a slice of a common picture, a non-video coding layer (VCL) NAL unit, a supplemental enhancement information (SEI) message NAL unit, and the access unit The method of claim 1, comprising at least one of a video layer, a different view component of the access unit, and a NAL unit aggregated from a plurality of NAL units.   The plurality of NAL units comprises a first plurality of NAL units, and the method further comprises constructing a third track including a second plurality of NAL units based on the encoded video data. The second plurality of NAL units form part of the access unit, and the second plurality of NAL units comprises the second identified NAL unit identified by the extractor. Item 2. The method according to Item 1.   The video sample comprises a first video sample, the plurality of NAL units comprises a first plurality of NAL units, and the first track further comprises a second sample comprising a second plurality of NAL units. The method of claim 1, wherein the access unit comprises the second sample, and the second plurality of NAL units comprises the second NAL unit identified by the extractor.   Among the plurality of NAL units of the video sample of the first track, wherein the second NAL unit is separated from the first identified NAL unit in the video sample by at least one byte of data. The method of claim 1 comprising a second NAL unit.   Based on the characteristics of each track, the first track and the second track are such that either the first track or the second track can be selected for decoding by a destination device. The method of claim 1, wherein a switch group is formed. Building the second track,
Signaling the frame rate of the second track;
Signaling a time identifier of the video sample of the first track for the second track;
Building the second track when the second track comprises more than one view;
Signaling a value representing the number of views to be displayed after decoding the second track;
And to signal one or more views identifier value for the display view to be for the second track,
14. The method of claim 13, further comprising signaling a value representing a number of views to be decoded for the second track. An apparatus for encoding video data, the apparatus comprising:
An encoder configured to encode video data;
Constructing a first track comprising video samples comprising a plurality of network access layer (NAL) units based on the encoded video data, wherein the video samples are included in an access unit; Constructing one track and constructing a second track including a plurality of extractors, wherein the plurality of extractors includes an extractor that identifies a plurality of NAL units of the first track. The plurality of identified NAL units includes: a first identified NAL unit of the NAL units in the video sample of the first track; and a second identified NAL of the access unit. It includes a unit, wherein the first identified NAL unit and the second identification Ri NAL units and non-contiguous der that the extractor identifies the first NAL unit and the second NAL unit, and configured multiplexer to build the second track,
An output interface configured to output the video file, wherein the multiplexer at least partially converts the first track and the second track to an International Organization for Standardization (ISO) base media file format. An apparatus further configured to perform inclusion in a compliant video file .   The video file includes an ISO base media file format, a scalable video coding (SVC) file format, an advanced video coding (AVC) file format, a third generation partnership project (3GPP) file format, and a multi-view video coding (MVC). 16. The apparatus of claim 15, wherein the apparatus conforms to at least one of a file format.   The multiplexer is configured to include one or more NAL units in the second track that are not included in the first track based on the encoded video data. 15. The apparatus according to 15.   A first extractor that identifies one or more of the plurality of NAL units of the first track; and one or more of the plurality of NAL units of the second track. The apparatus of claim 17, wherein the apparatus is configured to construct a third track including a second extractor that identifies the second extractor.   The extractor comprises a first extractor, and the multiplexer is configured to build a third extractor track including a plurality of NAL units based on the encoded video data, the multiplexer 16. The second track is configured to include a second extractor that identifies one or more of the plurality of NAL units of the third track. Equipment. The device is
An integrated circuit;
A microprocessor;
The apparatus of claim 16, comprising at least one of a wireless communication device including the video encoder and the multiplexer. An apparatus for encoding video data, the apparatus comprising:
Means for constructing a first track comprising video samples comprising a plurality of network access layer (NAL) units based on the encoded video data, wherein the video samples are included in the access units; Means for constructing the first track;
And means for constructing the second track including a plurality of extractors, a plurality of the plurality of extractor which comprises extractor identifying a plurality of NAL units of the first track, which is the identification NAL units of the first track include a first identified NAL unit of the NAL units in the video sample of the first track and a second identified NAL unit of the access unit, wherein wherein Ri first identified NAL unit and the second identified NAL units and non-contiguous der, the extractor identifies the first NAL unit and the second NAL unit, the second Means for building a track;
Means for including the first track and the second track in a video file that conforms at least in part to an International Organization for Standardization (ISO) base media file format;
Means for outputting the video file.   The video file includes an ISO base media file format, a scalable video coding (SVC) file format, an advanced video coding (AVC) file format, a third generation partnership project (3GPP) file format, and a multi-view video coding (MVC). 23. The apparatus of claim 21, wherein the apparatus conforms to at least one of a file format.   23. The means of claim 21, further comprising means for including in the second track one or more NAL units not included in the first track based on the encoded data. apparatus.   Identifying a first extractor identifying one or more of the plurality of NAL units of the first track and at least one of the one or more NAL units of the second track; 24. The apparatus of claim 23, further comprising means for building a third track that includes a second extractor.   The extractor comprises a first extractor, and the apparatus further comprises means for constructing a third extractor track comprising a plurality of NAL units based on the encoded video data; Building the second track such that the means for building a second track includes a second extractor that identifies one or more of the plurality of NAL units of the third track. The apparatus of claim 21, comprising means for: When executed
Constructing a first track including video samples comprising a plurality of network access layer (NAL) units based on the encoded video data, wherein the video samples are included in the access units; Building a track for
The method comprising: constructing a second track including a plurality of extractors, the plurality of extractors includes extractor identifying a plurality of NAL units of the first track, a plurality of NAL said identified A unit includes a first identified NAL unit of the NAL units in the video sample of the first track and a second identified NAL unit of the access unit, wherein the first 1 of the identified NAL unit and the second identified NAL units and non-contiguous der is, the extractor identifies the first NAL unit and the second NAL unit, the second track Building,
Including the first track and the second track in a video file that conforms at least in part to an International Organization for Standardization (ISO) base media file format;
A computer readable storage medium comprising instructions for causing a processor to output the video file.   The video file includes an ISO base media file format, a scalable video coding (SVC) file format, an advanced video coding (AVC) file format, a third generation partnership project (3GPP) file format, and a multi-view video coding (MVC). 27. The computer readable storage medium of claim 26, compliant with at least one of file format.   And further comprising instructions for causing the processor to include one or more NAL units in the second track that are not included in the first track based on the encoded data. Item 27. The computer-readable storage medium according to Item 26.   Identifying a first extractor identifying one or more of the plurality of NAL units of the first track and at least one of the one or more NAL units of the second track; 29. The computer readable storage medium of claim 28, further comprising instructions that cause the processor to construct a third track that includes a second extractor that is configured.   In the processor, the extractor comprises a first extractor, and the computer readable storage medium constructs a third extractor track including a plurality of NAL units based on the encoded video data. A second instruction that identifies one or more of the plurality of NAL units of the third track, wherein the instruction causes the processor to perform the building of the second track. 27. The computer readable storage medium of claim 26, comprising instructions that cause the processor to construct the second track to include an extractor. A method for decoding video data, said method comprising:
Receiving a video file at least partially compliant with an International Organization for Standardization (ISO) base media file format by a demultiplexer of a destination device, the video file comprising a first track and a second track; The first track includes a video sample comprising a plurality of network access layer (NAL) units corresponding to the encoded video data, the video sample is included in the access unit, and the second track is includes a plurality of extractors, the plurality of extractors includes extractor identifying a plurality of NAL units of the first track, the identified plurality of NAL units, said first track The first identified NAL unit of the NAL unit It includes a dot and a second identified NAL unit of the access unit, wherein the, Ri said first identified NAL unit and the second identified NAL units and non-contiguous der, wherein The extractor receives , identifying the first NAL unit and the second NAL unit ;
Selecting the second track to be decoded;
Sending the encoded video data of the first NAL unit and the second NAL unit identified by the extractor of the second track to a video decoder of the destination device.   32. The method of claim 31, further comprising discarding each of the plurality of NAL units of the first track that is not identified by the extractor of the second track.   The second track further comprises one or more NAL units not included in the first track, and the method encodes the one or more NAL units of the second track. 32. The method of claim 31, further comprising sending processed video data to the video decoder.   The video file further comprises a third track including a plurality of NAL units corresponding to the encoded video data, and the method includes the encoded video data of the plurality of NAL units of the third track. 32. The method of claim 31, further comprising: sending to the video decoder. An apparatus for decoding video data, the apparatus comprising:
A video decoder configured to decode the video data;
Receiving a video file that is at least partially compliant with an International Organization for Standardization (ISO) base media file format, the video file comprising a first track and a second track, wherein the first track is , Comprising video samples comprising a plurality of network access layer (NAL) units corresponding to the encoded video data, the video samples being included in the access units, and the second track comprising a plurality of extractors The plurality of extractors includes an extractor that identifies a plurality of NAL units of the first track, and the plurality of identified NAL units is a first of the NAL units of the first track. Identified NAL units and a second identifier of the access unit. It includes been NAL unit, wherein said first identified NAL unit and the second identified NAL unit Ri discontinuous der, the extractor, first NAL unit and the second A demultiplexer configured to receive , identifying the NAL unit , wherein the demultiplexer selects the second track to be decoded and by the extractor of the second track. the encoded video data of the identified first NAL unit and the second NAL unit Ru is further configured to perform the sending to the video decoder apparatus.   36. The apparatus of claim 35, wherein the demultiplexer is configured to discard each of the plurality of NAL units of the first track that is not identified by the extractor of the second track.   The second track further comprises one or more NAL units not included in the first track, and the demultiplexer includes a code of the one or more NAL units of the second track. 36. The apparatus of claim 35, configured to send digitized video data to the video decoder.   The video file further comprises a third track including a plurality of NAL units corresponding to encoded video data, and the demultiplexer encodes the plurality of NAL units of the third track. 36. The apparatus of claim 35, configured to send data to the video decoder. An apparatus for decoding video data, the apparatus comprising:
Means for receiving a video file that conforms at least in part to an International Organization for Standardization (ISO) base media file format, the video file comprising a first track and a second track, wherein the first track The track includes a video sample comprising a plurality of network access layer (NAL) units corresponding to the encoded video data, the video sample is included in the access unit, and the second track is a plurality of extractors And the plurality of extractors includes an extractor that identifies a plurality of NAL units of the first track, and the plurality of identified NAL units are the NAL units of the first track. A first identified NAL unit and a first of the access units; Of including the identified NAL unit, wherein the first identified NAL unit and the second identified NAL units and non-contiguous der is, the extractor includes a first NAL unit Means for receiving , identifying a second NAL unit ;
Means for selecting the second track to be decoded;
Means for sending the encoded video data of the first NAL unit and the second NAL unit identified by the extractor of the second track to a video decoder of the apparatus.   40. The apparatus of claim 39, further comprising means for discarding each of the plurality of NAL units of the first track that is not identified by the extractor of the second track.   The second track further comprises one or more NAL units not included in the first track, and the apparatus encodes the one or more NAL units of the second track. 40. The apparatus of claim 39, further comprising means for sending processed video data to the video decoder.   The video file further comprises a third track including a plurality of NAL units corresponding to the encoded video data, and the apparatus encodes the encoded video data of the plurality of NAL units of the third track. 40. The apparatus of claim 39, further comprising means for sending a message to the video decoder. When executed
When receiving a video file that conforms at least in part to an International Organization for Standardization (ISO) base media file format, selecting the second track to be decoded, wherein the video file is A second track, wherein the first track comprises a video sample comprising a plurality of network access layer (NAL) units corresponding to the encoded video data, the video sample being included in the access unit said second track comprises a plurality of extractors, the plurality of extractors includes extractor identifying a plurality of NAL units of the first track, the identified plurality of NAL units, The first identified of the NAL units of the first track Includes a AL unit, the second identified NAL unit of the access unit, wherein the, Ri said first identified NAL unit and the second identified NAL units and non-contiguous der, wherein The extractor selects the second track that identifies the first NAL unit and the second NAL unit ;
Computer readable comprising instructions for causing a processor to send encoded video data of the first NAL unit and the second NAL unit identified by the extractor of the second track to a video decoder. Storage medium.   44. The computer readable storage medium of claim 43, further comprising discarding each of the plurality of NAL units of the first track that is not identified by the extractor of the second track.   The second track further comprises one or more NAL units not included in the first track, and the method encodes the one or more NAL units of the second track. 44. The computer readable storage medium of claim 43, further comprising sending processed video data to the video decoder.   The video file further comprises a third track including a plurality of NAL units corresponding to the encoded video data, and the method includes the encoded video data of the plurality of NAL units of the third track. 44. The computer readable storage medium of claim 43, further comprising: sending to the video decoder.

Images