EP2596643B1 - Arranging sub-track fragments for streaming video data - Google Patents

Arranging sub-track fragments for streaming video data Download PDF

Info

Publication number
EP2596643B1
EP2596643B1 EP11736528.8A EP11736528A EP2596643B1 EP 2596643 B1 EP2596643 B1 EP 2596643B1 EP 11736528 A EP11736528 A EP 11736528A EP 2596643 B1 EP2596643 B1 EP 2596643B1
Authority
EP
European Patent Office
Prior art keywords
sub
video
track
fragment
fragments
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP11736528.8A
Other languages
German (de)
English (en)
French (fr)
Other versions
EP2596643A1 (en
Inventor
Ying Chen
Marta Karczewicz
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Qualcomm Inc
Original Assignee
Qualcomm Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Qualcomm Inc filed Critical Qualcomm Inc
Publication of EP2596643A1 publication Critical patent/EP2596643A1/en
Application granted granted Critical
Publication of EP2596643B1 publication Critical patent/EP2596643B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234327—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by decomposing into layers, e.g. base layer and one or more enhancement layers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/40—Support for services or applications
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/70—Media network packetisation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/75—Media network packet handling
    • H04L65/764—Media network packet handling at the destination 
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
    • H04N21/23608—Remultiplexing multiplex streams, e.g. involving modifying time stamps or remapping the packet identifiers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/845—Structuring of content, e.g. decomposing content into time segments
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/845—Structuring of content, e.g. decomposing content into time segments
    • H04N21/8456—Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/61—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio
    • H04L65/612—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio for unicast
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/75—Media network packet handling
    • H04L65/762—Media network packet handling at the source 
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/04—Protocols specially adapted for terminals or networks with limited capabilities; specially adapted for terminal portability
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
    • H04N21/4424—Monitoring of the internal components or processes of the client device, e.g. CPU or memory load, processing speed, timer, counter or percentage of the hard disk space used
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
    • H04N21/462—Content or additional data management, e.g. creating a master electronic program guide from data received from the Internet and a Head-end, controlling the complexity of a video stream by scaling the resolution or bit-rate based on the client capabilities

Definitions

  • Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite radio telephones, video teleconferencing devices, and the like.
  • Digital video devices implement video compression techniques, such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263 or ITU-T H.264/MPEG-4, Part 10, Advanced Video Coding (AVC), and extensions of such standards, to transmit and receive digital video information more efficiently.
  • video compression techniques such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263 or ITU-T H.264/MPEG-4, Part 10, Advanced Video Coding (AVC), and extensions of such standards, to transmit and receive digital video information more efficiently.
  • Efforts have been made to develop new video coding standards based on H.264/AVC.
  • One such standard is the scalable video coding (SVC) standard, which is the scalable extension to H.264/AVC.
  • SVC scalable video coding
  • MVC multi-view video coding
  • a joint draft of MVC is in described in JVT-AB204, "Joint Draft 8.0 on Multiview Video Coding," 28th JVT meeting, Hannover, Germany, July 2008, available at http://wftp3.itu.int/av-arch/jvt-site/2008_07_Hannover/JVT-AB204.zip .
  • JVT-AD007 "Editors' draft revision to ITU-T Rec. H.264
  • Section 8.8 of International Standard 'ISO/IEC 14996-12, 3 rd Edition' discusses movie fragments and track fragments.
  • Sections 3.12 and 14.35 of 'RFC 2616 Hypertext Transfer Protocol - HTTP/1.1" by Fielding et al describe requests specifying byte ranges in HTTP/1.1, including GET methods.
  • this disclosure describes techniques for creating sub-track fragments of video files to support streaming of video data.
  • the techniques of this disclosure include arranging the coded video pictures in an order according to a hierarchical level or layer to which the coded pictures belong.
  • Each hierarchical layer inside a video fragment may correspond to a respective sub-track fragment. That is, each sub-track fragment may include all coded video pictures of the corresponding hierarchical layer for a particular movie fragment in a continuous byte range of the movie fragment.
  • the video pictures in the sub-track fragment may still follow the decoding order.
  • a destination device may submit a single request to retrieve all pictures of the sub-track fragment of a movie fragment.
  • encapsulated coded video pictures may also be referred to as video samples.
  • a method includes arranging, in a track fragment of a video file, coded video pictures according to a hierarchical layer to which the coded video pictures belong, such that the resulting track fragment comprises a plurality of sub-track fragments, each of the sub-track fragments comprising a plurality of hierarchically related video pictures of the encoded video data arranged continuously within the respective sub-track fragment, wherein the plurality of hierarchically related video pictures each correspond to a common hierarchical layer, receiving a hypertext transfer protocol (HTTP) partial GET request in accordance with a streaming protocol, wherein the request specifies a byte range of the video file that corresponds to at least one of the plurality of sub-track fragments, and, in response to the HTTP partial GET request, outputting the plurality of hierarchically related video pictures of the at least one sub-track fragment to which the byte range of the video file corresponds.
  • HTTP hypertext transfer protocol
  • the sub-track fragments correspond to temporal scalability levels, scalable video coding layers or multiview video coding views.
  • the information that describes the hierarchical levels comprises one of: (i) a de-multiplexing header that specifies, for each of the hierarchical levels, a byte range in the movie fragment of the video data corresponding to the sub-track fragments; or (ii) a movie box of a video file including the video data, wherein the movie box defines an operation point corresponding to the subset of the sub-track fragments, and wherein determining the subset comprises selecting the operation point.
  • a track fragment can be a fragment of a video representation of the ISO base media file format, or a fragment of an MPEG-2 Systems stream, which can be any type of the following: packetized elementary stream (PES), program stream (PS), or transport stream (TS).
  • PES packetized elementary stream
  • PS program stream
  • TS transport stream
  • packets corresponding to access units are conventionally ordered in decoding order.
  • An access unit may be segmented into multiple transport packets in a TS stream.
  • the track fragment may be presented as a file unit, e.g., a file or a file segment.
  • the techniques of this disclosure may include reordering the access unit in a fragment into several sub-track fragments, each of which may correspond to a respective hierarchical layer of the access units (coded pictures) such that and coded pictures of a common hierarchical layer are presented continuously in a portion of the stream.
  • the sub-track fragments in a track fragment may be arranged according to decoding order.
  • coded video pictures of a common layer may be arranged contiguously within a video file.
  • a destination device may retrieve all coded pictures up to a particular hierarchical layer within a movie fragment using a single request, e.g., an HTTP partial GET request specifying a byte range of the coded video pictures up to the desired hierarchical layer.
  • AVC file format specifies that coded video pictures are arranged in a decoding order, in any track fragment or movie fragment.
  • a group of pictures may have a number of pictures encoded using various prediction schemes, e.g., intra-prediction (I-pictures) and inter-prediction (P-pictures and B-pictures).
  • I-pictures may be encoded without reference to other pictures
  • P-pictures may be encoded relative to one or more reference pictures in a single direction
  • B-pictures may be encoded relative to one or more pictures in both directions (forward and backward in a video sequence).
  • An inter-coded picture may have a hierarchical level equal to or greater than the hierarchical level of the reference picture for the inter-coded picture.
  • An example sequence of pictures in display order may be I 0 B 3 B 2 B 3 B 1 B 3 B 2 B 3 P 0 , where the letter indicates the encoding type for each picture and the number, in this case, indicates the hierarchical level to which the picture corresponds. Assume for purposes of illustration that each picture is associated with a numerical index corresponding to the picture's position in display order. As indicated above, the example sequence is set out in display order.
  • a reference picture for the encoded picture may first be decoded.
  • the techniques of this disclosure provide an ordering in terms of hierarchical layer of a sequence of coded pictures.
  • a source device may arrange the example sequence above as shown in Table 2: TABLE 2 Pictures in Display Order I 0 B 1 B 2 B 3 P 4 B 5 B 6 B 7 P 8 Temporal hierarchical level 0 2 1 2 0 2 1 2 0 Decoding Order 0 4 3 5 2 7 6 8 1 Order in File 0 5 3 6 1 7 4 8 2
  • a destination device may be configured to retrieve pictures up to hierarchical level one, which may correspond to two sub-track fragments: 0 and 1.
  • the destination device could issue a request based on the byte ranges of sub-track fragments 0 and 1.
  • a source device may provide the pictures in sub-track fragment 0 and 1, having display order 0, 8, 4, 2, 6, and so on.
  • the destination device may reorder the received video pictures, up to a hierarchical level, to form a correct decoding order, before sending the video pictures to a video decoder.
  • information describing the hierarchy of each sub-track fragment may be signaled, e.g., the temporal scalability, the frame rate, and the play rate, when used as fast forward.
  • a destination device may be configured to retrieve pictures only up to a particular hierarchical level for a variety of reasons. For example, a destination device may support a maximum frame rate that is lower than the maximum available frame rate of a video file. As another example, a destination device may support "trick modes," such as fast forward playback at rates two times, four times, eight times, or other multiples of the normal playback rate. In this manner, the techniques of this disclosure may support temporal scalability.
  • the ISO Base Media File Formal is designed to contain timed media information for a presentation in a flexible, extensible format that facilitates interchange, management, editing, and presentation of the media.
  • ISO Base Media File format (ISO/IEC 14496-12:2004) is specified in MPEG-4 Part-12, which defines a general structure for time-based media files. It is used as the basis for other file formats in the family such as AVC file format (ISO/IEC 14496-15) defined support for H.264/MPEG-4 AVC video compression, 3GPP file format, SVC file format, and MVC file format. 3GPP file format and MVC file format are extensions of the AVC file format.
  • ISO base media file format contains the timing, structure, and media information for timed sequences of media data, such as audio-visual presentations.
  • the file structure may be object-oriented. A file can be decomposed into basic objects very simply and the structure of the objects is implied from their type.
  • a presentation may be contained in several files. Timing and framing (position and size) information is generally in the ISO base media file and the ancillary files may essentially use any format. This presentation may be 'local' to the system containing the presentation, or may be provided via a network or other stream delivery mechanism.
  • the files may have a logical structure, a time structure, and a physical structure, and these structures are not required to be coupled.
  • the logical structure of the file may be of a movie that in turn contains a set of time-parallel tracks.
  • the time structure of the file may be that the tracks contain sequences of pictures in time, and those sequences are mapped into the timeline of the overall movie by optional edit lists.
  • the physical structure of the file may separate the data needed for logical, time, and structural decomposition, from the media data samples themselves. This structural information may be concentrated in a movie box, possibly extended in time by movie fragment boxes.
  • the movie box may document the logical and timing relationships of the samples, and may also contain pointers to where they are located. Those pointers may be into the same file or another one, e.g., referenced by a URL.
  • timed meta-data may be stored in an appropriate track, synchronized as desired with the media data it is describing.
  • the structural support is general, and allows, as in the media-data, the storage of meta-data resources elsewhere in the file or in another file. In addition, these resources may be named, and may be protected.
  • a sample grouping is an assignment of each of the samples in a track to be a member of one sample group. Samples in a sample group are not required to be contiguous. For example, when presenting H.264/AVC in AVC file format, video samples in one temporal level can be sampled into one sample group. Sample groups may be represented by two data structures: a SampleToGroup box (sbdp) and a SampleGroupDescription box. The SampleToGroup box represents the assignment of samples to sample groups. There may be one instance of the SampleGroupDescription box for each sample group entry, to describe the properties of the corresponding group.
  • Hint tracks contain general instructions for streaming servers as to how to form packet streams from media tracks for a specific protocol. Because the form of these instructions is media-independent, servers may not need to be revised when new codecs are introduced. In addition, encoding and editing software can be unaware of streaming servers. Once editing is finished on a file, a piece of software called a hinter may be used to add hint tracks to the file, before placing it on a streaming server. As an example, there is a defined hint track format for RTP streams in the MP4 file format specification.
  • 3GP (3GPP file format) is a multimedia container format defined by the Third Generation Partnership Project (3GPP) for 3G UMTS multimedia services. It is typically used on 3G mobile phones and other 3G capable devices, but can also be played on some 2G and 4G phones and devices.
  • 3GPP file format is based on ISO base media file format. The latest 3GP is specified in 3GPP TS26.244, "Transparent end-to-end packet switched streaming service (PSS); 3GPP file format (3GP)."
  • PSS Transparent end-to-end packet switched streaming service
  • 3GPP file format stores video streams as MPEG-4 Part 2 or H.263 or MPEG-4 Part 10 (AVC/H.264).
  • the ISO base media file format specification defines an alternate group of tracks.
  • An alternate group includes a subset of the total available tracks, and each track may correspond to one alternate group.
  • a destination device may select one track from each alternate group, to the exclusion of other tracks in the alternate groups.
  • the 3GPP file format specification defines a switch group of tracks, which is similar to an alternate group. During download streaming and playback, the destination device may switch between different tracks of a switch group. That is, tracks in the same switch group are available for switching during a session, whereas tracks in different switch groups are typically not available for switching.
  • MVC file format also supports an extractor track, which extracts the NAL units from different views to form an operation point, which is a subset of views in a certain frame rate.
  • the design of the MVC extractor track is similar to the extractor in SVC file format.
  • using the MVC extractor tracks to form an alternate group is not supported.
  • MPEG proposal is proposed to MPEG: P. Frojdh, A. Norkin, and C. Priddle, "File format sub-track selection and switching," ISO / IEC JTC1 / SC29 / WG11 MPEG M16665, London UK. This proposal tries to enable the alternate/switch group concept in a sub-track level.
  • the term "progressive download” is used to describe the transfer of digital media files from a server to a client, typically using the HTTP protocol.
  • the computer When initiated from a computer, the computer may begin playback of the media before the download is complete.
  • One difference between streaming media and progressive download is in how the digital media data is received and stored by the end user device that is accessing the digital media.
  • a media player that is capable of progressive download playback relies on metadata located in the header of the file to be intact and a local buffer of the digital media file as it is downloaded from a web server.
  • the device may begin to play the media. This specified amount of buffer may be embedded into the file by the producer of the content in the encoder settings and may be reinforced by additional buffer settings imposed by the media player of the client computer.
  • AVC and 3GPP are extensions of the ISO base media file format
  • SVC and MVC are extensions of the AVC file format. Accordingly, the techniques of this disclosure may be applied with respect to video files conforming to the ISO base media file format, the AVC file format and extensions thereof, e.g., SVC and MVC, and/or the 3GPP file format. The techniques may further be applied to these and other extensions of these formats, and may further be applied to extend other file formats to provide sub track fragments with assembling of video samples in various file formats for HTTP streaming.
  • HTTP/TCP/IP transport is supported for 3GPP files for download and progressive download.
  • HTTP streaming may provide certain advantages, including that existing Internet components and protocols may be used, such that new efforts are not needed to develop new techniques for transporting video data over a network.
  • Other transport protocols e.g., RTP payload format
  • HTTP streaming can be client-driven, which may avoid control issues.
  • the server may keep track of the size and content of packets which are not yet acknowledged.
  • the server may also analyze the file structure and reconstruct the state of the client buffer to make RD-optimal switching/thinning decisions.
  • constraints on the bit stream variations may be satisfied in order to stay compliant with negotiated profiles.
  • HTTP does not necessarily require new hardware or software implementations at a Web server that has HTTP 1.1 implemented.
  • HTTP streaming also provides TCP-friendliness and firewall traversal.
  • HTTP streaming frequently used operations include GET and partial GET.
  • the GET operation retrieves a whole file associated a given uniform resource locator (URL) or uniform resource name (URN).
  • the partial GET operation receives a byte range as an input parameter and retrieves a continuous number of bytes of a file corresponding to the received byte range.
  • movie fragments may be provided for HTTP streaming, because a partial GET operation can get one or more individual movie fragments. Note that, in a movie fragment, there can be several track fragments of different tracks.
  • a media presentation may be a structured collection of data that is accessible to the client. The client may request and download media data information to present a streaming service to a user.
  • FIG. 1 is a block diagram illustrating an example system 10 in which audio/video (A/V) source device 20 sends audio and video data to A/V destination device 40.
  • System 10 of FIG. 1 may correspond to a video teleconference system, a server/client system, a broadcaster/receiver system, or any other system in which video data is sent from a source device, such as A/V source device 20, to a destination device, such as A/V destination device 40.
  • A/V source device 20 and A/V destination device 40 may perform bidirectional information exchange. That is, A/V source device 20 and A/V destination device 40 may be capable of both encoding and decoding (and transmitting and receiving) audio and video data.
  • audio encoder 26 may comprise a voice encoder, also referred to as a vocoder.
  • A/V source device 20, in the example of FIG. 1 comprises audio source 22 and video source 24.
  • Audio source 22 may comprise, for example, a microphone that produces electrical signals representative of captured audio data to be encoded by audio encoder 26.
  • audio source 22 may comprise a storage medium storing previously recorded audio data, an audio data generator such as a computerized synthesizer, or any other source of audio data.
  • Video source 24 may comprise a video camera that produces video data to be encoded by video encoder 28, a storage medium encoded with previously recorded video data, a video data generation unit, or any other source of video data.
  • Raw audio and video data may comprise analog or digital data. Analog data may be digitized before being encoded by audio encoder 26 and/or video encoder 28. Audio source 22 may obtain audio data from a speaking participant while the speaking participant is speaking, and video source 24 may simultaneously obtain video data of the speaking participant. In other examples, audio source 22 may comprise a computer-readable storage medium comprising stored audio data, and video source 24 may comprise a computer-readable storage medium comprising stored video data. In this manner, the techniques described in this disclosure may be applied to live, streaming, real-time audio and video data or to archived, pre-recorded audio and video data.
  • Audio frames that correspond to video frames are generally audio frames containing audio data that was captured by audio source 22 contemporaneously with video data captured by video source 24 that is contained within the video frames.
  • audio source 22 captures the audio data
  • video source 24 captures video data of the speaking participant at the same time, that is, while audio source 22 is capturing the audio data.
  • an audio frame may temporally correspond to one or more particular video frames.
  • an audio frame corresponding to a video frame generally corresponds to a situation in which audio data and video data were captured at the same time and for which an audio frame and a video frame comprise, respectively, the audio data and the video data that was captured at the same time.
  • audio encoder 26 may encode a timestamp in each encoded audio frame that represents a time at which the audio data for the encoded audio frame was recorded
  • video encoder 28 may encode a timestamp in each encoded video frame that represents a time at which the video data for encoded video frame was recorded
  • an audio frame corresponding to a video frame may comprise an audio frame comprising a timestamp and a video frame comprising the same timestamp
  • A/V source device 20 may include an internal clock from which audio encoder 26 and/or video encoder 28 may generate the timestamps, or that audio source 22 and video source 24 may use to associate audio and video data, respectively, with a timestamp.
  • audio source 22 may send data to audio encoder 26 corresponding to a time at which audio data was recorded
  • video source 24 may send data to video encoder 28 corresponding to a time at which video data was recorded
  • audio encoder 26 may encode a sequence identifier in encoded audio data to indicate a relative temporal ordering of encoded audio data but without necessarily indicating an absolute time at which the audio data was recorded
  • video encoder 28 may also use sequence identifiers to indicate a relative temporal ordering of encoded video data.
  • a sequence identifier may be mapped or otherwise correlated with a timestamp.
  • video source 24 may provide a plurality of views of a scene to video encoder 28.
  • A/V source device 20 may provide a "service" to A/V destination device 40.
  • a service generally corresponds to a subset of available views of MVC data. For example, MVC data may be available for eight views, ordered zero through seven.
  • One service may correspond to stereo video having two views, while another service may correspond to four views, and still another service may correspond to all eight views.
  • a service corresponds to any combination (that is, any subset) of the available views.
  • a service may also correspond to a combination of available views as well as audio data.
  • An operation point may correspond to a service, such that A/V source device 20 may further provide an operation point descriptor for each service provided by A/V source device 20.
  • An elementary stream is a single, digitally coded (possibly compressed) component of a program.
  • the coded video or audio part of the program can be an elementary stream.
  • An elementary stream may be converted into a packetized elementary stream (PES) before being encapsulated within a video file.
  • PES packetized elementary stream
  • a stream ID is used to distinguish the PES-packets belonging to one elementary stream from the other.
  • the basic unit of data of an elementary stream is a packetized elementary stream (PES) packet.
  • PES packetized elementary stream
  • An MVC coded video sequence may be separated into several sub-bitstreams, each of which is an elementary stream. Each sub-bitstream may be identified using an MVC view_id subset. Based on the concept of each MVC view_id subset, an MVC video sub-bitstream is defined. An MVC video sub-bitstream contains the NAL units of the views listed in the MVC view_id subset. A program stream generally contains only the NAL units which are from those of the elementary streams. It is also designed that any two elementary streams cannot contain an identical view, but may instead contain views, e.g., different perspectives of a scene for creating a three-dimensional effect.
  • encapsulation unit 30 receives elementary streams comprising video data from video encoder 28 and elementary streams comprising audio data from audio encoder 26.
  • video encoder 28 and audio encoder 26 may each include packetizers for forming PES packets from encoded data.
  • video encoder 28 and audio encoder 26 may each interface with respective packetizers for forming PES packets from encoded data.
  • encapsulation unit 30 may include packetizers for forming PES packets from encoded audio and video data.
  • a "program,” as used in this disclosure, may comprise a combination of audio data and video data, e.g., an audio elementary stream and a subset of available views delivered by a service of A/V source device 20.
  • Each PES packet includes a stream_id that identifies the elementary stream to which the PES packet belongs.
  • Encapsulation unit 30 is responsible for assembling elementary streams into a video file.
  • Encapsulation unit 30 receives PES packets for elementary streams of a program from audio encoder 26 and video encoder 28 and forms corresponding network abstraction layer (NAL) units from the PES packets.
  • NAL network abstraction layer
  • coded video segments are organized into NAL units, which provide a "network-friendly" video representation addressing applications such as video telephony, storage, broadcast, or streaming.
  • NAL units can be categorized to Video Coding Layer (VCL) NAL units and non-VCL NAL units.
  • VCL units may contain the core compression engine and may include block, macroblock, and/or slice level data.
  • Other NAL units may be non-VCL NAL units.
  • a coded picture in one time instance normally presented as a primary coded picture, may be contained in an access unit, which may include one or more NAL units.
  • encapsulation unit 30 may signal reordering information in a video file that indicates how to reorder the coded video pictures of more than one sub-track fragment into decoding order.
  • encapsulation unit 30 may include reassembler objects in a sub-track fragment.
  • a reassembler object may act as a pointer to a coded video sample in a previous e.g., within the same or a lower level sub-track fragment.
  • Destination device 40 may use reassembler objects to re-arrange samples after sub-track fragments have been received.
  • encapsulation unit 30 may further include de-multiplexing headers (which may alternatively be referred to as "reassembling headers") that describe characteristics of one or more sub-track fragments.
  • Encapsulation unit 30 may include the de-multiplexing headers in various locations, such as, for example, a movie box, a movie fragment header, and/or a track fragment header.
  • the de-multiplexing headers may specify unique identifiers for each sub-track fragment, byte ranges for corresponding sub-track fragments, a number of pictures in each sub-track fragment, and timing information of the sub-track fragments.
  • the timing information may be described as relative timing information in terms of samples or coordinated universal times (UTC).
  • Encapsulation unit 30 need not include such timing information when sub-track fragments do not correspond to layers with different frame rates or temporal levels.
  • Encapsulation unit 30 may form NAL units comprising a header that identifies a program to which the NAL belongs, as well as a payload, e.g., audio data, video data, or data that describes the transport or program stream to which the NAL unit corresponds.
  • a NAL unit includes a 1-byte header and a payload of varying size.
  • a NAL unit header comprises a priority_id element, a temporal_id element, an anchor_pic_flag element, a view_id element, a non_idr_flag element, and an inter_view_flag element.
  • the NAL unit defined by H.264 is retained, except for prefix NAL units and MVC coded slice NAL units, which include a 4-byte MVC NAL unit header and the NAL unit payload.
  • the priority_id element of an NAL header may be used for a simple one-path bitstream adaptation process.
  • the temporal_id element may be used for specifying the temporal level of the corresponding NAL unit, where different temporal levels correspond to different frame rates.
  • the anchor_pic_flag element may indicate whether a picture is an anchor picture or non-anchor picture.
  • Anchor pictures and all the pictures succeeding it in the output order can be correctly decoded without decoding of previous pictures in the decoding order (that is, the bitstream order), and thus, can be used as random access points.
  • Anchor pictures and non-anchor pictures can have different dependencies, both of which are signaled in the sequence parameter set. Other flags are to be discussed and used in the following sections of this chapter.
  • Such an anchor picture may also be referred to as an open GOP (Group Of Pictures) access point, while a close GOP access point is also supported when the non_idr_flag element is equal to zero.
  • the non_idr_flag element indicates whether a picture is an instantaneous decoder refresh (IDR) or view IDR (V-IDR) picture.
  • IDR instantaneous decoder refresh
  • V-IDR view IDR
  • an IDR picture, and all the pictures succeeding it in output order or bitstream order, can be correctly decoded without decoding of previous pictures in either decoding order or display order.
  • the view_id element may comprise syntax information that may be used to identify a view, which may be used for data interactivity inside an MVC decoder, e.g., for inter-view prediction, and outside a decoder, e.g., for rendering.
  • the inter_view_flag element may specify whether the corresponding NAL unit is used by other views for inter-view prediction.
  • a prefix NAL unit is defined in MVC.
  • the base view access unit includes the VCL NAL units of the current time instance of the view as well as its prefix NAL unit, which contains only the NAL unit head.
  • An H.264/AVC decoder may ignore the prefix NAL unit.
  • a decoder conforming to a profile ordinarily supports all the features defined in the profile. For example, as a coding feature, B-picture coding is not supported in the baseline profile of H.264/AVC but is supported in other profiles of H.264/AVC.
  • a decoder conforming to a level should be capable of decoding any bitstream that does not require resources beyond the limitations defined in the level. Definitions of profiles and levels may be helpful for interpretability. For example, during video transmission, a pair of profile and level definitions may be negotiated and agreed for a whole transmission session.
  • packets containing video data may include a decoding time stamp.
  • the decoding time of each access unit may be determined, and such a reordering process would not require additional signaling.
  • the interleaving of a hierarchical layer with index i and hierarchical layer with index i+1 may follow a fixed pattern and thus very lightweight signaling, e.g., the number of video samples in hierarchical layer i and the other number of video samples following the video samples in hierarchical layer i+1, in a period can be signaled.
  • the prediction hierarchy is related to view order indexes, in that frames relatively higher in the prediction hierarchy should be decoded before decoding frames that are relatively lower in the hierarchy, such that those frames relatively higher in the hierarchy can be used as reference frames during decoding of the frames relatively lower in the hierarchy.
  • a view order index is an index that indicates the decoding order of view components in an access unit.
  • the view order indices is implied in the SPS MVC extension, as specified in Annex H of H.264/AVC (MVC amendment).
  • MVC MVC amendment
  • the corresponding view_id is signaled.
  • the decoding of the view components shall follow the ascending order of the view order index. If all the views are presented, then the view order indexes are in a consecutive order from 0 to num_views_minus_ 1.
  • a view order index is an index that indicates the decoding order of view components in an access unit. For each view order index i, the corresponding view_id is signaled. The decoding of the view components follows the ascending order of the view order indexes. If all the views are presented, then the set of view order indexes comprises a consecutively ordered set from zero to one less than the full number of views.
  • views S1 and S3 do not rely on each other for prediction, but instead are predicted only from views that are higher in the prediction hierarchy.
  • view S1 may be decoded before view S4, so long as view S1 is decoded after views S0 and S2.
  • frames at temporal location T0 are either intra-predicted or inter-view predicted from frames of other views at temporal location T0.
  • frames at temporal location T8 are either intra-predicted or inter-view predicted from frames of other views at temporal location T8. Accordingly, with respect to a temporal hierarchy, temporal locations T0 and T8 are at the top of the temporal hierarchy.
  • Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol.
  • Computer-readable media generally may correspond to (1) tangible computer-readable storage media which is non-transient or (2) a communication medium such as a signal or carrier wave.
  • Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementation of the techniques described in this disclosure.
  • a computer program product may include a computer-readable medium.
  • processors such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry.
  • DSPs digital signal processors
  • ASICs application specific integrated circuits
  • FPGAs field programmable logic arrays
  • processors may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein.
  • the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated in a combined codec. Also, the techniques could be fully implemented in one or more circuits or logic elements.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Databases & Information Systems (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
EP11736528.8A 2010-07-20 2011-07-20 Arranging sub-track fragments for streaming video data Active EP2596643B1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US12/840,146 US8930562B2 (en) 2010-07-20 2010-07-20 Arranging sub-track fragments for streaming video data
PCT/US2011/044745 WO2012012575A1 (en) 2010-07-20 2011-07-20 Arranging sub-track fragments for streaming video data

Publications (2)

Publication Number Publication Date
EP2596643A1 EP2596643A1 (en) 2013-05-29
EP2596643B1 true EP2596643B1 (en) 2016-03-30

Family

ID=44629108

Family Applications (1)

Application Number Title Priority Date Filing Date
EP11736528.8A Active EP2596643B1 (en) 2010-07-20 2011-07-20 Arranging sub-track fragments for streaming video data

Country Status (19)

Country Link
US (1) US8930562B2 (uk)
EP (1) EP2596643B1 (uk)
JP (1) JP5551315B2 (uk)
KR (1) KR101437798B1 (uk)
CN (1) CN103026721B (uk)
AR (1) AR082303A1 (uk)
AU (1) AU2011282166B2 (uk)
BR (1) BR112013001337B1 (uk)
CA (1) CA2805274C (uk)
ES (1) ES2579630T3 (uk)
HK (1) HK1180862A1 (uk)
HU (1) HUE029013T2 (uk)
IL (1) IL224196A (uk)
MY (1) MY156121A (uk)
RU (1) RU2541155C2 (uk)
SG (1) SG187014A1 (uk)
TW (1) TWI489843B (uk)
UA (1) UA108893C2 (uk)
WO (1) WO2012012575A1 (uk)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107135215A (zh) * 2017-04-28 2017-09-05 山东易构软件技术股份有限公司 一种基于svc的p2p流媒体不等时长分片及调度方法

Families Citing this family (79)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9826197B2 (en) 2007-01-12 2017-11-21 Activevideo Networks, Inc. Providing television broadcasts over a managed network and interactive content over an unmanaged network to a client device
ES2627521T3 (es) 2010-01-18 2017-07-28 Telefonaktiebolaget Lm Ericsson (Publ) Método y disposición para soportar reproducción de contenidos
KR101777348B1 (ko) * 2010-02-23 2017-09-11 삼성전자주식회사 데이터 전송 방법 및 장치와 데이터 수신 방법 및 장치
KR20120016991A (ko) * 2010-08-17 2012-02-27 오수미 인터 프리딕션 방법
KR101620151B1 (ko) * 2010-10-05 2016-05-12 텔레폰악티에볼라겟엘엠에릭슨(펍) 클라이언트와, 콘텐트 생성기 엔티티 및 미디어 스트리밍을 위한 이들의 방법
US9706227B2 (en) 2011-03-10 2017-07-11 Qualcomm Incorporated Video coding techniques for coding dependent pictures after random access
WO2012125006A2 (ko) * 2011-03-16 2012-09-20 한국전자통신연구원 레프리젠테이션을 사용하는 스트리밍 콘텐츠 제공 장치 및 방법
BR112014011425B1 (pt) 2011-11-11 2022-08-23 GE Video Compression, LLC. Codificação de multi-visualização eficiente utilizando estimativa e atualização de mapa por profundidade
EP2777273B1 (en) 2011-11-11 2019-09-04 GE Video Compression, LLC Efficient multi-view coding using depth-map estimate for a dependent view
EP3739886A1 (en) 2011-11-18 2020-11-18 GE Video Compression, LLC Multi-view coding with efficient residual handling
US10409445B2 (en) * 2012-01-09 2019-09-10 Activevideo Networks, Inc. Rendering of an interactive lean-backward user interface on a television
US9800945B2 (en) 2012-04-03 2017-10-24 Activevideo Networks, Inc. Class-based intelligent multiplexing over unmanaged networks
DK3174295T3 (en) 2012-04-13 2019-04-01 Ge Video Compression Llc Low delay image encoding
KR101995270B1 (ko) * 2012-04-25 2019-07-03 삼성전자주식회사 비디오 데이터를 재생하는 방법 및 장치
WO2013163555A1 (en) 2012-04-26 2013-10-31 Huawei Technologies Co., Ltd. System and method for signaling segment encryption and key derivation for adaptive streaming
WO2013172667A1 (en) * 2012-05-17 2013-11-21 Samsung Electronics Co., Ltd. Recording medium, reproducing device for performing trick play for data of the recording medium, and method thereof
US20130318251A1 (en) * 2012-05-22 2013-11-28 Alimuddin Mohammad Adaptive multipath content streaming
US9674532B2 (en) * 2012-06-24 2017-06-06 Lg Electronics Inc. Image decoding method using information on a random access picture and apparatus using same
EP2680527A1 (en) * 2012-06-28 2014-01-01 Alcatel-Lucent Adaptive streaming aware node, encoder and client enabling smooth quality transition
CN115442624A (zh) 2012-06-29 2022-12-06 Ge视频压缩有限责任公司 视频数据流、编码器、编码视频内容的方法以及解码器
KR101835360B1 (ko) 2012-10-01 2018-03-08 지이 비디오 컴프레션, 엘엘씨 향상 레이어에서 변환 계수 블록들의 서브블록-기반 코딩을 이용한 스케일러블 비디오 코딩
US8892693B2 (en) * 2012-10-02 2014-11-18 Nextbit Systems Inc. Enabling fragment-based mobile device application streaming
US8977723B2 (en) * 2012-10-02 2015-03-10 Nextbit Systems Inc. Cloud based application fragmentation
US9112885B2 (en) 2012-10-02 2015-08-18 Nextbit Systems Inc. Interactive multi-tasker
US9776078B2 (en) 2012-10-02 2017-10-03 Razer (Asia-Pacific) Pte. Ltd. Application state backup and restoration across multiple devices
US9717985B2 (en) 2012-10-02 2017-08-01 Razer (Asia-Pacific) Pte. Ltd. Fragment-based mobile device application streaming utilizing crowd-sourcing
US10425471B2 (en) 2012-10-02 2019-09-24 Razer (Asia-Pacific) Pte. Ltd. Multi-tasker
US9654556B2 (en) 2012-10-02 2017-05-16 Razer (Asia-Pacific) Pte. Ltd. Managing applications on an electronic device
US9106721B2 (en) 2012-10-02 2015-08-11 Nextbit Systems Application state synchronization across multiple devices
US9747000B2 (en) 2012-10-02 2017-08-29 Razer (Asia-Pacific) Pte. Ltd. Launching applications on an electronic device
US9210203B2 (en) 2012-10-02 2015-12-08 Nextbit Systems Inc. Resource based mobile device application streaming
US9268655B2 (en) 2012-10-02 2016-02-23 Nextbit Systems Inc. Interface for resolving synchronization conflicts of application states
US8764555B2 (en) 2012-10-02 2014-07-01 Nextbit Systems Inc. Video game application state synchronization across multiple devices
US9600552B2 (en) 2012-10-02 2017-03-21 Nextbit Systems Inc. Proximity based application state synchronization
US11290510B2 (en) * 2012-11-29 2022-03-29 Samsung Electronics Co., Ltd. Method and apparatus for encapsulation of motion picture experts group media transport assets in international organization for standardization base media files
US9648299B2 (en) 2013-01-04 2017-05-09 Qualcomm Incorporated Indication of presence of texture and depth views in tracks for multiview coding plus depth
KR101965374B1 (ko) 2013-01-18 2019-04-03 캐논 가부시끼가이샤 비디오 데이터의 재생 방법 및 비디오 데이터를 재생하기 위한 디바이스
US10275128B2 (en) 2013-03-15 2019-04-30 Activevideo Networks, Inc. Multiple-mode system and method for providing user selectable video content
US10123189B2 (en) 2013-03-21 2018-11-06 Razer (Asia-Pacific) Pte. Ltd. Electronic device system restoration by tapping mechanism
US8954611B2 (en) 2013-03-21 2015-02-10 Nextbit Systems Inc. Mechanism for sharing states of applications and devices across different user profiles
WO2014197879A1 (en) 2013-06-06 2014-12-11 Activevideo Networks, Inc. Overlay rendering of user interface onto source video
EP3014896B1 (en) * 2013-06-24 2020-06-03 Samsung Electronics Co., Ltd. Method and apparatus for converting content in multimedia system
USD768162S1 (en) 2013-09-30 2016-10-04 Nextbit Systems Inc. Display screen or portion thereof with graphical user interface
US9330171B1 (en) * 2013-10-17 2016-05-03 Google Inc. Video annotation using deep network architectures
GB2542282B (en) 2013-10-22 2018-08-01 Canon Kk Method, device, and computer program for encapsulating partitioned timed media data in a server
CN104919812B (zh) * 2013-11-25 2018-03-06 华为技术有限公司 处理视频的设备和方法
KR20150065289A (ko) * 2013-12-05 2015-06-15 삼성전자주식회사 데이터 재사용 방법 및 전자장치
JP2015136057A (ja) * 2014-01-17 2015-07-27 ソニー株式会社 通信装置、通信データ生成方法、および通信データ処理方法
US9788029B2 (en) 2014-04-25 2017-10-10 Activevideo Networks, Inc. Intelligent multiplexing using class-based, multi-dimensioned decision logic for managed networks
JP6706784B2 (ja) * 2014-09-12 2020-06-10 パナソニックＩｐマネジメント株式会社 送信装置、受信装置、送信方法及び受信方法
FR3029381A1 (fr) * 2014-11-27 2016-06-03 Orange Procede de composition d’une representation video intermediaire
WO2016108188A1 (en) 2014-12-31 2016-07-07 Nokia Technologies Oy Inter-layer prediction for scalable video coding and decoding
US10148969B2 (en) * 2015-02-11 2018-12-04 Qualcomm Incorporated Of sample entry and operation point signalling in a layered video file format
US20160373771A1 (en) * 2015-06-18 2016-12-22 Qualcomm Incorporated Design of tracks and operation point signaling in layered hevc file format
US10034010B2 (en) * 2015-10-14 2018-07-24 Qualcomm Incorporated Alignment of operation point sample group in multi-layer bitstreams file format
GB2551323B (en) * 2016-06-08 2021-02-10 Companion Ltd System providing a shared environment
US20180013811A1 (en) * 2016-07-07 2018-01-11 Novatek Microelectronics Corp. Display sysyem, a source device, a sink device and an image displaying method
US11197040B2 (en) * 2016-10-17 2021-12-07 Mediatek Inc. Deriving and signaling a region or viewport in streaming media
US10713174B2 (en) * 2016-12-20 2020-07-14 Texas Instruments Incorporated Streaming engine with early and late address and loop count registers to track architectural state
US10819763B2 (en) 2017-03-31 2020-10-27 At&T Intellectual Property I, L.P. Apparatus and method of video streaming
US10484308B2 (en) 2017-03-31 2019-11-19 At&T Intellectual Property I, L.P. Apparatus and method of managing resources for video services
US10939086B2 (en) * 2018-01-17 2021-03-02 Mediatek Singapore Pte. Ltd. Methods and apparatus for encoding and decoding virtual reality content
CN111837396B (zh) * 2018-04-03 2023-05-09 华为技术有限公司 基于子图像码流视角相关视频编码中的误差抑制
US10944977B2 (en) 2018-04-03 2021-03-09 Mediatek Singapore Pte. Ltd. Methods and apparatus for encoding and decoding overlay compositions
US10869016B2 (en) 2018-04-12 2020-12-15 Mediatek Singapore Pte. Ltd. Methods and apparatus for encoding and decoding virtual reality content
GB2585760B (en) * 2018-06-06 2022-04-20 Canon Kk Method, device, and computer program for transmitting media content
GB2583885A (en) * 2018-09-20 2020-11-18 Canon Kk Method, device, and computer program for improving transmission of encoded media data
US11044185B2 (en) 2018-12-14 2021-06-22 At&T Intellectual Property I, L.P. Latency prediction and guidance in wireless communication systems
EP3702933A1 (en) * 2019-02-28 2020-09-02 Rohde & Schwarz GmbH & Co. KG System for analyzing and interpreting at least one data stream as well as method of tagging data fragments
US11457231B2 (en) 2019-03-15 2022-09-27 Mediatek Singapore Pte. Ltd. Methods and apparatus for signaling spatial relationships for point cloud multimedia data tracks
US11245926B2 (en) * 2019-03-19 2022-02-08 Mediatek Singapore Pte. Ltd. Methods and apparatus for track derivation for immersive media data tracks
US11366845B2 (en) * 2019-05-20 2022-06-21 Accenture Global Solutions Limited Facilitating merging of concept hierarchies
KR20210110097A (ko) * 2020-02-28 2021-09-07 삼성전자주식회사 영상 스트리밍 방법 및 이를 지원하는 전자 장치
CN113766272B (zh) * 2020-06-04 2023-02-10 腾讯科技（深圳）有限公司 一种沉浸媒体的数据处理方法
US11588876B2 (en) * 2020-06-16 2023-02-21 T-Mobile Usa, Inc. Device-side playback restrictions on high throughput networks
CN116210225A (zh) * 2020-09-29 2023-06-02 Lg 电子株式会社 生成媒体文件的方法及设备
US11553017B1 (en) 2021-03-09 2023-01-10 Nokia Technologies Oy Timed media HTTP request aggregation
CN113805499B (zh) * 2021-09-14 2023-04-25 北京邮电大学 一种航天器遥控指令计划生成方法
EP4376419A1 (en) * 2022-11-28 2024-05-29 Ateme Method for media stream processing and apparatus for implementing the same

Family Cites Families (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6909708B1 (en) * 1996-11-18 2005-06-21 Mci Communications Corporation System, method and article of manufacture for a communication system architecture including video conferencing
FI114527B (fi) * 2002-01-23 2004-10-29 Nokia Corp Kuvakehysten ryhmittely videokoodauksessa
KR100834750B1 (ko) 2004-01-29 2008-06-05 삼성전자주식회사 엔코더 단에서 스케일러빌리티를 제공하는 스케일러블비디오 코딩 장치 및 방법
US7664109B2 (en) 2004-09-03 2010-02-16 Microsoft Corporation System and method for distributed streaming of scalable media
DE102004059978B4 (de) 2004-10-15 2006-09-07 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Vorrichtung und Verfahren zum Erzeugen einer codierten Videosequenz und zum Decodieren einer codierten Videosequenz unter Verwendung einer Zwischen-Schicht-Restwerte-Prädiktion sowie ein Computerprogramm und ein computerlesbares Medium
DE102005001286A1 (de) * 2005-01-11 2006-07-20 Siemens Ag Verfahren und Vorrichtung zur Übertragung von skalierbaren Daten
DE102005001287A1 (de) * 2005-01-11 2006-07-20 Siemens Ag Verfahren und Vorrichtung zur Verarbeitung von skalierbaren Daten
US8228994B2 (en) 2005-05-20 2012-07-24 Microsoft Corporation Multi-view video coding based on temporal and view decomposition
US8879857B2 (en) 2005-09-27 2014-11-04 Qualcomm Incorporated Redundant data encoding methods and device
FR2894421B1 (fr) 2005-12-07 2008-01-18 Canon Kk Procede et dispositif de decodage d'un flux video code suivant un codage hierarchique
KR20070108434A (ko) * 2006-01-09 2007-11-12 한국전자통신연구원 ＳＶＣ(Ｓｃａｌａｂｌｅ Ｖｉｄｅｏ Ｃｏｄｉｎｇ)파일포맷에서의 데이터 공유 개선방법
KR20070108433A (ko) * 2006-01-09 2007-11-12 한국전자통신연구원 청크 디스크립터를 이용한 ｓｖｃ 파일포맷에서의 비디오데이터 공유방법
JP4818373B2 (ja) * 2006-01-09 2011-11-16 韓國電子通信研究院 Ｓｖｃファイルのデータ共有方法およびそのファイル
RU2395174C1 (ru) * 2006-03-30 2010-07-20 ЭлДжи ЭЛЕКТРОНИКС ИНК. Способ и устройство для декодирования/кодирования сигнала видео
CN101282348B (zh) * 2007-04-06 2011-03-30 上海晨兴电子科技有限公司 运用http协议实现流媒体功能的方法
FR2917262A1 (fr) 2007-06-05 2008-12-12 Thomson Licensing Sas Dispositif et procede de codage d'un contenu video sous la forme d'un flux scalable.
JP5542912B2 (ja) * 2009-04-09 2014-07-09 テレフオンアクチーボラゲット エル エム エリクソン（パブル） メディア・コンテナ・ファイル管理

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
"Scalable Continuous Media Streaming Systems", 1 January 2005, JOHN WILEY & SONS LTD., CHICHESTER, WEST SUSSEX, ENGLAND, ISBN: 978-0-47-085754-0, article JACK Y. B. LEE: "Chapter 2: Media Compression", pages: 25 - 42, XP055041298, DOI: 10.1002/0470001539X *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107135215A (zh) * 2017-04-28 2017-09-05 山东易构软件技术股份有限公司 一种基于svc的p2p流媒体不等时长分片及调度方法
CN107135215B (zh) * 2017-04-28 2020-08-11 山东易构软件技术股份有限公司 一种基于svc的p2p流媒体不等时长分片及调度方法

Also Published As

Publication number Publication date
UA108893C2 (uk) 2015-06-25
SG187014A1 (en) 2013-02-28
WO2012012575A1 (en) 2012-01-26
CN103026721A (zh) 2013-04-03
IL224196A (en) 2016-10-31
CA2805274A1 (en) 2012-01-26
KR20130056287A (ko) 2013-05-29
TW201230747A (en) 2012-07-16
HUE029013T2 (en) 2017-02-28
BR112013001337B1 (pt) 2022-05-17
HK1180862A1 (zh) 2013-10-25
EP2596643A1 (en) 2013-05-29
JP5551315B2 (ja) 2014-07-16
TWI489843B (zh) 2015-06-21
ES2579630T3 (es) 2016-08-12
CA2805274C (en) 2015-11-17
AU2011282166A1 (en) 2013-02-07
JP2013535900A (ja) 2013-09-12
RU2013107363A (ru) 2014-08-27
AR082303A1 (es) 2012-11-28
US8930562B2 (en) 2015-01-06
US20120023250A1 (en) 2012-01-26
CN103026721B (zh) 2016-05-11
KR101437798B1 (ko) 2014-09-03
BR112013001337A2 (pt) 2016-05-17
RU2541155C2 (ru) 2015-02-10
MY156121A (en) 2016-01-15
AU2011282166B2 (en) 2014-08-14

Similar Documents

Publication Publication Date Title
EP2596643B1 (en) Arranging sub-track fragments for streaming video data
EP2596632B1 (en) Providing sequence data sets for streaming video data
EP2478703B1 (en) Multi-track video coding methods and apparatus using an extractor that references two or more non-consecutive nal units
EP2601789B1 (en) Signaling attributes for network-streamed video data
EP2594073B1 (en) Video switching for streaming video data
EP2589221B1 (en) Signaling random access points for streaming video data
EP2594071A1 (en) Signaling data for multiplexing video components

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20130220

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAX Request for extension of the european patent (deleted)
17Q First examination report despatched

Effective date: 20140602

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602011024658

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: H04N0021234300

Ipc: H04L0029060000

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

RIC1 Information provided on ipc code assigned before grant

Ipc: H04L 29/06 20060101AFI20150827BHEP

Ipc: H04N 21/442 20110101ALI20150827BHEP

Ipc: H04N 21/845 20110101ALI20150827BHEP

Ipc: H04N 21/2343 20110101ALI20150827BHEP

Ipc: H04N 21/462 20110101ALI20150827BHEP

Ipc: H04N 21/236 20110101ALI20150827BHEP

Ipc: H04L 29/08 20060101ALI20150827BHEP

INTG Intention to grant announced

Effective date: 20150910

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

INTG Intention to grant announced

Effective date: 20151221

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 786496

Country of ref document: AT

Kind code of ref document: T

Effective date: 20160415

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602011024658

Country of ref document: DE

REG Reference to a national code

Ref country code: NL

Ref legal event code: FP

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 6

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160701

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160630

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

REG Reference to a national code

Ref country code: ES

Ref legal event code: FG2A

Ref document number: 2579630

Country of ref document: ES

Kind code of ref document: T3

Effective date: 20160812

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 786496

Country of ref document: AT

Kind code of ref document: T

Effective date: 20160330

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160730

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160801

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602011024658

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

REG Reference to a national code

Ref country code: HU

Ref legal event code: AG4A

Ref document number: E029013

Country of ref document: HU

Ref country code: CH

Ref legal event code: PL

26N No opposition filed

Effective date: 20170103

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160731

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160731

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 7

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160720

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160720

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 8

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160731

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160330

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602011024658

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: H04L0029060000

Ipc: H04L0065000000

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20230619

Year of fee payment: 13

Ref country code: FR

Payment date: 20230616

Year of fee payment: 13

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FI

Payment date: 20230627

Year of fee payment: 13

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: IT

Payment date: 20230711

Year of fee payment: 13

Ref country code: GB

Payment date: 20230614

Year of fee payment: 13

Ref country code: ES

Payment date: 20230802

Year of fee payment: 13

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: HU

Payment date: 20230628

Year of fee payment: 13

Ref country code: DE

Payment date: 20230614

Year of fee payment: 13