WO2013188112A2 - Systems and methods for adaptive streaming systems with interactive video timelines - Google Patents

Systems and methods for adaptive streaming systems with interactive video timelines Download PDF

Info

Publication number
WO2013188112A2
WO2013188112A2 PCT/US2013/043176 US2013043176W WO2013188112A2 WO 2013188112 A2 WO2013188112 A2 WO 2013188112A2 US 2013043176 W US2013043176 W US 2013043176W WO 2013188112 A2 WO2013188112 A2 WO 2013188112A2
Authority
WO
WIPO (PCT)
Prior art keywords
video
frames
stream
video data
timeline
Prior art date
Application number
PCT/US2013/043176
Other languages
French (fr)
Other versions
WO2013188112A3 (en
Inventor
Abhishek Shivadas
Kourosh Soroushian
Original Assignee
Divx, Llc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Divx, Llc filed Critical Divx, Llc
Publication of WO2013188112A2 publication Critical patent/WO2013188112A2/en
Publication of WO2013188112A3 publication Critical patent/WO2013188112A3/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/0485—Scrolling or panning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/04842—Selection of displayed objects or displayed text elements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/048—Indexing scheme relating to G06F3/048
    • G06F2203/04803—Split screen, i.e. subdividing the display area or the window area into separate subareas
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
    • H04N21/47217—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/845—Structuring of content, e.g. decomposing content into time segments
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/85—Assembly of content; Generation of multimedia applications
    • H04N21/854—Content authoring
    • H04N21/8545—Content authoring for generating interactive applications

Definitions

  • the present invention is directed, in general, to systems and methods for adaptive streaming systems and more specifically to systems and methods for adaptive streaming systems configured to generate interactive video timelines.
  • streaming media describes the playback of media on a playback device, where the media is stored on a server and continuously sent to the playback device over a network during playback.
  • the playback device stores a sufficient quantity of media in a buffer at any given time during playback to prevent disruption of playback due to the playback device completing playback of all the buffered media prior to receipt of the next portion of media.
  • Adaptive bitrate streaming or adaptive streaming involves detecting the present streaming conditions (e.g. the user's network bandwidth and CPU capacity) in real time and adjusting the quality of the streamed media accordingly.
  • Streaming video over the Internet has become a phenomenon in modern times. Many popular websites, such as YouTube, a service of Google, Inc. of Mountain View, California, and WatchESPN, a service of ESPN of Bristol, Connecticut, utilize streaming video in order to provide video and television programming to consumers via the Internet.
  • Scalable Video Coding is an extension of the H.264/MPEG-4 AVC video compression standard, which is specified by the ITU-T H.264 standard by the International Telecommunication Union Telecommunication Standardization Sector of Geneva, Switzerland.
  • SVC enables the encoding of a video bitstream that additionally contains one or more sub-bitstreams.
  • the sub-bitstreams are derived from the video bitstream by dropping packets of data from the video bitstream, resulting in a sub- bitstream of lower quality and lower bandwidth than the original video bitstream.
  • SVC supports three forms of scaling a video bitstream into sub-bitstreams: temporal scaling, spatial scaling, and quality scaling. Each of these scaling techniques can be used individually or combined depending on the specific video system.
  • the multimedia content is typically stored on a media server as a top level index file pointing to a number of alternate streams that contain the actual video and audio data. Each stream is typically stored in one or more container files.
  • Different adaptive streaming solutions typically utilize different index and media containers.
  • the Matroska container is a media container developed as an open standard project by the Matroska non-profit organization of Aussonne, France.
  • the Matroska container is based upon Extensible Binary Meta Language (EBML), which is a binary derivative of the Extensible Markup Language (XML). Decoding of the Matroska container is supported by many consumer electronics (CE) devices.
  • CE consumer electronics
  • the DivX Plus file format developed by DivX, LLC of San Diego, California utilizes an extension of the Matroska container format, including elements that are not specified within the Matroska format.
  • a method for selecting video to stream and playback using an interactive video timeline includes selecting a plurality of frames of video from a first stream of video data using a network client, displaying the plurality of frames of video in an interactive video timeline using the network client, selecting a playback frame of video data from the plurality of frames of video in the interactive video timeline using the network client, receiving a second stream of video data beginning from the playback frame of video data using the network client, and playing the second stream of video data using the network client.
  • the first stream of video data is a stream of intra frames.
  • the first stream of video data and the second stream of video data are stored in container files.
  • the container files are Matroska container files.
  • the first stream of video data and the second stream of video data are encoded using scalable video coding.
  • the plurality of frames of video are displayed in a grid.
  • playing the second stream of video data using the network client includes closing the interactive video timeline using the network client and playing the second stream of video data in a full screen mode using the network client.
  • playing the second stream of video data using the network client includes playing the second stream of video data in a window on top of the interactive video timeline using the network client.
  • playing the second stream of video data using the network client includes playing the second stream of video data in the interactive video timeline using the network client.
  • displaying the plurality of frames of video in an interactive video timeline further includes receiving a structure file using the network client, wherein the structure file contains the structure defining the layout of the interactive video timeline, receiving the plurality of frames of video data using the network client, creating the interactive video timeline using the structure file, the plurality of frames of video data and the network client, and displaying the interactive video timeline using the network client.
  • selecting video to stream and playback using an interactive video timeline further includes displaying metadata for a plurality of frames of video in the plurality of frames of video in the interactive video timeline using the network client.
  • selecting a plurality of frames of video from a first stream of video data further includes selecting a first set of frames of video from the first stream of video data using the network client and displaying the plurality of frames of video in an interactive video timeline further includes displaying the first set of frames of video in an interactive video timeline using the network client.
  • selecting video to stream and playback using an interactive video timeline further includes selecting a first displayed frame of video from the first set of frames of video in the interactive video timeline using the network client, selecting a second set of frames of video from the first stream of video data using the network client, and displaying the second set of frames of video in the interactive video timeline using the network client.
  • the frames of video in the first set of frames of video in the interactive video timeline are selected from the first stream of video data at intervals spaced by at least a first time interval and the frames of video in the second set of frames of video in the interactive video timeline are selected from the first stream of video data at intervals spaced by at least a second time interval.
  • the first time interval is greater than the second time interval.
  • the first time interval is equal to the second time interval.
  • the first time interval is smaller than the second time interval.
  • Still another embodiment of the invention includes a network client, including memory configured store a client application and video data and a processor, wherein the client application configures the processor to select a plurality of frames of video from a first stream of video data, display the plurality of frames of video in an interactive video timeline, select a playback frame of video data from the plurality of frames of video in the interactive video timeline, receive a second stream of video data beginning from the playback frame of video data, and play the second stream of video data.
  • the client application further configures the processor to receive a structure file from a media server, wherein the structure file contains the structure defining the layout of the interactive video timeline, receive the plurality of frames of video data from a media server, and create the interactive video timeline using the structure file and the plurality of frames of video data.
  • the client application further configures the processor to display an overlay for one or more frames of video in the plurality of frames of video, where the overlay includes metadata related to the one or more frames of video.
  • Yet another embodiment of the invention includes a machine readable medium containing processor instructions, where execution of the instructions by a processor causes the processor to perform a process including selecting a plurality of frames of video from a first stream of video data, displaying the plurality of frames of video in an interactive video timeline, selecting a playback frame of video data from the plurality of frames of video in the interactive video timeline, receiving a second stream of video data beginning from the playback frame of video data, and playing the second stream of video data.
  • Still another embodiment of the invention includes a media server, including a video decoder, memory configured store a server application and video data, and a processor, wherein the server application configures the processor to receive a request for an interactive video timeline, select a plurality of frames of video from a first stream of video data, render a plurality of images using the plurality of frames of video data and the video decoder, generate an interactive video timeline using the plurality of images, and transmit the interactive video timeline in response to the request for the interactive video timeline.
  • FIG. 1 is a system diagram of a system for the adaptive streaming of video data in accordance with an embodiment of the invention.
  • FIG. 2 conceptually illustrates a network configured to decode video data and utilize interactive video timelines in adaptive streaming systems in accordance with an embodiment of the invention.
  • FIG. 3 is a flow chart illustrating a process for creating a linear timeline of video data for use in adaptive streaming systems in accordance with an embodiment of the invention.
  • FIG. 4 is a flow chart illustrating a process for creating a hierarchical timeline of video data for use in adaptive streaming systems in accordance with an embodiment of the invention.
  • FIG. 5 is an illustration of a linear timeline of video data for use in adaptive streaming systems in accordance with an embodiment of the invention.
  • FIG. 6 is an illustration of a linear timeline of video data for use in adaptive streaming systems with an overlay in accordance with an embodiment of the invention.
  • Interactive video timelines provide a user interface allowing for the efficient search and playback of video content using a variety of input methods, including touchscreens, keyboards, mice, and remote controls.
  • linear interactive video timelines present a set of frames of video. Upon selecting a frame of video data, playback of the corresponding video stream (and any accompanying streams) begins from the location of the selected frame.
  • a first or initial set of frames of video is displayed. In many embodiments, the frames in the first set of frames are separated by at least a first time interval.
  • a second set of frames is displayed.
  • the second set of frames includes frames bounded by the time interval between the time of the frames from the first set of frames following and preceding the selected frame.
  • the frames in the second set of frames are separated by at least a second time interval.
  • the second time interval is shorter than the first time interval.
  • the user interface can provide the option to playback the video content from point in the content corresponding to the selected frame of video data.
  • the corresponding video stream can be played back from one of the frames, or a more granular selection of frames can again be displayed. Overlays may be displayed over the displayed frames of video.
  • overlays include a variety of information, including, but not limited to, time indicators, advertisements, prices, and/or other metadata associated with the frame of video data.
  • playback of video streams can occur in a full-screen mode, in a separate window, and/or within the interactive video timeline itself.
  • Adaptive streaming systems are configured to stream multimedia content encoded at different maximum bitrates and resolutions over a network, such as the Internet. Adaptive streaming systems stream the highest quality multimedia content, which can be supported based upon current streaming conditions. Multimedia content typically includes video and audio data, subtitles, and other related metadata.
  • adaptive streaming systems are configured to switch between the available sources of video data throughout the delivery of the video data according to a variety of factors, including, but not limited to, the available network data rate and video decoder performance.
  • available source of video data include alternative streams of video data encoded from source video data.
  • network clients in adaptive streaming systems When displaying a video stream, network clients in adaptive streaming systems often initialize a video decoder in order to decode and display the video stream.
  • Systems and methods for adaptive streaming systems in accordance with embodiments of the invention are described in U.S. Patent Application No. 13/221 ,682, entitled “Systems and Methods for Adaptive Streaming of Media Stored in Matroska Container Files Using Hypertext Transfer Protocol" to Braness et al., filed August 30, 201 1 , the disclosure of which is incorporated by reference herein in its entirety.
  • network clients in adaptive streaming systems utilize trick play streams to provide advanced playback features, including smooth visual search.
  • Trick play streams encode multimedia content in such a way that playback of the trick play stream appears to the user to be multimedia content played back at a higher speed.
  • the trick play stream can be utilized to perform a smooth visual search as opposed to the disjointed sequence of frames experienced when visual search is performed by jumping between intra or IDR frames in one of the video streams utilized for normal playback.
  • trick play streams consist of key frames from one or more of the video streams utilized for normal playback.
  • each stream of video data, including the trick play stream is stored in a Matroska (MKV) container file.
  • MKV Matroska
  • interactive timelines can be constructed using the intra or IDR frames of a video stream used for playback and/or frames of video data contained in a trick play stream.
  • interactive video timelines provide an interface for browsing streams of video data before renting or purchasing the full version of the multimedia content to which the video data belongs. For example, users can browse through selected scenes from a stream of video data and play back portions of the full media asset as a preview of the multimedia content.
  • Systems and methods for browsing and purchasing multimedia content in accordance with embodiments of the invention are disclosed in U.S. Patent Application No. 13/436,900, entitled “Systems and Methods for Accessing Digital Content Using Electronic Tickets and Ticket Tokens" to Ziskind Ticket Tokens" to Ziskind et al., filed March 31 , 2012, the disclosure of which is incorporated by reference herein in its entirety.
  • Systems and methods for adaptive streaming systems with interactive video timelines in accordance with embodiments of the invention are discussed further below.
  • Video data networks in accordance with embodiments of the invention are configured to adapt the bitrate of the video transmitted to network clients based on streaming conditions and to enable browsing of video content using interactive video timelines.
  • a video data network in accordance with an embodiment of the invention is illustrated in FIG. 1 .
  • the illustrated video data network 10 includes a video source 100.
  • the video source 100 contains pre- encoded video data encoded at multiple bitrates and resolutions for adaptive streaming.
  • the video source 100 encodes video data in real time.
  • the video source 100 contains video data encoded utilizing Scalable Video Coding (SVC).
  • SVC Scalable Video Coding
  • the video source 100 contains multiple streams with equal timelines as video data.
  • the video source 100 contains trick play streams as video data.
  • the video source 100 is connected to a network renderer 102.
  • the network renderer 102 is connected to a plurality of network clients 104 via a network 1 10.
  • the network renderer 102 is configured to stream video data to one or more network clients 104.
  • the network renderer 102 is implemented using a single machine. In several embodiments of the invention, the network renderer 102 is implemented using a plurality of machines. In many embodiments of the invention, the network renderer 102 and the video source 100 are implemented using a media server. In many embodiments, the network 1 10 is the Internet. In several embodiments, the network 1 10 is any IP network. In a number of embodiments, the network 1 10 is a cellular data network. [0039] The network clients 104 each contain a video decoder 106 and a client application 108. Network clients 104 are configured to decode video streams using the video decoder 106. In many embodiments, the network clients 104 are configured to switch between video streams based on a variety of factors, including the bandwidth and/or latency of the network connection and the performance of the video decoder 106.
  • the client application 108 is configured to create and control the creation of interactive video timelines.
  • the client application 108 is configured to create interactive video timelines using frames of video from one or more video streams.
  • the client application 108 is configured to create interactive video timelines using trick play streams.
  • the client application 108 is configured to display interactive video timelines retrieved from a media server.
  • an interactive video timeline is created by rendering an image containing frames from one or more video streams.
  • an interactive video timeline is created using an interactive page containing images rendered from frames from one or more video streams.
  • the interactive page may have a structure described in, but not limited to, hypertext markup language (HTML), extensible markup language (XML), and Adobe Flash, a product of Adobe Systems of San Jose, California.
  • the client application 108 is configured to overlay metadata related to the frames of video incorporated in the interactive video timeline.
  • the client application 108 controls the playback of video streams selected using the interactive timeline using the video decoder 106.
  • network clients can include consumer electronics devices such as DVD players, Blu-ray players, televisions, set top boxes, video game consoles, tablets, and other devices that are capable of connecting to a server and playing back encoded media.
  • the basic architecture of a network client in accordance with an embodiment of the invention is illustrated in FIG. 2.
  • the network client 200 includes a processor 210 in communication with non-volatile memory 230 and volatile memory 220.
  • the non-volatile 230 memory includes a video decoder 232 that configures the processor to decode scalable video data.
  • the non-volatile memory 230 further includes a client application 234 configured to create, operate, and terminate the video decoder 232 and to create, operate, and terminate interactive video timelines.
  • the video decoder 232 and/or the client application 234 are implemented using the processor 210.
  • the video decoder 232 and/or the client application 234 are implemented using dedicated hardware.
  • the network client 200 also includes a network interface 240 configured to send and receive data over a network connection.
  • the network interface 240 is in communication with the processor 210, the non-volatile memory 230, and the volatile memory 220.
  • the network client 200 includes multiple processors.
  • the video decoder 232 and/or the client application 234 are utilize separate processors.
  • the video decoder 232 and/or the client application 234 are implemented using dedicated hardware.
  • FIG. 2 a specific network client architecture is illustrated in FIG. 2, any of a variety of architectures including architectures where the video decoder is located on disk or some other form of storage, including cloud storage, and is loaded into volatile memory at runtime can be utilized to implement network clients for use in adaptive video data streaming systems with interactive video timelines in accordance with embodiments of the invention.
  • FIG. 1 Although a specific architecture of a video data network that supports selection of streams using an interactive video timeline is shown in FIG. 1 , other implementations appropriate to a specific application, including those which render interactive video timelines in a manner not specifically described above, can be utilized in accordance with embodiments of the invention. Processes for adaptive streaming with interactive video timelines in accordance with embodiments of the invention are discussed further below. Linear Interactive Video Timelines
  • FIG. 3 A process for adaptive streaming using linear interactive video timelines in accordance with an embodiment of the invention is illustrated in FIG. 3.
  • the process 300 includes selecting (310) which frames of video are to be displayed in a linear interactive video timeline.
  • a linear interactive video timeline is created (312).
  • an overlay is displayed (314) on one or more of the frames of video in the linear interactive video timeline.
  • a frame of video is selected (316). Playback of the video stream begins (318) from the selected (316) frame of video.
  • the frames of video that are displayed as part of the interactive video timeline are a plurality of frames of video selected from a stream of video data.
  • the frames of video are selected from a trick play stream.
  • the frames are an entire trick play stream.
  • a network client selects the frames of video used to create (312) a linear interactive video timeline.
  • the network client renders images from the selected (310) frames of video, which are used to create (312) the linear interactive video timeline.
  • the selected (310) frames of video may be retrieved from a media server.
  • a media server selects the frames of video used to create (312) a linear interactive video timeline.
  • the network client can retrieve the structure of and display the linear interactive video timeline, which can be used to select content to stream to the network client.
  • the media server selects (310) frames of video and the network client receives the selected frames during the creation (312) of a linear interactive video timeline.
  • a linear interactive video timeline can take a variety of forms, including, but not limited to, a grid of frames of video.
  • the grid of frames of video exceeds the display size; in these embodiments, users may use a variety of interface techniques to view the entire grid of frames of video, including flipping through pages of frames of video or scrolling in a variety of directions to view the entire grid.
  • a still image representing the selected (310) frames is retrieved from a media server and used in the creation (312) and display of a linear interactive video timeline.
  • An example linear interactive video timeline is illustrated in FIG. 5.
  • the linear interactive video timeline 500 includes a plurality of frames of frames of video 510 arranged in a 4x4 grid.
  • the techniques utilized to display and interact with the grid of frames of video are equally applicable to those embodiments of the invention utilizing linear interactive video timelines with still images.
  • the displayed (314) overlay includes a variety of data, including, but not limited to, the time in the video stream corresponding to the displayed frame of video.
  • An example linear interactive video timeline with a displayed overlay is illustrated in FIG. 6.
  • the linear interactive video timeline 600 includes a plurality of frames of video 610 arranged in a 4x4 grid, where each frame has the time of the frame of video (620) displayed on top of the frame of video.
  • the metadata displayed in the overlay may be selected using a network client and/or selected using a media server and retrieved by a network client for display.
  • Playback of the video stream may begin (318) in a variety of ways in accordance with embodiments of the invention, including playback in a full screen mode, displacing the linear interactive video timeline, in a window displayed on top of the linear interactive video timeline, or within the linear interactive video timeline itself.
  • the video stream played back may be contained on the network client and/or retrieved from a media server.
  • beginning (318) playback of stream of video data from the selected (316) frame of video data involves measuring the network bandwidth available for adaptive streaming and identifying the location of the selected (316) frame of video data using the measured network bandwidth to determine where in the stream of video data playback should begin (318).
  • Hierarchical interactive video timelines allow for the browsing of video data in a rapid manner by showing a high level overview of the video data, and then progressively showing more detail depending on the portion of the video data selected.
  • a process for adaptive streaming with hierarchical interactive video timelines in accordance with an embodiment of the invention is illustrated in FIG. 4.
  • the process 400 includes selecting (410) a first set of frames of video from a stream of video data.
  • the frames of video are displayed (412).
  • a frame of video data is selected (414).
  • the stream of video data may be played (416) beginning from the selected frame of video data, in which case playback of the video stream begins (418) from the selected frame of video data.
  • the selected frame of video data from which playback begins (418) is the playback frame of video data. If the video data is not played (416), a second set of frames of video is selected (410) and the process begins again.
  • selecting (410) a first set of frames of video from stream of video data may be performed using a hierarchical index.
  • Systems and methods for creating a hierarchical index for a stream of video data in accordance with embodiments of the invention are described in U.S. Patent Application No. 13/340,623, entitled “Systems and Methods for Playing Back Alternative Streams of Protected Content Protected Using Common Cryptographic Information" to Kiefer et al., filed December 29, 201 1 , the disclosure of which is incorporated by reference herein in its entirety.
  • a network client creates a hierarchical index using a stream of video data in order to facilitate selecting sets of frames of video for presentation at different levels of a hierarchical interactive video timeline.
  • the hierarchical index is retrieved from a media server.
  • the entire hierarchical index may be retrieved, or, in a number of embodiments, only the portion of the hierarchical index used to generate different levels of a hierarchical interactive video timeline are retrieved as a user interacts with the timeline.
  • the hierarchical index may be created using one or more streams of video data contained in the multimedia content, including trick play streams.
  • the levels of the hierarchical interactive video timeline are generated using a conventional or non- hierarchical index to retrieve frames of video from a stream.
  • selecting (410) a set of frames of video from a stream of video data involves selecting a subset of frames of video representing sections of the video data to be displayed (412).
  • the frames of video are frames selected from a stream of video data.
  • the frames of video are selected from a trick play stream.
  • the displayed (412) frames of video may be selected using the network client displaying the hierarchical interactive video timeline in accordance with embodiments of the invention.
  • the displayed (412) frames of video may be retrieved from a media server, along with the structure of the hierarchical interactive video timeline. As described above with respect to FIG. 3, overlays may be displayed on the displayed (412) frames of video and the playback (418) of the video stream may occur in full screen, in a window, or within the hierarchical interactive video timeline.
  • first frames of video in the first set of frames of video are separated by at least a first time interval and the frames of video in the second set of frames of video are separated by at least a second time interval.
  • the first time interval may be less than, equal to, or greater than the second time interval.
  • the hierarchical interactive video timeline displays (412) a first set of frames of video correspond to the following time: 0 minutes, 30 minutes, 60 minutes, and 90 minutes. In this way, the first set of frames of video has a first time interval of at least 30 minutes.
  • the frame of video data corresponding to minute 60 is selected (414).
  • the second set of frames of video corresponding to 60, 70, 80, and 90 minutes are displayed.
  • the second set of frames of video has a second time interval of at least 10 minutes. Playback can begin (418) from any of the displayed frames of video, or, if the 70 minute frame of video data is selected, a third set of frames of video corresponding to minutes 70, 73, 76, and 80 can be displayed (412).

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Television Signal Processing For Recording (AREA)

Abstract

Systems and methods for adaptive streaming systems configured to generate interactive video timelines in accordance with embodiments of the invention are disclosed. In one embodiment of the invention, a method for selecting video to stream and playback using an interactive video timeline includes selecting a plurality of frames of video from a first stream of video data using a network client, displaying the plurality of frames of video in an interactive video timeline using the network client, selecting a playback frame of video data from the plurality of frames of video in the interactive video timeline using the network client, receiving a second stream of video data beginning from the playback frame of video data using the network client, and playing the second stream of video data using the network client.

Description

SYSTEMS AND METHODS FOR ADAPTIVE STREAMING SYSTEMS WITH
INTERACTIVE VIDEO TIMELINES
FIELD OF THE INVENTION
[0001] The present invention is directed, in general, to systems and methods for adaptive streaming systems and more specifically to systems and methods for adaptive streaming systems configured to generate interactive video timelines.
BACKGROUND
[0002] The term streaming media describes the playback of media on a playback device, where the media is stored on a server and continuously sent to the playback device over a network during playback. Typically, the playback device stores a sufficient quantity of media in a buffer at any given time during playback to prevent disruption of playback due to the playback device completing playback of all the buffered media prior to receipt of the next portion of media. Adaptive bitrate streaming or adaptive streaming involves detecting the present streaming conditions (e.g. the user's network bandwidth and CPU capacity) in real time and adjusting the quality of the streamed media accordingly. Streaming video over the Internet has become a phenomenon in modern times. Many popular websites, such as YouTube, a service of Google, Inc. of Mountain View, California, and WatchESPN, a service of ESPN of Bristol, Connecticut, utilize streaming video in order to provide video and television programming to consumers via the Internet.
[0003] Scalable Video Coding (SVC) is an extension of the H.264/MPEG-4 AVC video compression standard, which is specified by the ITU-T H.264 standard by the International Telecommunication Union Telecommunication Standardization Sector of Geneva, Switzerland. SVC enables the encoding of a video bitstream that additionally contains one or more sub-bitstreams. The sub-bitstreams are derived from the video bitstream by dropping packets of data from the video bitstream, resulting in a sub- bitstream of lower quality and lower bandwidth than the original video bitstream. SVC supports three forms of scaling a video bitstream into sub-bitstreams: temporal scaling, spatial scaling, and quality scaling. Each of these scaling techniques can be used individually or combined depending on the specific video system.
[0004] In adaptive streaming systems, the multimedia content is typically stored on a media server as a top level index file pointing to a number of alternate streams that contain the actual video and audio data. Each stream is typically stored in one or more container files. Different adaptive streaming solutions typically utilize different index and media containers. The Matroska container is a media container developed as an open standard project by the Matroska non-profit organization of Aussonne, France. The Matroska container is based upon Extensible Binary Meta Language (EBML), which is a binary derivative of the Extensible Markup Language (XML). Decoding of the Matroska container is supported by many consumer electronics (CE) devices. The DivX Plus file format developed by DivX, LLC of San Diego, California utilizes an extension of the Matroska container format, including elements that are not specified within the Matroska format.
SUMMARY OF THE INVENTION
[0005] Systems and methods for adaptive streaming systems configured to generate interactive video timelines in accordance with embodiments of the invention are disclosed. In one embodiment of the invention, a method for selecting video to stream and playback using an interactive video timeline includes selecting a plurality of frames of video from a first stream of video data using a network client, displaying the plurality of frames of video in an interactive video timeline using the network client, selecting a playback frame of video data from the plurality of frames of video in the interactive video timeline using the network client, receiving a second stream of video data beginning from the playback frame of video data using the network client, and playing the second stream of video data using the network client.
[0006] In another embodiment of the invention, the first stream of video data is a stream of intra frames.
[0007] In an additional embodiment of the invention, the first stream of video data and the second stream of video data are stored in container files. [0008] In yet another additional embodiment of the invention, the container files are Matroska container files.
[0009] In still another additional embodiment of the invention, the first stream of video data and the second stream of video data are encoded using scalable video coding.
[0010] In yet still another additional embodiment of the invention, the plurality of frames of video are displayed in a grid.
[0011] In yet another embodiment of the invention, playing the second stream of video data using the network client includes closing the interactive video timeline using the network client and playing the second stream of video data in a full screen mode using the network client.
[0012] In still another embodiment of the invention, playing the second stream of video data using the network client includes playing the second stream of video data in a window on top of the interactive video timeline using the network client.
[0013] In yet still another embodiment of the invention, playing the second stream of video data using the network client includes playing the second stream of video data in the interactive video timeline using the network client.
[0014] In yet another additional embodiment of the invention, displaying the plurality of frames of video in an interactive video timeline further includes receiving a structure file using the network client, wherein the structure file contains the structure defining the layout of the interactive video timeline, receiving the plurality of frames of video data using the network client, creating the interactive video timeline using the structure file, the plurality of frames of video data and the network client, and displaying the interactive video timeline using the network client.
[0015] In still another additional embodiment of the invention, selecting video to stream and playback using an interactive video timeline further includes displaying metadata for a plurality of frames of video in the plurality of frames of video in the interactive video timeline using the network client. [0016] In yet still another additional embodiment of the invention, selecting a plurality of frames of video from a first stream of video data further includes selecting a first set of frames of video from the first stream of video data using the network client and displaying the plurality of frames of video in an interactive video timeline further includes displaying the first set of frames of video in an interactive video timeline using the network client.
[0017] In yet another embodiment of the invention, selecting video to stream and playback using an interactive video timeline further includes selecting a first displayed frame of video from the first set of frames of video in the interactive video timeline using the network client, selecting a second set of frames of video from the first stream of video data using the network client, and displaying the second set of frames of video in the interactive video timeline using the network client.
[0018] In still another embodiment of the invention, the frames of video in the first set of frames of video in the interactive video timeline are selected from the first stream of video data at intervals spaced by at least a first time interval and the frames of video in the second set of frames of video in the interactive video timeline are selected from the first stream of video data at intervals spaced by at least a second time interval.
[0019] In yet still another embodiment of the invention, the first time interval is greater than the second time interval.
[0020] In yet another additional embodiment of the invention, the first time interval is equal to the second time interval.
[0021] In still another additional embodiment of the invention, the first time interval is smaller than the second time interval.
[0022] Still another embodiment of the invention includes a network client, including memory configured store a client application and video data and a processor, wherein the client application configures the processor to select a plurality of frames of video from a first stream of video data, display the plurality of frames of video in an interactive video timeline, select a playback frame of video data from the plurality of frames of video in the interactive video timeline, receive a second stream of video data beginning from the playback frame of video data, and play the second stream of video data. [0023] In yet another additional embodiment of the invention, the client application further configures the processor to receive a structure file from a media server, wherein the structure file contains the structure defining the layout of the interactive video timeline, receive the plurality of frames of video data from a media server, and create the interactive video timeline using the structure file and the plurality of frames of video data.
[0024] In still another additional embodiment of the invention, the client application further configures the processor to display an overlay for one or more frames of video in the plurality of frames of video, where the overlay includes metadata related to the one or more frames of video.
[0025] Yet another embodiment of the invention includes a machine readable medium containing processor instructions, where execution of the instructions by a processor causes the processor to perform a process including selecting a plurality of frames of video from a first stream of video data, displaying the plurality of frames of video in an interactive video timeline, selecting a playback frame of video data from the plurality of frames of video in the interactive video timeline, receiving a second stream of video data beginning from the playback frame of video data, and playing the second stream of video data.
[0026] Still another embodiment of the invention includes a media server, including a video decoder, memory configured store a server application and video data, and a processor, wherein the server application configures the processor to receive a request for an interactive video timeline, select a plurality of frames of video from a first stream of video data, render a plurality of images using the plurality of frames of video data and the video decoder, generate an interactive video timeline using the plurality of images, and transmit the interactive video timeline in response to the request for the interactive video timeline.
BRIEF DESCRIPTION OF THE DRAWINGS
[0027] FIG. 1 is a system diagram of a system for the adaptive streaming of video data in accordance with an embodiment of the invention. [0028] FIG. 2 conceptually illustrates a network configured to decode video data and utilize interactive video timelines in adaptive streaming systems in accordance with an embodiment of the invention.
[0029] FIG. 3 is a flow chart illustrating a process for creating a linear timeline of video data for use in adaptive streaming systems in accordance with an embodiment of the invention.
[0030] FIG. 4 is a flow chart illustrating a process for creating a hierarchical timeline of video data for use in adaptive streaming systems in accordance with an embodiment of the invention.
[0031] FIG. 5 is an illustration of a linear timeline of video data for use in adaptive streaming systems in accordance with an embodiment of the invention.
[0032] FIG. 6 is an illustration of a linear timeline of video data for use in adaptive streaming systems with an overlay in accordance with an embodiment of the invention.
DETAILED DESCRIPTION
[0033] Turning now to the drawings, systems and methods for adaptive streaming systems with interactive video timelines in accordance with embodiments of the invention are disclosed. Interactive video timelines provide a user interface allowing for the efficient search and playback of video content using a variety of input methods, including touchscreens, keyboards, mice, and remote controls. In accordance with embodiments of the invention, linear interactive video timelines present a set of frames of video. Upon selecting a frame of video data, playback of the corresponding video stream (and any accompanying streams) begins from the location of the selected frame. In hierarchical interactive video timelines, a first or initial set of frames of video is displayed. In many embodiments, the frames in the first set of frames are separated by at least a first time interval. When a frame is selected from the first set of frames, a second set of frames is displayed. In a number of embodiments, the second set of frames includes frames bounded by the time interval between the time of the frames from the first set of frames following and preceding the selected frame. In addition, the frames in the second set of frames are separated by at least a second time interval. In many embodiments, the second time interval is shorter than the first time interval. The user interface can provide the option to playback the video content from point in the content corresponding to the selected frame of video data. The corresponding video stream can be played back from one of the frames, or a more granular selection of frames can again be displayed. Overlays may be displayed over the displayed frames of video. In several embodiments, overlays include a variety of information, including, but not limited to, time indicators, advertisements, prices, and/or other metadata associated with the frame of video data. In accordance with many embodiments of the invention, playback of video streams can occur in a full-screen mode, in a separate window, and/or within the interactive video timeline itself.
[0034] In a number of embodiments, interactive timelines are utilized in streaming systems. Adaptive streaming systems are configured to stream multimedia content encoded at different maximum bitrates and resolutions over a network, such as the Internet. Adaptive streaming systems stream the highest quality multimedia content, which can be supported based upon current streaming conditions. Multimedia content typically includes video and audio data, subtitles, and other related metadata. In order to provide the highest quality video experience independent of the network data rate, adaptive streaming systems are configured to switch between the available sources of video data throughout the delivery of the video data according to a variety of factors, including, but not limited to, the available network data rate and video decoder performance. In accordance with embodiments of the invention, available source of video data include alternative streams of video data encoded from source video data. When displaying a video stream, network clients in adaptive streaming systems often initialize a video decoder in order to decode and display the video stream. Systems and methods for adaptive streaming systems in accordance with embodiments of the invention are described in U.S. Patent Application No. 13/221 ,682, entitled "Systems and Methods for Adaptive Streaming of Media Stored in Matroska Container Files Using Hypertext Transfer Protocol" to Braness et al., filed August 30, 201 1 , the disclosure of which is incorporated by reference herein in its entirety. [0035] In accordance with embodiments of the invention, network clients in adaptive streaming systems utilize trick play streams to provide advanced playback features, including smooth visual search. Trick play streams encode multimedia content in such a way that playback of the trick play stream appears to the user to be multimedia content played back at a higher speed. For example, the trick play stream can be utilized to perform a smooth visual search as opposed to the disjointed sequence of frames experienced when visual search is performed by jumping between intra or IDR frames in one of the video streams utilized for normal playback. In accordance with embodiments of the invention, trick play streams consist of key frames from one or more of the video streams utilized for normal playback. In many embodiments, each stream of video data, including the trick play stream, is stored in a Matroska (MKV) container file. Systems and methods for using and encoding trick play streams for use in adaptive streaming systems in accordance with embodiments of the invention are disclosed in U.S. Patent Application No. 13/223,186, entitled "Systems and Methods for Performing Smooth Visual Search of Media Encoded for Adaptive Streaming via Hypertext Transfer Protocol using Trick Play Streams" to Braness et al., filed August 31 , 201 1 , the disclosure of which is incorporated by reference herein in its entirety. As is discussed further below, interactive timelines can be constructed using the intra or IDR frames of a video stream used for playback and/or frames of video data contained in a trick play stream. Although embodiments of the invention are described herein with respect to adaptive streaming systems, the systems and methods described are equally applicable in conventional streaming systems, including conventional streaming systems that utilize trick play tracks.
[0036] In several embodiments, interactive video timelines provide an interface for browsing streams of video data before renting or purchasing the full version of the multimedia content to which the video data belongs. For example, users can browse through selected scenes from a stream of video data and play back portions of the full media asset as a preview of the multimedia content. Systems and methods for browsing and purchasing multimedia content in accordance with embodiments of the invention are disclosed in U.S. Patent Application No. 13/436,900, entitled "Systems and Methods for Accessing Digital Content Using Electronic Tickets and Ticket Tokens" to Ziskind et al., filed March 31 , 2012, the disclosure of which is incorporated by reference herein in its entirety. Systems and methods for adaptive streaming systems with interactive video timelines in accordance with embodiments of the invention are discussed further below.
System Overview
[0037] Video data networks in accordance with embodiments of the invention are configured to adapt the bitrate of the video transmitted to network clients based on streaming conditions and to enable browsing of video content using interactive video timelines. A video data network in accordance with an embodiment of the invention is illustrated in FIG. 1 . The illustrated video data network 10 includes a video source 100. In a number of embodiments of the invention, the video source 100 contains pre- encoded video data encoded at multiple bitrates and resolutions for adaptive streaming. In several embodiments of the invention, the video source 100 encodes video data in real time. In many embodiments of the invention, the video source 100 contains video data encoded utilizing Scalable Video Coding (SVC). In a number of embodiments, the video source 100 contains multiple streams with equal timelines as video data. In several embodiments, the video source 100 contains trick play streams as video data. The video source 100 is connected to a network renderer 102. The network renderer 102 is connected to a plurality of network clients 104 via a network 1 10. The network renderer 102 is configured to stream video data to one or more network clients 104.
[0038] In many embodiments of the invention, the network renderer 102 is implemented using a single machine. In several embodiments of the invention, the network renderer 102 is implemented using a plurality of machines. In many embodiments of the invention, the network renderer 102 and the video source 100 are implemented using a media server. In many embodiments, the network 1 10 is the Internet. In several embodiments, the network 1 10 is any IP network. In a number of embodiments, the network 1 10 is a cellular data network. [0039] The network clients 104 each contain a video decoder 106 and a client application 108. Network clients 104 are configured to decode video streams using the video decoder 106. In many embodiments, the network clients 104 are configured to switch between video streams based on a variety of factors, including the bandwidth and/or latency of the network connection and the performance of the video decoder 106.
[0040] In a number of embodiments, the client application 108 is configured to create and control the creation of interactive video timelines. In many embodiments, the client application 108 is configured to create interactive video timelines using frames of video from one or more video streams. In several embodiments, the client application 108 is configured to create interactive video timelines using trick play streams. In a number of embodiments, the client application 108 is configured to display interactive video timelines retrieved from a media server. In several embodiments, an interactive video timeline is created by rendering an image containing frames from one or more video streams. In a number of embodiments, an interactive video timeline is created using an interactive page containing images rendered from frames from one or more video streams. In accordance with many embodiments of the invention, the interactive page may have a structure described in, but not limited to, hypertext markup language (HTML), extensible markup language (XML), and Adobe Flash, a product of Adobe Systems of San Jose, California. In many embodiments, the client application 108 is configured to overlay metadata related to the frames of video incorporated in the interactive video timeline. In a number of embodiments, the client application 108 controls the playback of video streams selected using the interactive timeline using the video decoder 106.
[0041] In many embodiments of the invention, network clients can include consumer electronics devices such as DVD players, Blu-ray players, televisions, set top boxes, video game consoles, tablets, and other devices that are capable of connecting to a server and playing back encoded media. The basic architecture of a network client in accordance with an embodiment of the invention is illustrated in FIG. 2. The network client 200 includes a processor 210 in communication with non-volatile memory 230 and volatile memory 220. In the illustrated embodiment, the non-volatile 230 memory includes a video decoder 232 that configures the processor to decode scalable video data. In the illustrated embodiment, the non-volatile memory 230 further includes a client application 234 configured to create, operate, and terminate the video decoder 232 and to create, operate, and terminate interactive video timelines. In several embodiments, the video decoder 232 and/or the client application 234 are implemented using the processor 210. In a number of embodiments, the video decoder 232 and/or the client application 234 are implemented using dedicated hardware. The network client 200 also includes a network interface 240 configured to send and receive data over a network connection. In a number of embodiments, the network interface 240 is in communication with the processor 210, the non-volatile memory 230, and the volatile memory 220.
[0042] In many embodiments of the invention, the network client 200 includes multiple processors. In several embodiments, the video decoder 232 and/or the client application 234 are utilize separate processors. In a number of embodiments, the video decoder 232 and/or the client application 234 are implemented using dedicated hardware. Although a specific network client architecture is illustrated in FIG. 2, any of a variety of architectures including architectures where the video decoder is located on disk or some other form of storage, including cloud storage, and is loaded into volatile memory at runtime can be utilized to implement network clients for use in adaptive video data streaming systems with interactive video timelines in accordance with embodiments of the invention.
[0043] Although a specific architecture of a video data network that supports selection of streams using an interactive video timeline is shown in FIG. 1 , other implementations appropriate to a specific application, including those which render interactive video timelines in a manner not specifically described above, can be utilized in accordance with embodiments of the invention. Processes for adaptive streaming with interactive video timelines in accordance with embodiments of the invention are discussed further below. Linear Interactive Video Timelines
[0044] When viewing streams of video data, users often wish to skip to a particular place in the stream of video data. Linear interactive video timelines allow for the quick and efficient search and navigation of streams of video data. A process for adaptive streaming using linear interactive video timelines in accordance with an embodiment of the invention is illustrated in FIG. 3. The process 300 includes selecting (310) which frames of video are to be displayed in a linear interactive video timeline. A linear interactive video timeline is created (312). In accordance with embodiments of the invention, an overlay is displayed (314) on one or more of the frames of video in the linear interactive video timeline. A frame of video is selected (316). Playback of the video stream begins (318) from the selected (316) frame of video.
[0045] In many embodiments, the frames of video that are displayed as part of the interactive video timeline are a plurality of frames of video selected from a stream of video data. In several embodiments, the frames of video are selected from a trick play stream. In a number of embodiments, the frames are an entire trick play stream.
[0046] In many embodiments of the invention, a network client selects the frames of video used to create (312) a linear interactive video timeline. In many embodiments, the network client renders images from the selected (310) frames of video, which are used to create (312) the linear interactive video timeline. In several embodiments, the selected (310) frames of video may be retrieved from a media server.
[0047] In a number of embodiments of the invention, a media server selects the frames of video used to create (312) a linear interactive video timeline. The network client can retrieve the structure of and display the linear interactive video timeline, which can be used to select content to stream to the network client. In several embodiments, the media server selects (310) frames of video and the network client receives the selected frames during the creation (312) of a linear interactive video timeline.
[0048] A linear interactive video timeline can take a variety of forms, including, but not limited to, a grid of frames of video. In many embodiments, the grid of frames of video exceeds the display size; in these embodiments, users may use a variety of interface techniques to view the entire grid of frames of video, including flipping through pages of frames of video or scrolling in a variety of directions to view the entire grid. In a number of embodiments, a still image representing the selected (310) frames is retrieved from a media server and used in the creation (312) and display of a linear interactive video timeline. An example linear interactive video timeline is illustrated in FIG. 5. The linear interactive video timeline 500 includes a plurality of frames of frames of video 510 arranged in a 4x4 grid.
[0049] The techniques utilized to display and interact with the grid of frames of video are equally applicable to those embodiments of the invention utilizing linear interactive video timelines with still images. The displayed (314) overlay includes a variety of data, including, but not limited to, the time in the video stream corresponding to the displayed frame of video. An example linear interactive video timeline with a displayed overlay is illustrated in FIG. 6. The linear interactive video timeline 600 includes a plurality of frames of video 610 arranged in a 4x4 grid, where each frame has the time of the frame of video (620) displayed on top of the frame of video. In accordance with embodiments of the invention, the metadata displayed in the overlay may be selected using a network client and/or selected using a media server and retrieved by a network client for display.
[0050] Playback of the video stream may begin (318) in a variety of ways in accordance with embodiments of the invention, including playback in a full screen mode, displacing the linear interactive video timeline, in a window displayed on top of the linear interactive video timeline, or within the linear interactive video timeline itself. In accordance with embodiments of the invention, the video stream played back may be contained on the network client and/or retrieved from a media server. In many embodiments, there is a timestamp associated with the selected (316) frame of video data; beginning (318) playback of stream of video data from the selected (316) frame of video data involves measuring the network bandwidth available for adaptive streaming and identifying the location of the selected (316) frame of video data using the measured network bandwidth to determine where in the stream of video data playback should begin (318). [0051] Although a specific process for adaptive streaming with linear interactive video timelines is described above, a variety of processes not specifically described may be utilized in accordance with embodiments of the invention. A process for adaptive streaming with hierarchical interactive video timelines in accordance with an embodiment of the invention is discussed below.
Hierarchical Interactive Video Timelines
[0052] Hierarchical interactive video timelines allow for the browsing of video data in a rapid manner by showing a high level overview of the video data, and then progressively showing more detail depending on the portion of the video data selected. A process for adaptive streaming with hierarchical interactive video timelines in accordance with an embodiment of the invention is illustrated in FIG. 4. The process 400 includes selecting (410) a first set of frames of video from a stream of video data. The frames of video are displayed (412). A frame of video data is selected (414). The stream of video data may be played (416) beginning from the selected frame of video data, in which case playback of the video stream begins (418) from the selected frame of video data. In accordance with embodiments of the invention, the selected frame of video data from which playback begins (418) is the playback frame of video data. If the video data is not played (416), a second set of frames of video is selected (410) and the process begins again.
[0053] In accordance with embodiments of the invention, selecting (410) a first set of frames of video from stream of video data may be performed using a hierarchical index. Systems and methods for creating a hierarchical index for a stream of video data in accordance with embodiments of the invention are described in U.S. Patent Application No. 13/340,623, entitled "Systems and Methods for Playing Back Alternative Streams of Protected Content Protected Using Common Cryptographic Information" to Kiefer et al., filed December 29, 201 1 , the disclosure of which is incorporated by reference herein in its entirety. In a number of embodiments, a network client creates a hierarchical index using a stream of video data in order to facilitate selecting sets of frames of video for presentation at different levels of a hierarchical interactive video timeline. In several embodiments, the hierarchical index is retrieved from a media server. When a hierarchical index is retrieved, the entire hierarchical index may be retrieved, or, in a number of embodiments, only the portion of the hierarchical index used to generate different levels of a hierarchical interactive video timeline are retrieved as a user interacts with the timeline. In accordance with embodiments of the invention, the hierarchical index may be created using one or more streams of video data contained in the multimedia content, including trick play streams. In other embodiments, the levels of the hierarchical interactive video timeline are generated using a conventional or non- hierarchical index to retrieve frames of video from a stream.
[0054] In a number of embodiments, selecting (410) a set of frames of video from a stream of video data involves selecting a subset of frames of video representing sections of the video data to be displayed (412). In many embodiments, the frames of video are frames selected from a stream of video data. In several embodiments, the frames of video are selected from a trick play stream. The displayed (412) frames of video may be selected using the network client displaying the hierarchical interactive video timeline in accordance with embodiments of the invention. In several embodiments, the displayed (412) frames of video may be retrieved from a media server, along with the structure of the hierarchical interactive video timeline. As described above with respect to FIG. 3, overlays may be displayed on the displayed (412) frames of video and the playback (418) of the video stream may occur in full screen, in a window, or within the hierarchical interactive video timeline.
[0055] In several embodiments of the invention, first frames of video in the first set of frames of video are separated by at least a first time interval and the frames of video in the second set of frames of video are separated by at least a second time interval. The first time interval may be less than, equal to, or greater than the second time interval. For example, for a 120 minute long stream of video data, the hierarchical interactive video timeline displays (412) a first set of frames of video correspond to the following time: 0 minutes, 30 minutes, 60 minutes, and 90 minutes. In this way, the first set of frames of video has a first time interval of at least 30 minutes. The frame of video data corresponding to minute 60 is selected (414). The second set of frames of video corresponding to 60, 70, 80, and 90 minutes are displayed. The second set of frames of video has a second time interval of at least 10 minutes. Playback can begin (418) from any of the displayed frames of video, or, if the 70 minute frame of video data is selected, a third set of frames of video corresponding to minutes 70, 73, 76, and 80 can be displayed (412).
[0056] A specific process for adaptive streaming with hierarchical interactive video timelines is described above; however, a variety of processes for hierarchical interactive video timelines not specifically described above, including those not specifically described by way of example, may be utilized in accordance with embodiments of the invention.
[0057] Although the present invention has been described in certain specific aspects, many additional modifications and variations would be apparent to those skilled in the art. It is therefore to be understood that the present invention may be practiced otherwise than specifically described without departing from the scope and spirit of the present invention. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.

Claims

WHAT IS CLAIMED:
1 . A method for selecting video to stream and playback using an interactive video timeline, comprising:
selecting a plurality of frames of video from a first stream of video data using a network client;
displaying the plurality of frames of video in an interactive video timeline using the network client;
selecting a playback frame of video data from the plurality of frames of video in the interactive video timeline using the network client;
receiving a second stream of video data beginning from the playback frame of video data using the network client; and
playing the second stream of video data using the network client.
2. The method of claim 1 , wherein the first stream of video data is a stream of intra frames.
3. The method of claim 1 , wherein the first stream of video data and the second stream of video data are stored in container files.
4. The method of claim 3, wherein the container files are Matroska container files.
5. The method of claim 1 , wherein the first stream of video data and the second stream of video data are encoded using scalable video coding.
6. The method of claim 1 , wherein the plurality of frames of video are displayed in a grid.
7. The method of claim 1 , wherein playing the second stream of video data using the network client comprises:
closing the interactive video timeline using the network client; and playing the second stream of video data in a full screen mode using the network client.
8. The method of claim 1 , wherein playing the second stream of video data using the network client comprises playing the second stream of video data in a window on top of the interactive video timeline using the network client.
9. The method of claim 1 , wherein playing the second stream of video data using the network client comprises playing the second stream of video data in the interactive video timeline using the network client.
10. The method of claim 1 , wherein displaying the plurality of frames of video in an interactive video timeline further comprises:
receiving a structure file using the network client, wherein the structure file contains the structure defining the layout of the interactive video timeline;
receiving the plurality of frames of video data using the network client; creating the interactive video timeline using the structure file, the plurality of frames of video data; and the network client; and
displaying the interactive video timeline using the network client.
1 1 . The method of claim 1 , further comprising displaying metadata for a plurality of frames of video in the plurality of frames of video in the interactive video timeline using the network client.
12. The method of claim 1 , wherein:
selecting a plurality of frames of video from a first stream of video data further comprises selecting a first set of frames of video from the first stream of video data using the network client; and displaying the plurality of frames of video in an interactive video timeline further comprises displaying the first set of frames of video in an interactive video timeline using the network client.
13. The method of claim 12, further comprising:
selecting a first displayed frame of video from the first set of frames of video in the interactive video timeline using the network client;
selecting a second set of frames of video from the first stream of video data using the network client; and
displaying the second set of frames of video in the interactive video timeline using the network client.
14. The method of claim 13, wherein:
the frames of video in the first set of frames of video in the interactive video timeline are selected from the first stream of video data at intervals spaced by at least a first time interval; and
the frames of video in the second set of frames of video in the interactive video timeline are selected from the first stream of video data at intervals spaced by at least a second time interval.
15. The method of claim 14, wherein the first time interval is greater than the second time interval.
16. The method of claim 14, wherein the first time interval is equal to the second time interval.
17. The method of claim 14, wherein the first time interval is smaller than the second time interval.
18. A network client, comprising:
memory configured store a client application and video data; and a processor;
wherein the client application configures the processor to:
select a plurality of frames of video from a first stream of video data;
display the plurality of frames of video in an interactive video timeline;
select a playback frame of video data from the plurality of frames of video in the interactive video timeline;
receive a second stream of video data beginning from the playback frame of video data; and
play the second stream of video data.
19. The network client of claim 18, wherein the client application further configures the processor to:
receive a structure file from a media server, wherein the structure file contains the structure defining the layout of the interactive video timeline;
receive the plurality of frames of video data from a media server; and create the interactive video timeline using the structure file and the plurality of frames of video data.
20. The network client of claim 18, wherein the client application further configures the processor to display an overlay for one or more frames of video in the plurality of frames of video, where the overlay comprises metadata related to the one or more frames of video.
21 . A machine readable medium containing processor instructions, where execution of the instructions by a processor causes the processor to perform a process comprising:
selecting a plurality of frames of video from a first stream of video data; displaying the plurality of frames of video in an interactive video timeline; selecting a playback frame of video data from the plurality of frames of video in the interactive video timeline;
receiving a second stream of video data beginning from the playback frame of video data; and
playing the second stream of video data.
22. A media server, comprising:
a video decoder;
memory configured store a server application and video data; and a processor;
wherein the server application configures the processor to:
receive a request for an interactive video timeline;
select a plurality of frames of video from a first stream of video data;
render a plurality of images using the plurality of frames of video data and the video decoder;
generate an interactive video timeline using the plurality of images; and
transmit the interactive video timeline in response to the request for the interactive video timeline.
PCT/US2013/043176 2012-06-13 2013-05-29 Systems and methods for adaptive streaming systems with interactive video timelines WO2013188112A2 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US13/495,757 US9354799B2 (en) 2012-06-13 2012-06-13 Systems and methods for adaptive streaming systems with interactive video timelines
US13/495,757 2012-06-13

Publications (2)

Publication Number Publication Date
WO2013188112A2 true WO2013188112A2 (en) 2013-12-19
WO2013188112A3 WO2013188112A3 (en) 2014-03-06

Family

ID=49757149

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2013/043176 WO2013188112A2 (en) 2012-06-13 2013-05-29 Systems and methods for adaptive streaming systems with interactive video timelines

Country Status (2)

Country Link
US (1) US9354799B2 (en)
WO (1) WO2013188112A2 (en)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5635850B2 (en) * 2010-09-16 2014-12-03 任天堂株式会社 Information processing apparatus, information processing program, information processing system, and information processing method
WO2014028068A1 (en) 2012-08-17 2014-02-20 Flextronics Ap, Llc Media center
GB2516826B (en) * 2013-07-23 2016-06-22 Canon Kk Method, device and computer program for encapsulating partitioned timed media data by creating tracks to be independently encapsulated in at least one media f
US9621613B1 (en) * 2013-11-05 2017-04-11 Visualon, Inc. Bitrate adaptation transitioning using key frames
US10764343B2 (en) 2015-12-28 2020-09-01 Google Llc Methods, systems, and media for navigating through a stream of content items
KR101939806B1 (en) 2016-09-07 2019-01-17 (주)잼투고 Computer Implemented Method and User Device for Searching Video Section Using Touch Sensitive User Interface
CN111565334B (en) * 2020-04-30 2021-12-28 广州酷狗计算机科技有限公司 Live broadcast playback method, device, terminal, server and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080184120A1 (en) * 2007-01-31 2008-07-31 Obrien-Strain Eamonn Concurrent presentation of video segments enabling rapid video file comprehension
US20090125812A1 (en) * 2007-10-17 2009-05-14 Yahoo! Inc. System and method for an extensible media player
US20110268178A1 (en) * 2009-08-18 2011-11-03 Anthony Neal Park Encoding video streams for adaptive video streaming
US20120039582A1 (en) * 2009-04-20 2012-02-16 Koninklijke Philips Electronics N.V. Verification and synchronization of files obtained separately from a video content

Family Cites Families (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6069606A (en) * 1997-05-15 2000-05-30 Sony Corporation Display of multiple images based on a temporal relationship among them with various operations available to a user as a function of the image size
US7559039B2 (en) * 1998-07-14 2009-07-07 Brad Ridgley Method and device for finding, collecting and acting upon units of information
GB0004688D0 (en) * 2000-02-28 2000-04-19 Radley Smith Philip J Bracelet
US6807361B1 (en) * 2000-07-18 2004-10-19 Fuji Xerox Co., Ltd. Interactive custom video creation system
US7246322B2 (en) * 2002-07-09 2007-07-17 Kaleidescope, Inc. Grid-like guided user interface for video selection and display
US7194527B2 (en) * 2002-06-18 2007-03-20 Microsoft Corporation Media variations browser
JP2004164618A (en) * 2002-10-14 2004-06-10 Oce Technol Bv Selection mechanism in portable terminal
US9756349B2 (en) * 2002-12-10 2017-09-05 Sony Interactive Entertainment America Llc User interface, system and method for controlling a video stream
US8572104B2 (en) * 2003-04-18 2013-10-29 Kaleidescape, Inc. Sales of collections excluding those already purchased
US20050097135A1 (en) * 2003-04-18 2005-05-05 Ian Epperson Touch panel user interface
US7120619B2 (en) * 2003-04-22 2006-10-10 Microsoft Corporation Relationship view
TW200506727A (en) * 2003-08-14 2005-02-16 Behavior Tech Computer Corp Method for processing multi-functional menu of man-machine-interface device, and its man-machine-interface system
US20050138564A1 (en) * 2003-12-17 2005-06-23 Fogg Brian J. Visualization of a significance of a set of individual elements about a focal point on a user interface
US7437005B2 (en) * 2004-02-17 2008-10-14 Microsoft Corporation Rapid visual sorting of digital files and data
AU2004240229B2 (en) * 2004-12-20 2011-04-07 Canon Kabushiki Kaisha A radial, three-dimensional, hierarchical file system view
US8065604B2 (en) * 2004-12-30 2011-11-22 Massachusetts Institute Of Technology Techniques for relating arbitrary metadata to media files
US7730425B2 (en) * 2005-11-30 2010-06-01 De Los Reyes Isabelo Function-oriented user interface
US7877707B2 (en) * 2007-01-06 2011-01-25 Apple Inc. Detecting and interpreting real-world and security gestures on touch and hover sensitive devices
US7956848B2 (en) * 2007-09-04 2011-06-07 Apple Inc. Video chapter access and license renewal
WO2009042858A1 (en) * 2007-09-28 2009-04-02 Gracenote, Inc. Synthesizing a presentation of a multimedia event
US8584048B2 (en) * 2008-05-29 2013-11-12 Telcordia Technologies, Inc. Method and system for multi-touch-based browsing of media summarizations on a handheld device
US8555170B2 (en) * 2010-08-10 2013-10-08 Apple Inc. Tool for presenting and editing a storyboard representation of a composite presentation
US8914534B2 (en) * 2011-01-05 2014-12-16 Sonic Ip, Inc. Systems and methods for adaptive bitrate streaming of media stored in matroska container files using hypertext transfer protocol

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080184120A1 (en) * 2007-01-31 2008-07-31 Obrien-Strain Eamonn Concurrent presentation of video segments enabling rapid video file comprehension
US20090125812A1 (en) * 2007-10-17 2009-05-14 Yahoo! Inc. System and method for an extensible media player
US20120039582A1 (en) * 2009-04-20 2012-02-16 Koninklijke Philips Electronics N.V. Verification and synchronization of files obtained separately from a video content
US20110268178A1 (en) * 2009-08-18 2011-11-03 Anthony Neal Park Encoding video streams for adaptive video streaming

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
'SRCA2009.Apple iMovie on iPad 2 guided tour.' YOUTUBE, [Online] 2011, Retrieved from the Internet: <URL:http://www.youtube.com/watch?v=JztCEHpq7yc> [retrieved on 2013-11-25] *

Also Published As

Publication number Publication date
US9354799B2 (en) 2016-05-31
US20130339855A1 (en) 2013-12-19
WO2013188112A3 (en) 2014-03-06

Similar Documents

Publication Publication Date Title
US9354799B2 (en) Systems and methods for adaptive streaming systems with interactive video timelines
US11895348B2 (en) Systems and methods for providing variable speeds in a trick-play mode
US11546643B2 (en) Systems and methods for providing audio content during trick-play playback
US9804668B2 (en) Systems and methods for rapid content switching to provide a linear TV experience using streaming content distribution
US10591984B2 (en) Systems and methods for rapid content switching to provide a linear TV experience using streaming content distribution
KR20130139970A (en) A client, a content creator entity and methods thereof for media streaming
JP2015523789A (en) System and method for reuse of encoding information in encoding an alternative stream of video data
US9060184B2 (en) Systems and methods for adaptive streaming with augmented video stream transitions using a media server
US20140282250A1 (en) Menu interface with scrollable arrangements of selectable elements
WO2013163221A1 (en) Systems and methods for adaptive streaming with augmented video stream transitions
US20150172347A1 (en) Presentation of content based on playlists
US20150163564A1 (en) Content distribution/consumption with tertiary content
US20130287092A1 (en) Systems and Methods for Adaptive Streaming with Augmented Video Stream Transitions

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 13803720

Country of ref document: EP

Kind code of ref document: A2

122 Ep: pct application non-entry in european phase

Ref document number: 13803720

Country of ref document: EP

Kind code of ref document: A2