US20180191868A1 - Method and System for Downloading Multiple Resolutions Bitrate for Virtual Reality (VR) Video Streaming Optimization - Google Patents

Method and System for Downloading Multiple Resolutions Bitrate for Virtual Reality (VR) Video Streaming Optimization Download PDF

Info

Publication number
US20180191868A1
US20180191868A1 US15/860,430 US201815860430A US2018191868A1 US 20180191868 A1 US20180191868 A1 US 20180191868A1 US 201815860430 A US201815860430 A US 201815860430A US 2018191868 A1 US2018191868 A1 US 2018191868A1
Authority
US
United States
Prior art keywords
video
bitrate
resolution
resolutions
video segments
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US15/860,430
Inventor
Zhuo Wang
Yongtao Tang
Ruoxi Zhao
Haoyan Zu
Chia-Chi Chang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Black Sails Technology Inc
Original Assignee
Black Sails Technology Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Black Sails Technology Inc filed Critical Black Sails Technology Inc
Priority to US15/860,430 priority Critical patent/US20180191868A1/en
Assigned to Black Sails Technology Inc. reassignment Black Sails Technology Inc. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: CHANG, CHIA-CHI, TANG, YONGTAO, WANG, ZHUO, ZHAO, RUOXI, ZU, HAOYAN
Publication of US20180191868A1 publication Critical patent/US20180191868A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • H04L67/38
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00Details of colour television systems
    • H04N9/12Picture reproducers
    • H04N9/31Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H04N9/3179Video signal processing therefor
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/398Synchronisation thereof; Control thereof
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012Head tracking input arrangements
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/003D [Three Dimensional] image rendering
    • G06T15/04Texture mapping
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/003D [Three Dimensional] image rendering
    • G06T15/10Geometric effects
    • G06T15/20Perspective computation
    • G06T15/205Image-based rendering
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00Three dimensional [3D] modelling, e.g. data description of 3D objects
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00Image coding
    • G06T9/001Model-based coding, e.g. wire frame
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00Arrangements for monitoring or testing data switching networks
    • H04L43/08Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H04L43/0876Network utilisation, e.g. volume of load or congestion level
    • H04L43/0888Throughput
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60Network streaming of media packets
    • H04L65/75Media network packet handling
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60Network streaming of media packets
    • H04L65/75Media network packet handling
    • H04L65/752Media network packet handling adapting media to network capabilities
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/131Protocols for games, networked simulations or virtual reality
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106Processing image signals
    • H04N13/111Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
    • H04N13/117Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106Processing image signals
    • H04N13/122Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106Processing image signals
    • H04N13/139Format conversion, e.g. of frame-rate or size
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106Processing image signals
    • H04N13/161Encoding, multiplexing or demultiplexing different image signal components
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/189Recording image signals; Reproducing recorded image signals
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/275Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/332Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/332Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H04N13/344Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/366Image reproducers using viewer tracking
    • H04N13/378Image reproducers using viewer tracking for tracking rotational head movements around an axis perpendicular to the screen
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/366Image reproducers using viewer tracking
    • H04N13/383Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/40Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/44Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/70Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/231Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/233Processing of audio elementary streams
    • H04N21/2335Processing of audio elementary streams involving reformatting operations of audio signals, e.g. by converting from one coding standard to another
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234345Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234363Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the spatial resolution, e.g. for clients with a lower screen resolution
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234381Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/238Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams
    • H04N21/2387Stream processing in response to a playback request from an end-user, e.g. for trick-play
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/239Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests
    • H04N21/2393Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests involving handling client requests
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/24Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
    • H04N21/2405Monitoring of the internal components or processes of the server, e.g. server load
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/266Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
    • H04N21/2662Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41Structure of client; Structure of client peripherals
    • H04N21/4104Peripherals receiving signals from specially adapted client devices
    • H04N21/4122Peripherals receiving signals from specially adapted client devices additional display device, e.g. video projector
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44012Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H04N21/440236Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81Monomedia components thereof
    • H04N21/816Monomedia components thereof involving special video data, e.g 3D video
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/845Structuring of content, e.g. decomposing content into time segments
    • H04N21/8456Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/76Television signal recording
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00Details of colour television systems
    • H04N9/12Picture reproducers
    • H04N9/31Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00Details of colour television systems
    • H04N9/12Picture reproducers
    • H04N9/31Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H04N9/3179Video signal processing therefor
    • H04N9/3185Geometric adjustment, e.g. keystone or convergence
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00Details of colour television systems
    • H04N9/79Processing of colour television signals in connection with recording
    • H04N9/87Regeneration of colour television signals
    • H04N9/8715Regeneration of colour television signals involving the mixing of the reproduced video signal with a non-recorded signal, e.g. a text signal
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2215/00Indexing scheme for image rendering
    • G06T2215/16Using real world measurements to influence rendering
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformations in the plane of the image
    • G06T3/40Scaling of whole images or parts thereof, e.g. expanding or contracting

Definitions

  • the present disclosure relates to video processing technology, and more particularly, to a method and a system for downloading a VR video.
  • Virtual Reality is a computer simulation technology for creating and experiencing a virtual world. For example, a three-dimensional real-time image can be presented based on a technology which tracks a user's head, eyes or hands.
  • full-view video data can be pre-stored on a server, and then transmitted to a display device.
  • a display device can be glasses, a head-mounted display, etc.
  • a video is displayed on the display device in accordance with a field of view of the user.
  • a high-resolution video data occupies large transmission of bandwidth and requires high computing power from the display devices.
  • Presenting high-resolution VR video on internet is difficult, precisely, the existing video streaming technology cannot fulfill the virtual reality. Therefore, in order to present VR video smoothly in real-time, it is desirable to further improve the existing video streaming technology to save bandwidth and reduce performance requirements for display devices.
  • the present disclosure provides a method and a system for downloading a VR video for optimizing video transmission.
  • a method for downloading a VR video comprising: receiving a play request for a first video; calculating an average download speed; selecting a plurality of first video segments from a plurality of video segments of the first video according to the average download speed, wherein the plurality of video segments of the first video correspond to different resolutions and bitrates; determining a download link according to the plurality of first video segments; and downloading the plurality of video segments according to the download link.
  • the method further comprises: obtaining a mapping table between resolutions and bitrates during a video transcoding process; and determining the plurality of first video segments by looking up the mapping table.
  • the step of obtaining the mapping table between resolutions and bitrates during the video transcoding process comprises: defining a plurality of resolution levels, and calculating a plurality of resolutions in the plurality of resolution levels according to a predetermined target resolution; and calculating bitrates corresponding to the resolutions in the plurality of resolution levels based on a target bitrate corresponding to the predetermined target resolution.
  • the step of defining a plurality of resolution levels, and calculating a plurality of resolutions in the plurality of resolution levels according to a predetermined target resolution comprises: calculating the resolution in one of the plurality of resolution levels following formulas below,
  • n represents the one of the plurality of resolution levels
  • R0 represents a maximum value of the plurality of resolutions
  • ⁇ X 0 enhance Y 0 enhance ⁇ represents the resolution of an enhancement video corresponding to the one of the plurality of resolution levels
  • q represents a resolution ratio
  • R n ⁇ X n , Y n ⁇ represents the resolution in the one of the plurality of resolution levels
  • the step of calculating bitrates corresponding to the plurality of resolutions in the plurality of resolution levels based on a target bitrate of the predetermined target resolution comprises: calculating the bitrates corresponding to the resolutions in the plurality of resolution levels following formulas below,
  • bitrate n bitrate n enhance and bitrate n base , there is:
  • bitrate n + 1 bitrate n q 2 ,
  • bitrate n base > 1 q 2 - 1 ⁇ bitrate n enhance ,
  • bitrate 0 represents a maximum value of the bitrates
  • bitrate n represents a maximum value of the bitrates
  • bitrate n base represents bitrate n + bitrate n enhance respectively represent the bitrate, the bitrate of the basic video, and the bitrate of the enhancement video corresponding to the one of the plurality of resolution levels, wherein n is an integer equal to or larger than 1.
  • the play request comprises a field of view of a user
  • the plurality of first video segments have a plurality of resolutions
  • a resolution distribution of the plurality of first video segments is determined by the field of view of the user.
  • the resolution of a first video segment within the field of view is higher than the resolution of a first video segment outside the field of view.
  • the average download speed and the resolution distribution of the plurality of first video segments are positively correlated.
  • a system for downloading a VR video comprising: a request receiving module configured to receive a play request of a first video; a speed calculating module configured to calculate an average download speed; a segment obtaining module configured to select a plurality of first video segments from a plurality of video segments of the first video according to the average download speed, wherein the plurality of video segments of the first video correspond to different resolutions and bitrates; a link obtaining module configured to determine a download link according to the plurality of first video segments; and a download executing module configured to download the plurality of video segments according to the download link.
  • the system further comprises: a mapping table obtaining module configured to obtaining a mapping table between resolutions and bitrates during a video transcoding process, and to determine the plurality of first video segments by looking up the mapping table.
  • a mapping table obtaining module configured to obtaining a mapping table between resolutions and bitrates during a video transcoding process, and to determine the plurality of first video segments by looking up the mapping table.
  • the mapping table obtaining module comprises: a first calculating unit configured to define a plurality of resolution levels and calculate a plurality of resolutions in the plurality of resolution levels according to the predetermined target resolution; and calculating bitrates corresponding to the resolutions in the plurality of resolution levels based on a target bitrate corresponding to the predetermined target resolution.
  • the present disclosure provides a plurality of video segments with proper resolutions and proper bitrates for downloading according to the average download speed.
  • a resolution reflects the image precision
  • a bitrate reflects bit number of data transmitted per unit time
  • the resolution, the bitrate and average download speed can be synthesized to optimize the video transmission by using an optimal strategy to select the appropriate video segments.
  • the video segments with high resolutions and low resolutions are selected according to the user's field of view, which can decrease the amount of transferred data without affecting the user's demands for viewing the video.
  • FIG. 1 is a diagram illustrating an example network of a VR playback system
  • FIG. 2 is a flowchart showing a method used in the VR playback system of FIG. 1 ;
  • FIG. 3 is a flowchart presenting the method for downloading a VR video according to an embodiment of the present disclosure
  • FIG. 4 is a flowchart presenting a detailed methodology of obtaining the mapping table between resolutions and bitrates during the video transcoding process.
  • FIG. 5 is a structural diagram presenting a system for downloading a VR video according to an embodiment of the present disclosure.
  • FIG. 1 is a diagram illustrating an example network of a VR playback system.
  • the VR playback system 10 includes a server 100 and a display device 120 which are coupled with each other through a network 110 , and a VR device.
  • the server 100 may be a stand-alone computer server or a server cluster.
  • the server 100 is used to store various video data and to store various applications that process these video data.
  • various daemons run on the server 100 in real time, so as to process various video data in the server 100 and to respond various requests from VR devices and the display device 120 .
  • the network 110 may be a selected one or selected ones from the group consisting of an internet, a local area network, an internet of things, and the like.
  • the display device 120 may be any of the computing devices, including a computer device having an independent display screen and a processing capability.
  • the display device 120 may be a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a palmtop computer, a personal digital assistant, a smart phone, an intelligent electrical apparatus, a game console, an iPad/iPhone, a video player, a DVD recorder/player, a television, or a home entertainment system.
  • the display device 120 may store VR player software as a VR player. When the VR player is started, it requests and downloads various video data from the server 100 , and renders and plays the video data in the display device.
  • the VR device 130 is a stand-alone head-mounted device that can interact with the display device 120 and the server 100 , to communicate the user's current information to the display device 120 and/or the server 100 through signaling.
  • the user's current information is, for example, a field of view of the user, a position modification of a helmet, a modification of the user's sight orientation, etc.
  • the display device 120 can flexibly process the currently played video data.
  • the display device 120 determines that a core viewing region for the user has been changed and starts to play video data with high resolution in the changed core viewing region.
  • the VR device 130 is a stand-alone head-mounted device.
  • the VR device 130 is not limited thereto, and the VR device 130 may also be an all-in-one head-mounted device.
  • the all-in-one head-mounted device itself has a display screen, so that it is not necessary to connect the all-in-one head-mounted device with the external display device.
  • the display device 120 may be eliminated.
  • the all-in-one head-mounted device is configured to obtain video data from the server 100 and to perform playback operation, and the all-in-one head-mounted device is also configured to detect user's current field of view modification information and to adjust the playback operation according to the field of view modification information.
  • FIG. 2 is a flowchart showing a method used in the VR playback system of FIG. 1 . The method comprises following steps.
  • step S 10 a source video data process is operated on server side.
  • step S 20 the display device obtains some related information of the user based on interactions with the VR device.
  • step S 30 the display device requests video data and receives video data from the server according to the related information.
  • step S 40 the display device side performs a rendering process based on the received video data.
  • the video data is downloaded. Furthermore, the present disclosure provides a method for downloading a VR video. The method downloads the VR video by predicting a network condition according to a calculated average download speed and providing a download link according to bitrates, in order to optimize VR video streaming transmission.
  • FIG. 3 is a flowchart presenting the method for downloading a VR video according to an embodiment of the present disclosure. Specifically, following steps are comprised.
  • step S 100 a play request for a first video is received.
  • the first video in this step refers to a specific video.
  • step S 200 an average download speed is calculated.
  • the average download speed is calculated by making statistics on an amount of downloaded data and download time.
  • the average download speed is calculated once in real time. For example, the average download speed is calculated once before data of the first video starts to be downloaded, or the average download speed is calculated once before each video segments of the first video starts to be downloaded.
  • step S 300 a plurality of first video segments are selected from a plurality of video segments of the first video according to the average download speed, wherein the plurality of video segments of the first video correspond to different resolutions and bitrates.
  • a complete source VR video can be processed into the plurality of video segments, for example, the plurality of video segments may be obtained by segmenting the source VR video in temporal domain and/or in spatial domain.
  • the plurality of video segments have different resolutions and bitrates.
  • the plurality of first video segments can be selected from the plurality of video segments with different resolutions and bitrates. Therefore, in this step, each of the plurality of first video segments is the video segment with a proper resolution and a proper bitrate selected from the plurality of video segments, according to the average download speed.
  • the video segments with high resolutions and high bitrates are selected as the plurality of first video segments
  • the video segments with low resolutions and low bitrates are selected as the plurality of first video segments.
  • step S 400 a download link is determined according to the plurality of first video segments.
  • the download link of the first video is determined according to a predetermined download strategy, and the plurality of first video segments waiting to be downloaded and a download order are specified by the download link.
  • the download order for downloading the first video segments, which need to be downloaded is an ascending order of the bitrates.
  • the present disclosure is not limited thereto. Those skilled in the art can determine the download strategy according to the bitrates and the average download speed, in order to optimize video transmission.
  • step S 500 the plurality of first video segments are downloaded according to the download link.
  • download processes are operated on the plurality of first video segments.
  • the video segments which need to be downloaded and are selected according to the average speed, are downloaded, so as to optimize video transmission.
  • the method for downloading the VR video further comprises: obtaining a mapping table between resolutions and bitrates, and determining the plurality of first video segments by looking up the mapping table.
  • the mapping table defines correspondences between a plurality of resolutions and a plurality of bitrates.
  • the mapping table is looked up to obtain the set bitrates of the plurality of video segments, so that the plurality of first video segments can be determined according to the average download speed and the set bitrates.
  • the mapping table is generated during a video transcoding process.
  • raw video data is processed into the plurality of video segments, for example, the raw video data can be processed into a basic video and an enhancement video which need to be overlaid together during playback.
  • the bitrate of each of the plurality of video segments are recorded and are summarized as the mapping table between resolutions and bitrates.
  • information provided by the play request comprises a field of view of the user
  • the plurality of first video segments have a plurality of resolutions
  • a resolution distribution of the plurality of first video segments is determined by the field of view of the user. For example, it is assumed that a video with 360-degree panoramic images is provided, the user's field of view ranges in 135 degrees, then the video segments with high resolutions are displayed in a screen region inside the user's field of view, and the video segments with low resolutions are displayed on the screen regions outside the user's field of view. Therefore, by decreasing times for downloading the high-resolution video segments, the amount of downloaded data can be reduced and the video transmission can be optimized.
  • the user's field of view can be obtained according to parameters of a gyroscope in the head-mounted device.
  • FIG. 4 is a flowchart presenting a detailed methodology of obtaining the mapping table between resolutions and bitrates during the video transcoding process.
  • step S 1001 a plurality of resolution levels are defined, and a plurality of resolutions in the plurality of resolution levels are calculated according to a predetermined target resolution.
  • step S 1002 bitrates corresponding to the resolutions in the plurality of resolution levels are calculated based on a target bitrate corresponding to the predetermined target resolution.
  • step S 1003 the video transcoding process is performed according to the corresponding resolutions and the corresponding bitrates, in order to obtain a plurality of video segments of video data with different resolutions and different bitrates.
  • N represents a total number of the plurality of resolution levels
  • R 0 represents the predetermined target resolution
  • bitrate 0 represents the target bitrate according to the target resolution.
  • the resolution R 1 in the 1 st resolution level, the resolution R 2 in the 2 nd resolution level, . . . , and the resolution R N ⁇ 1 in the (N ⁇ 1) th resolution level can be calculated according to a resolution ratio between R n and R n+1 , so as to obtain the plurality of resolutions in all of the plurality of resolution levels.
  • All of the bitrates can be obtained according to resolution ratios between bitrate n and bitrate n+1 , so that the mapping table between resolutions and bitrates is obtained.
  • the plurality of video segments can be obtained by transcoding based on the corresponding resolutions and the corresponding bitrates.
  • mapping table between resolutions and bitrates is not limited thereto.
  • the mapping table can be obtained by other manners, for example, the mapping table between resolutions and bitrates may be set by experience.
  • n one of the plurality of resolution level, wherein a lower n corresponding to a higher resolution, n ⁇ [0, 1, 2, . . . ] R n the resolution in the n th resolution level, a two-element array ( X n , Y n ) bitrate n the bitrate corresponding to the n th resolution level bitrate n base the bitrate of the basic video corresponding to the n th resolution level bitrate n enhance the bitrate of the enhancement video corresponding to the n th resolution level q a resolution ratio (which is, for example, 2n)
  • a maximum resolution is defined as R 0 , bitrate 0 , X 0 base , Y 0 base , X 0 enhance , Y 0 enhance , and q are given by user, and also
  • q X 0 enhance X 1 enhance > 1 ( 2 )
  • bitrate corresponding to the n th resolution level satisfy following formulas:
  • bitrate n For each one of bitrate n , bitrate n enhance , and bitrate n base there is:
  • bitrate n + 1 bitrate n q 2 ( 7 )
  • bitrate n base >bitrate n+1 base +bitrate n+1 enhance (8)
  • bitrate n base > 1 q 2 - 1 ⁇ bitrate n enhance ( 9 )
  • mapping table can be generated when X 0 base , Y 0 base , X 0 enhance , Y 0 enhance and q are provided by the user and satisfy an inequality:
  • FIG. 5 is a structural diagram presenting a system for downloading a VR video according to an embodiment of the present disclosure.
  • the system 500 for downloading a VR video comprises a request receiving module 501 , a speed calculation module 502 , a segment obtaining module 503 , a link obtaining module 504 and a download executing module 505 .
  • the request receiving module 501 is configured to receive a play request for a first video.
  • the play request is sent from a display device or a VR device.
  • Information provided by the play request includes a name of the current playing video and a current field of view of a user.
  • the first video here refers to a specific video.
  • the speed calculating module 502 is configured to calculate an average download speed.
  • the average download speed is calculated by making statistics on the amount of downloaded and download time.
  • the segment obtaining module 503 is configured to select a plurality of first video segments from a plurality of video segments of the first video according to the average download speed, wherein the plurality of video segments of the first video correspond to different resolutions and bitrates.
  • the plurality of video segments of the first video is obtained during a video transcoding process and have different resolutions and bitrates.
  • the segment obtaining module 503 is configured to obtain the video segments for downloading selected from the plurality of video segments for downloading according to the current average download speed.
  • a download strategy is set to select the video segments with low bitrates and low resolutions for downloading when the average download speed is low, and to select the video segments with high bitrates and high resolutions for downloading when the average download speed is relatively high.
  • the link obtaining module 504 is configured to determine a download link according to the plurality of first video segments. Specifically, the link obtaining module 504 is configured to determine a download manner and a download order according to the predetermined download strategy, for example, the link obtaining module 504 downloads the video segments with high resolutions first, and then downloads the video segments with low resolutions.
  • the download executing module 505 is configured to download the plurality of video segments according to the download link. That is, the download executing module performs a download operation according to the download link provided by the link obtaining module 505 .
  • the above-described system for downloading the VR video further comprises: a mapping table obtaining module which is configured to generate a mapping table between resolutions and bitrates during the process when the plurality of video segments are obtained according to the video data.
  • the mapping table obtaining module comprises: a first calculating unit which is configured to define a plurality of resolution levels, and calculate resolutions in the plurality of resolution levels according to a predetermined target resolution; and a second calculating unit which is configured to calculate bitrates corresponding to the resolutions in the plurality of resolution levels based on a target bitrate corresponding to the predetermined target resolution.
  • the play request comprises a field of view of a user
  • the plurality of first video segments have a plurality of resolutions
  • a resolution distribution of the plurality of first video segments is determined by the field of view of the user.
  • the video segments, which need to be downloaded are determined according to the average download speed, so that the amount of downloaded data can be decreased and the video transmission can be optimized. It should be pointed out that the video data needs to be continuously downloaded during VR video playback, therefore, the process for selecting the video segments with proper resolutions and bitrates for downloading can be continuously performed according to the current average download speed, that is, the video transmission optimization is achieved by dynamic selection processes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Computer Graphics (AREA)
  • Human Computer Interaction (AREA)
  • Geometry (AREA)
  • Environmental & Geological Engineering (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Controls And Circuits For Display Device (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Image Generation (AREA)
  • Processing Or Creating Images (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Image Processing (AREA)

Abstract

A method for downloading a VR video is disclosed. The method comprises: receiving a play request for a first video; calculating an average download speed; selecting a plurality of first video segments from a plurality of video segments of the first video according to the average download speed, wherein the plurality of video segments of the first video correspond to different resolutions and bitrates; determining a download link according to the plurality of first video segments; and downloading the plurality of first video segments according to the download link. The video segments, which need to be downloaded, are determined according to the average download speed, so that the amount of downloaded data can be decreased and the video transmission can be optimized. The present disclosure also provides a system for downloading a VR video.

Description

    CROSS-REFERENCE TO RELATED APPLICATION
  • This application claims the priority and benefit of U.S. provisional application 62/441,936, filed on Jan. 3, 2017, which is incorporated herein by reference in its entirety.
  • BACKGROUND OF THE DISCLOSURE Field of the Disclosure
  • The present disclosure relates to video processing technology, and more particularly, to a method and a system for downloading a VR video.
  • Background of the Disclosure
  • Virtual Reality (VR) is a computer simulation technology for creating and experiencing a virtual world. For example, a three-dimensional real-time image can be presented based on a technology which tracks a user's head, eyes or hands. In the network-based virtual reality technology, full-view video data can be pre-stored on a server, and then transmitted to a display device. A display device can be glasses, a head-mounted display, etc. A video is displayed on the display device in accordance with a field of view of the user.
  • However, a high-resolution video data occupies large transmission of bandwidth and requires high computing power from the display devices. Presenting high-resolution VR video on internet is difficult, precisely, the existing video streaming technology cannot fulfill the virtual reality. Therefore, in order to present VR video smoothly in real-time, it is desirable to further improve the existing video streaming technology to save bandwidth and reduce performance requirements for display devices.
  • How to improve transmission effect by network prediction without changing a total amount of transferred data is always a problem waiting to be solved by industry.
  • SUMMARY OF THE DISCLOSURE
  • In view of this, the present disclosure provides a method and a system for downloading a VR video for optimizing video transmission.
  • According to a first aspect of the present disclosure, there is provided a method for downloading a VR video, comprising: receiving a play request for a first video; calculating an average download speed; selecting a plurality of first video segments from a plurality of video segments of the first video according to the average download speed, wherein the plurality of video segments of the first video correspond to different resolutions and bitrates; determining a download link according to the plurality of first video segments; and downloading the plurality of video segments according to the download link.
  • Preferably, the method further comprises: obtaining a mapping table between resolutions and bitrates during a video transcoding process; and determining the plurality of first video segments by looking up the mapping table.
  • Preferably, the step of obtaining the mapping table between resolutions and bitrates during the video transcoding process comprises: defining a plurality of resolution levels, and calculating a plurality of resolutions in the plurality of resolution levels according to a predetermined target resolution; and calculating bitrates corresponding to the resolutions in the plurality of resolution levels based on a target bitrate corresponding to the predetermined target resolution.
  • Preferably, the step of defining a plurality of resolution levels, and calculating a plurality of resolutions in the plurality of resolution levels according to a predetermined target resolution, comprises: calculating the resolution in one of the plurality of resolution levels following formulas below,
  • k = X 0 base * Y 0 base X 0 enhance * Y 0 enhance q = X 0 enhance X 1 enhance > 1 R n + 1 = ( X n , q , Y n q )
  • wherein, n represents the one of the plurality of resolution levels, R0 represents a maximum value of the plurality of resolutions, {X0 base, Y0 base} represents the resolution of a basic video corresponding to the one of the plurality of the resolution levels, {X0 enhance, Y0 enhance} represents the resolution of an enhancement video corresponding to the one of the plurality of resolution levels, q represents a resolution ratio, Rn{Xn, Yn} represents the resolution in the one of the plurality of resolution levels; wherein the step of calculating bitrates corresponding to the plurality of resolutions in the plurality of resolution levels based on a target bitrate of the predetermined target resolution, comprises: calculating the bitrates corresponding to the resolutions in the plurality of resolution levels following formulas below,
  • bitrate n = bitrate n base + bitrate n enhance bitrate n base = k k + 1 bitrate n bitrate n enhance = 1 k + 1 bitrate n ,
  • wherein, for each one of bitraten, bitraten enhance and bitraten base, there is:
  • bitrate n + 1 = bitrate n q 2 ,
  • and an inequality is satisfied:
  • bitrate n base > 1 q 2 - 1 bitrate n enhance ,
  • wherein, bitrate0 represents a maximum value of the bitrates, bitraten, bitraten base, bitraten enhance respectively represent the bitrate, the bitrate of the basic video, and the bitrate of the enhancement video corresponding to the one of the plurality of resolution levels, wherein n is an integer equal to or larger than 1.
  • Preferably, the play request comprises a field of view of a user, the plurality of first video segments have a plurality of resolutions, and a resolution distribution of the plurality of first video segments is determined by the field of view of the user.
  • Preferably, the resolution of a first video segment within the field of view is higher than the resolution of a first video segment outside the field of view.
  • Preferably, the average download speed and the resolution distribution of the plurality of first video segments are positively correlated.
  • According to a second aspect of the present disclosure, there is provided a system for downloading a VR video, comprising: a request receiving module configured to receive a play request of a first video; a speed calculating module configured to calculate an average download speed; a segment obtaining module configured to select a plurality of first video segments from a plurality of video segments of the first video according to the average download speed, wherein the plurality of video segments of the first video correspond to different resolutions and bitrates; a link obtaining module configured to determine a download link according to the plurality of first video segments; and a download executing module configured to download the plurality of video segments according to the download link.
  • Preferably, the system further comprises: a mapping table obtaining module configured to obtaining a mapping table between resolutions and bitrates during a video transcoding process, and to determine the plurality of first video segments by looking up the mapping table.
  • Preferably, the mapping table obtaining module comprises: a first calculating unit configured to define a plurality of resolution levels and calculate a plurality of resolutions in the plurality of resolution levels according to the predetermined target resolution; and calculating bitrates corresponding to the resolutions in the plurality of resolution levels based on a target bitrate corresponding to the predetermined target resolution.
  • The present disclosure provides a plurality of video segments with proper resolutions and proper bitrates for downloading according to the average download speed. As the average download speed reflects current network conditions, a resolution reflects the image precision, a bitrate reflects bit number of data transmitted per unit time, the resolution, the bitrate and average download speed can be synthesized to optimize the video transmission by using an optimal strategy to select the appropriate video segments. Furthermore, the video segments with high resolutions and low resolutions are selected according to the user's field of view, which can decrease the amount of transferred data without affecting the user's demands for viewing the video.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • The above and other objects, advantages and features of the present disclosure will become more fully understood from the detailed description given hereinbelow in connection with the appended drawings, and wherein:
  • FIG. 1 is a diagram illustrating an example network of a VR playback system;
  • FIG. 2 is a flowchart showing a method used in the VR playback system of FIG. 1;
  • FIG. 3 is a flowchart presenting the method for downloading a VR video according to an embodiment of the present disclosure;
  • FIG. 4 is a flowchart presenting a detailed methodology of obtaining the mapping table between resolutions and bitrates during the video transcoding process; and
  • FIG. 5 is a structural diagram presenting a system for downloading a VR video according to an embodiment of the present disclosure.
  • DETAILED DESCRIPTION OF THE DISCLOSURE
  • Exemplary embodiments the present disclosure will be described in more details below with reference to the accompanying drawings. In the drawings, like reference numerals denote like members. The figures are not drawn to scale, for the sake of clarity. Moreover, some well-known parts may not be shown.
  • FIG. 1 is a diagram illustrating an example network of a VR playback system. The VR playback system 10 includes a server 100 and a display device 120 which are coupled with each other through a network 110, and a VR device. For example, the server 100 may be a stand-alone computer server or a server cluster. The server 100 is used to store various video data and to store various applications that process these video data. For example, various daemons run on the server 100 in real time, so as to process various video data in the server 100 and to respond various requests from VR devices and the display device 120. The network 110 may be a selected one or selected ones from the group consisting of an internet, a local area network, an internet of things, and the like. For example, the display device 120 may be any of the computing devices, including a computer device having an independent display screen and a processing capability. The display device 120 may be a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a palmtop computer, a personal digital assistant, a smart phone, an intelligent electrical apparatus, a game console, an iPad/iPhone, a video player, a DVD recorder/player, a television, or a home entertainment system. The display device 120 may store VR player software as a VR player. When the VR player is started, it requests and downloads various video data from the server 100, and renders and plays the video data in the display device. In this example, the VR device 130 is a stand-alone head-mounted device that can interact with the display device 120 and the server 100, to communicate the user's current information to the display device 120 and/or the server 100 through signaling. The user's current information is, for example, a field of view of the user, a position modification of a helmet, a modification of the user's sight orientation, etc. According to the information, the display device 120 can flexibly process the currently played video data. In some embodiments, when a position of user's helmet changes, the display device 120 determines that a core viewing region for the user has been changed and starts to play video data with high resolution in the changed core viewing region.
  • In the above embodiment, the VR device 130 is a stand-alone head-mounted device. However, those skilled in the art should understand that the VR device 130 is not limited thereto, and the VR device 130 may also be an all-in-one head-mounted device. The all-in-one head-mounted device itself has a display screen, so that it is not necessary to connect the all-in-one head-mounted device with the external display device. For example, in this example, if the all-in-one head-mounted device is used as the VR device, the display device 120 may be eliminated. At this point, the all-in-one head-mounted device is configured to obtain video data from the server 100 and to perform playback operation, and the all-in-one head-mounted device is also configured to detect user's current field of view modification information and to adjust the playback operation according to the field of view modification information.
  • FIG. 2 is a flowchart showing a method used in the VR playback system of FIG. 1. The method comprises following steps.
  • In step S10, a source video data process is operated on server side.
  • In step S20, the display device obtains some related information of the user based on interactions with the VR device.
  • In step S30, the display device requests video data and receives video data from the server according to the related information.
  • In step S40, the display device side performs a rendering process based on the received video data.
  • Wherein, in the step S30, the video data is downloaded. Furthermore, the present disclosure provides a method for downloading a VR video. The method downloads the VR video by predicting a network condition according to a calculated average download speed and providing a download link according to bitrates, in order to optimize VR video streaming transmission.
  • FIG. 3 is a flowchart presenting the method for downloading a VR video according to an embodiment of the present disclosure. Specifically, following steps are comprised.
  • In step S100, a play request for a first video is received.
  • The first video in this step refers to a specific video.
  • In step S200, an average download speed is calculated.
  • The average download speed is calculated by making statistics on an amount of downloaded data and download time. In general, the average download speed is calculated once in real time. For example, the average download speed is calculated once before data of the first video starts to be downloaded, or the average download speed is calculated once before each video segments of the first video starts to be downloaded.
  • In step S300, a plurality of first video segments are selected from a plurality of video segments of the first video according to the average download speed, wherein the plurality of video segments of the first video correspond to different resolutions and bitrates.
  • In the source video data process on the server side, a complete source VR video can be processed into the plurality of video segments, for example, the plurality of video segments may be obtained by segmenting the source VR video in temporal domain and/or in spatial domain. The plurality of video segments have different resolutions and bitrates. When the first video is requested, the plurality of first video segments can be selected from the plurality of video segments with different resolutions and bitrates. Therefore, in this step, each of the plurality of first video segments is the video segment with a proper resolution and a proper bitrate selected from the plurality of video segments, according to the average download speed. For example, when the average download speed is relatively high, the video segments with high resolutions and high bitrates are selected as the plurality of first video segments, and when the average download speed is relatively low, the video segments with low resolutions and low bitrates are selected as the plurality of first video segments.
  • In step S400, a download link is determined according to the plurality of first video segments.
  • In this step, the download link of the first video is determined according to a predetermined download strategy, and the plurality of first video segments waiting to be downloaded and a download order are specified by the download link. For example, the download order for downloading the first video segments, which need to be downloaded, is an ascending order of the bitrates. However, the present disclosure is not limited thereto. Those skilled in the art can determine the download strategy according to the bitrates and the average download speed, in order to optimize video transmission.
  • In step S500, the plurality of first video segments are downloaded according to the download link.
  • In this step, download processes are operated on the plurality of first video segments.
  • In this embodiment, the video segments, which need to be downloaded and are selected according to the average speed, are downloaded, so as to optimize video transmission.
  • In an optional embodiment, the method for downloading the VR video further comprises: obtaining a mapping table between resolutions and bitrates, and determining the plurality of first video segments by looking up the mapping table. The mapping table defines correspondences between a plurality of resolutions and a plurality of bitrates. The mapping table is looked up to obtain the set bitrates of the plurality of video segments, so that the plurality of first video segments can be determined according to the average download speed and the set bitrates. Further, the mapping table is generated during a video transcoding process. During the video transcoding process, raw video data is processed into the plurality of video segments, for example, the raw video data can be processed into a basic video and an enhancement video which need to be overlaid together during playback. In this process, the bitrate of each of the plurality of video segments are recorded and are summarized as the mapping table between resolutions and bitrates.
  • In a preferred embodiment, information provided by the play request comprises a field of view of the user, the plurality of first video segments have a plurality of resolutions, and a resolution distribution of the plurality of first video segments is determined by the field of view of the user. For example, it is assumed that a video with 360-degree panoramic images is provided, the user's field of view ranges in 135 degrees, then the video segments with high resolutions are displayed in a screen region inside the user's field of view, and the video segments with low resolutions are displayed on the screen regions outside the user's field of view. Therefore, by decreasing times for downloading the high-resolution video segments, the amount of downloaded data can be reduced and the video transmission can be optimized. The user's field of view can be obtained according to parameters of a gyroscope in the head-mounted device.
  • FIG. 4 is a flowchart presenting a detailed methodology of obtaining the mapping table between resolutions and bitrates during the video transcoding process.
  • In step S1001, a plurality of resolution levels are defined, and a plurality of resolutions in the plurality of resolution levels are calculated according to a predetermined target resolution.
  • In step S1002, bitrates corresponding to the resolutions in the plurality of resolution levels are calculated based on a target bitrate corresponding to the predetermined target resolution.
  • In step S1003, the video transcoding process is performed according to the corresponding resolutions and the corresponding bitrates, in order to obtain a plurality of video segments of video data with different resolutions and different bitrates.
  • This embodiment will be understood by a following example.
  • It is assumed that N represents a total number of the plurality of resolution levels, R0 represents the predetermined target resolution, and bitrate0 represents the target bitrate according to the target resolution. The resolution R1 in the 1st resolution level, the resolution R2 in the 2nd resolution level, . . . , and the resolution RN−1 in the (N−1)th resolution level, can be calculated according to a resolution ratio between Rn and Rn+1, so as to obtain the plurality of resolutions in all of the plurality of resolution levels. All of the bitrates can be obtained according to resolution ratios between bitraten and bitraten+1, so that the mapping table between resolutions and bitrates is obtained. Finally, during the video transcoding process, the plurality of video segments can be obtained by transcoding based on the corresponding resolutions and the corresponding bitrates.
  • Those skilled in the art should understand that the manner for obtaining the mapping table between resolutions and bitrates is not limited thereto. The mapping table can be obtained by other manners, for example, the mapping table between resolutions and bitrates may be set by experience.
  • In a following example, formulas about the resolutions and the bitrates and derivations of the formulas are provided. In this example, it is assumed that the video data is processed into multiple video data each have a corresponding resolution in each of the plurality of resolution levels, and each of the multiple video data having the corresponding resolution comprises a basic video and an enhancement video which are overlaid together during playback. Table 1 shows definitions of variables included in the formulas.
  • TABLE 1
    variate meanings
    n one of the plurality of resolution level,
    wherein a lower n corresponding to a higher resolution,
    n□[0, 1, 2, . . . ]
    Rn the resolution in the nth resolution level,
    a two-element array ( Xn, Yn)
    bitraten the bitrate corresponding to the nth
    resolution level
    bitraten base the bitrate of the basic video corresponding to the nth
    resolution level
    bitraten enhance the bitrate of the enhancement video corresponding to
    the nth resolution level
    q a resolution ratio (which is, for example, 2n)
  • In general, a maximum resolution is defined as R0, bitrate0, X0 base, Y0 base, X0 enhance, Y0 enhance, and q are given by user, and also
  • k = bitrate 0 base bitrate 0 enhance = σ X 0 base * Y 0 base σ X 0 enhance * Y 0 enhance = X 0 base * Y 0 base X 0 enhance * Y 0 enhance ( 1 ) q = X 0 enhance X 1 enhance > 1 ( 2 ) R n + 1 = ( X n + 1 , Y n + 1 ) = ( X n , q , Y n q ) ( 3 )
  • The bitrate corresponding to the nth resolution level satisfy following formulas:
  • bitrate n = bitrate n base + bitrate n enhance ( 4 ) bitrate n base = k k + 1 bitrate n ( 5 ) bitrate n enhance = 1 k + 1 bitrate n ( 6 )
  • For each one of bitraten, bitraten enhance, and bitraten base there is:
  • bitrate n + 1 = bitrate n q 2 ( 7 )
  • and an inequality is satisfied:

  • bitraten base>bitraten+1 base+bitraten+1 enhance  (8)
  • By substituting (7) into (8), there is:
  • bitrate n base > 1 q 2 - 1 bitrate n enhance ( 9 )
  • So that the mapping table can be generated when X0 base, Y0 base, X0 enhance, Y0 enhance and q are provided by the user and satisfy an inequality:
  • X 0 base * Y 0 base X 0 enhance * Y 0 enhance > 1 q 2 - 1
  • FIG. 5 is a structural diagram presenting a system for downloading a VR video according to an embodiment of the present disclosure. The system 500 for downloading a VR video comprises a request receiving module 501, a speed calculation module 502, a segment obtaining module 503, a link obtaining module 504 and a download executing module 505.
  • The request receiving module 501 is configured to receive a play request for a first video. The play request is sent from a display device or a VR device. Information provided by the play request includes a name of the current playing video and a current field of view of a user. The first video here refers to a specific video.
  • The speed calculating module 502 is configured to calculate an average download speed. The average download speed is calculated by making statistics on the amount of downloaded and download time.
  • The segment obtaining module 503 is configured to select a plurality of first video segments from a plurality of video segments of the first video according to the average download speed, wherein the plurality of video segments of the first video correspond to different resolutions and bitrates. The plurality of video segments of the first video is obtained during a video transcoding process and have different resolutions and bitrates. The segment obtaining module 503 is configured to obtain the video segments for downloading selected from the plurality of video segments for downloading according to the current average download speed. In general, a download strategy is set to select the video segments with low bitrates and low resolutions for downloading when the average download speed is low, and to select the video segments with high bitrates and high resolutions for downloading when the average download speed is relatively high.
  • The link obtaining module 504 is configured to determine a download link according to the plurality of first video segments. Specifically, the link obtaining module 504 is configured to determine a download manner and a download order according to the predetermined download strategy, for example, the link obtaining module 504 downloads the video segments with high resolutions first, and then downloads the video segments with low resolutions.
  • The download executing module 505 is configured to download the plurality of video segments according to the download link. That is, the download executing module performs a download operation according to the download link provided by the link obtaining module 505.
  • Preferably, the above-described system for downloading the VR video further comprises: a mapping table obtaining module which is configured to generate a mapping table between resolutions and bitrates during the process when the plurality of video segments are obtained according to the video data. The mapping table obtaining module comprises: a first calculating unit which is configured to define a plurality of resolution levels, and calculate resolutions in the plurality of resolution levels according to a predetermined target resolution; and a second calculating unit which is configured to calculate bitrates corresponding to the resolutions in the plurality of resolution levels based on a target bitrate corresponding to the predetermined target resolution.
  • In a preferred embodiment, the play request comprises a field of view of a user, the plurality of first video segments have a plurality of resolutions, and a resolution distribution of the plurality of first video segments is determined by the field of view of the user.
  • In the above-described embodiment, the video segments, which need to be downloaded, are determined according to the average download speed, so that the amount of downloaded data can be decreased and the video transmission can be optimized. It should be pointed out that the video data needs to be continuously downloaded during VR video playback, therefore, the process for selecting the video segments with proper resolutions and bitrates for downloading can be continuously performed according to the current average download speed, that is, the video transmission optimization is achieved by dynamic selection processes.
  • The embodiments were chosen and described in order to best explain the principles of the disclosure and its practical applications, to thereby enable others skilled in the art to best utilize the disclosure. The disclosure is intended to cover alternatives, modifications and equivalents that may be included within the spirit and scope of the disclosure as defined by the appended claims.
  • The foregoing descriptions of specific embodiments of the present disclosure have been presented, but are not intended to limit the disclosure to the precise forms disclosed. It will be readily apparent to one skilled in the art that many modifications and changes may be made in the present disclosure. Any modifications, equivalence, variations of the preferred embodiments can be made without departing from the doctrine and spirit of the present disclosure.

Claims (10)

1. A method for downloading a VR video, comprising:
receiving a play request for a first video;
calculating an average download speed;
selecting a plurality of first video segments from a plurality of video segments of the first video according to the average download speed, wherein the plurality of video segments of the first video correspond to different resolutions and bitrates;
determining a download link according to the plurality of first video segments; and
downloading the plurality of first video segments according to the download link.
2. The method according to claim 1, further comprising: obtaining a mapping table between resolutions and bitrates during a video transcoding process; and determining the plurality of first video segments by looking up the mapping table.
3. The method according to claim 2, wherein the step of obtaining the mapping table between resolutions and bitrates during the video transcoding process comprises:
defining a plurality of resolution levels, and calculating a plurality of resolutions in the plurality of resolution levels according to a predetermined target resolution; and
calculating bitrates corresponding to the resolutions in the plurality of resolution levels based on a target bitrate corresponding to the predetermined target resolution.
4. The method according to claim 3, wherein the step of defining the plurality of resolution levels and calculating the plurality of resolutions in the plurality of resolution levels according to the predetermined target resolution comprises:
calculating the resolution in one of the plurality of resolution levels following formulas below,
k = X 0 base * Y 0 base X 0 enhance * Y 0 enhance q = X 0 enhance X 1 enhance > 1 R n + 1 = ( X n , q , Y n q )
wherein, n represents the one of the plurality of resolution levels, R0 represents a maximum value of the plurality of resolutions, {X0 base, Y0 base} represents the resolution of a basic video corresponding to the one of the plurality of the resolution levels, {X0 enhance, Y0 enhance} represents the resolution of an enhancement video corresponding to the one of the plurality of resolution levels, q represents a resolution ratio, Rn{Xn, Yn} represents the resolution in the one of the plurality of resolution levels; wherein the step of calculating bitrates corresponding to the plurality of resolutions in the plurality of resolution levels based on a target bitrate of the predetermined target resolution, comprises:
calculating the bitrates corresponding to the resolutions in the plurality of resolution levels following formulas below,
bitrate n = bitrate n base + bitrate n enhance bitrate n base = k k + 1 bitrate n bitrate n enhance = 1 k + 1 bitrate n ,
wherein for each one of bitraten, bitraten enhance and bitraten base, there is:
bitrate n + 1 = bitrate n q 2 ,
 and an inequality is satisfied:
bitrate n base > 1 q 2 - 1 bitrate n enhance ,
wherein, bitrate0 represents a maximum value of the bitrates, bitraten, bitraten base bitraten enhance respectively represent the bitrate, the bitrate of the basic video, and the bitrate of the enhancement video corresponding to the one of the plurality of resolution levels, wherein n is an integer equal to or larger than 1.
5. The method according to claim 1, wherein the play request comprises a field of view of a user, the plurality of first video segments have a plurality of resolutions, and a resolution distribution of the plurality of first video segments is determined by the field of view of the user.
6. The method according to claim 5, wherein the resolution of a first video segment within the field of view is higher than the resolution of a first video segment outside the field of view.
7. The method according to claim 1, wherein the average download speed and the resolution distribution of the plurality of first video segments are positively correlated.
8. A system for downloading a VR video, comprising:
a request receiving module configured to receive a play request of a first video;
a speed calculating module configured to calculate an average download speed;
a segment obtaining module configured to select a plurality of first video segments from a plurality of video segments of the first video according to the average download speed, wherein the plurality of video segments of the first video correspond to different resolutions and bitrates;
a link obtaining module configured to determine a download link according to the plurality of first video segments; and
a download executing module configured to download the plurality of video segments according to the download link.
9. The system according to claim 8, further comprising: a mapping table obtaining module configured to obtaining a mapping table between resolutions and bitrates during a video transcoding process, and to determine the plurality of first video segments by looking up the mapping table.
10. The system according to claim 9, wherein the mapping table obtaining module comprises:
a first calculating unit configured to define a plurality of resolution levels and calculate a plurality of resolutions in the plurality of resolution levels according to the predetermined target resolution; and
a second calculating unit configured to calculate bitrates corresponding to the plurality of resolutions in the plurality of resolution levels, based on a target bitrate corresponding to the predetermined target resolution.
US15/860,430 2017-01-03 2018-01-02 Method and System for Downloading Multiple Resolutions Bitrate for Virtual Reality (VR) Video Streaming Optimization Abandoned US20180191868A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US15/860,430 US20180191868A1 (en) 2017-01-03 2018-01-02 Method and System for Downloading Multiple Resolutions Bitrate for Virtual Reality (VR) Video Streaming Optimization

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201762441936P 2017-01-03 2017-01-03
US15/860,430 US20180191868A1 (en) 2017-01-03 2018-01-02 Method and System for Downloading Multiple Resolutions Bitrate for Virtual Reality (VR) Video Streaming Optimization

Publications (1)

Publication Number Publication Date
US20180191868A1 true US20180191868A1 (en) 2018-07-05

Family

ID=62711388

Family Applications (6)

Application Number Title Priority Date Filing Date
US15/860,449 Expired - Fee Related US10334238B2 (en) 2017-01-03 2018-01-02 Method and system for real-time rendering displaying high resolution virtual reality (VR) video
US15/860,494 Abandoned US20180189980A1 (en) 2017-01-03 2018-01-02 Method and System for Providing Virtual Reality (VR) Video Transcoding and Broadcasting
US15/860,392 Abandoned US20180192044A1 (en) 2017-01-03 2018-01-02 Method and System for Providing A Viewport Division Scheme for Virtual Reality (VR) Video Streaming
US15/860,471 Abandoned US20180192022A1 (en) 2017-01-03 2018-01-02 Method and System for Real-time Rendering Displaying Virtual Reality (VR) On Mobile Using Head-Up Display Devices
US15/860,430 Abandoned US20180191868A1 (en) 2017-01-03 2018-01-02 Method and System for Downloading Multiple Resolutions Bitrate for Virtual Reality (VR) Video Streaming Optimization
US15/860,358 Abandoned US20180192063A1 (en) 2017-01-03 2018-01-02 Method and System for Virtual Reality (VR) Video Transcode By Extracting Residual From Different Resolutions

Family Applications Before (4)

Application Number Title Priority Date Filing Date
US15/860,449 Expired - Fee Related US10334238B2 (en) 2017-01-03 2018-01-02 Method and system for real-time rendering displaying high resolution virtual reality (VR) video
US15/860,494 Abandoned US20180189980A1 (en) 2017-01-03 2018-01-02 Method and System for Providing Virtual Reality (VR) Video Transcoding and Broadcasting
US15/860,392 Abandoned US20180192044A1 (en) 2017-01-03 2018-01-02 Method and System for Providing A Viewport Division Scheme for Virtual Reality (VR) Video Streaming
US15/860,471 Abandoned US20180192022A1 (en) 2017-01-03 2018-01-02 Method and System for Real-time Rendering Displaying Virtual Reality (VR) On Mobile Using Head-Up Display Devices

Family Applications After (1)

Application Number Title Priority Date Filing Date
US15/860,358 Abandoned US20180192063A1 (en) 2017-01-03 2018-01-02 Method and System for Virtual Reality (VR) Video Transcode By Extracting Residual From Different Resolutions

Country Status (2)

Country Link
US (6) US10334238B2 (en)
CN (6) CN108366293A (en)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109343518A (en) * 2018-09-03 2019-02-15 浙江大丰实业股份有限公司 Universal ride scene drives platform
US10616621B2 (en) * 2018-06-29 2020-04-07 At&T Intellectual Property I, L.P. Methods and devices for determining multipath routing for panoramic video content
US10812774B2 (en) 2018-06-06 2020-10-20 At&T Intellectual Property I, L.P. Methods and devices for adapting the rate of video content streaming
CN112468806A (en) * 2020-11-12 2021-03-09 中山大学 Panoramic video transmission optimization method for cloud VR platform
US11019361B2 (en) 2018-08-13 2021-05-25 At&T Intellectual Property I, L.P. Methods, systems and devices for adjusting panoramic view of a camera for capturing video content
US20210329255A1 (en) * 2018-10-22 2021-10-21 Bitmovin, Inc. Video Encoding Based on Customized Bitrate Table
US11190820B2 (en) 2018-06-01 2021-11-30 At&T Intellectual Property I, L.P. Field of view prediction in live panoramic video streaming
CN114979799A (en) * 2022-05-20 2022-08-30 北京字节跳动网络技术有限公司 Panoramic video processing method, device, equipment and storage medium

Families Citing this family (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10291910B2 (en) * 2016-02-12 2019-05-14 Gopro, Inc. Systems and methods for spatially adaptive video encoding
US10331862B2 (en) * 2017-04-20 2019-06-25 Cisco Technology, Inc. Viewport decryption
CN115842907A (en) * 2018-03-27 2023-03-24 京东方科技集团股份有限公司 Rendering method, computer product and display device
US11232532B2 (en) * 2018-05-30 2022-01-25 Sony Interactive Entertainment LLC Multi-server cloud virtual reality (VR) streaming
CN109375369B (en) * 2018-11-23 2021-05-18 国网天津市电力公司 Distortion preprocessing method in VR (virtual reality) large-screen cinema mode
CN111510777B (en) 2019-01-30 2021-11-23 上海哔哩哔哩科技有限公司 Method and device for measuring network speed, computer equipment and readable storage medium
CN111669666A (en) * 2019-03-08 2020-09-15 北京京东尚科信息技术有限公司 Method, device and system for simulating reality
CN111866485A (en) * 2019-04-25 2020-10-30 中国移动通信有限公司研究院 Stereoscopic picture projection and transmission method, device and computer readable storage medium
CN110336994B (en) * 2019-07-04 2021-06-22 上海索倍信息科技有限公司 Naked eye 3D display system
CN110381331A (en) * 2019-07-23 2019-10-25 深圳市道通智能航空技术有限公司 A kind of image processing method, device, equipment of taking photo by plane and storage medium
CN110490962B (en) * 2019-08-20 2023-09-15 武汉邦拓信息科技有限公司 Remote rendering method based on video stream
CN110544425A (en) * 2019-09-13 2019-12-06 广州城市职业学院 ancient building VR display system
EP4032311A1 (en) 2019-09-19 2022-07-27 InterDigital CE Patent Holdings Devices and methods for generating and rendering immersive video
CN111489428B (en) * 2020-04-20 2023-06-30 北京字节跳动网络技术有限公司 Image generation method, device, electronic equipment and computer readable storage medium
US11245911B1 (en) * 2020-05-12 2022-02-08 Whirlwind 3D, LLC Video encoder/decoder (codec) for real-time applications and size/b and width reduction
CN111754614B (en) * 2020-06-30 2024-07-02 平安国际智慧城市科技股份有限公司 VR-based video rendering method and device, electronic equipment and storage medium
CN114286142B (en) * 2021-01-18 2023-03-28 海信视像科技股份有限公司 Virtual reality equipment and VR scene screen capturing method
CN113347402A (en) * 2021-06-28 2021-09-03 筑友建筑装饰装修工程有限公司 Improved method, device and storage medium for rendering immersive content based on Unity
US11812146B2 (en) * 2021-12-02 2023-11-07 Fotonation Limited Method and system for camera motion blur reduction
CN114466220A (en) * 2022-01-29 2022-05-10 维沃移动通信有限公司 Video downloading method and electronic equipment
CN115002519A (en) * 2022-05-31 2022-09-02 北京势也网络技术有限公司 Method for playing 8K panoramic video file in low-bandwidth network
CN115396731A (en) * 2022-08-10 2022-11-25 北京势也网络技术有限公司 Panoramic video playing method and device, electronic equipment and readable storage medium
CN116880723B (en) * 2023-09-08 2023-11-17 江西格如灵科技股份有限公司 3D scene display method and system

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120254455A1 (en) * 2011-03-30 2012-10-04 Verizon Patent And Licensing Inc. Downloading video using excess bandwidth
US8897370B1 (en) * 2009-11-30 2014-11-25 Google Inc. Bitrate video transcoding based on video coding complexity estimation
US20150208103A1 (en) * 2012-08-08 2015-07-23 National University Of Singapore System and Method for Enabling User Control of Live Video Stream(s)
US20160191585A1 (en) * 2014-12-24 2016-06-30 Intel Corporation Link-aware streaming adaptation

Family Cites Families (36)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3196889B2 (en) * 1996-09-05 2001-08-06 株式会社アルファ Three-dimensional image processing method and computer-readable recording medium storing a program for causing a computer to execute the three-dimensional image processing method
TWI262725B (en) * 2005-06-30 2006-09-21 Cheertek Inc Video decoding apparatus and digital audio and video display system capable of controlling presentation of subtitles and method thereof
JP5891426B2 (en) * 2011-03-31 2016-03-23 パナソニックIpマネジメント株式会社 An image drawing apparatus, an image drawing method, and an image drawing program for drawing an all-around stereoscopic image
US8810598B2 (en) * 2011-04-08 2014-08-19 Nant Holdings Ip, Llc Interference based augmented reality hosting platforms
GB2501929B (en) * 2012-05-11 2015-06-24 Sony Comp Entertainment Europe Apparatus and method for augmented reality
US9129429B2 (en) * 2012-10-24 2015-09-08 Exelis, Inc. Augmented reality on wireless mobile devices
GB2509953B (en) * 2013-01-18 2015-05-20 Canon Kk Method of displaying a region of interest in a video stream
US9196199B2 (en) * 2013-02-12 2015-11-24 Pixtronix, Inc. Display having staggered display element arrangement
CN103702139B (en) * 2013-12-13 2017-02-01 华中科技大学 Video-on-demand system based on scalable coding under mobile environment
US9398250B2 (en) * 2014-01-06 2016-07-19 Arun Sobti & Associates, Llc System and apparatus for smart devices based conferencing
CN105025351B (en) * 2014-04-30 2018-06-29 深圳Tcl新技术有限公司 The method and device of DST PLAYER buffering
JP6337614B2 (en) * 2014-05-23 2018-06-06 セイコーエプソン株式会社 Control device, robot, and control method
US9918136B2 (en) * 2014-05-29 2018-03-13 Nextvr Inc. Methods and apparatus for delivering content and/or playing back content
CN104240281A (en) * 2014-08-28 2014-12-24 东华大学 Virtual reality head-mounted device based on Unity3D engine
CN104268922B (en) * 2014-09-03 2017-06-06 广州博冠信息科技有限公司 A kind of image rendering method and image rendering device
CN104616243B (en) * 2015-01-20 2018-02-27 北京道和汇通科技发展有限公司 A kind of efficient GPU 3 D videos fusion method for drafting
US20160261908A1 (en) * 2015-03-05 2016-09-08 Htc Corporation Media streaming system and control method thereof
CN104735464A (en) * 2015-03-31 2015-06-24 华为技术有限公司 Panorama video interactive transmission method, server and client end
CN104717507A (en) * 2015-03-31 2015-06-17 北京奇艺世纪科技有限公司 Video transcoding method and device
US10083363B2 (en) * 2015-05-26 2018-09-25 Nbcuniversal Media, Llc System and method for customizing content for a user
WO2016201423A1 (en) * 2015-06-12 2016-12-15 Google Inc. Electronic display stabilization for head mounted display
US9956054B2 (en) * 2015-06-25 2018-05-01 EchoPixel, Inc. Dynamic minimally invasive surgical-aware assistant
WO2017060423A1 (en) * 2015-10-08 2017-04-13 Koninklijke Kpn N.V. Enhancing a region of interest in video frames of a video stream
CN106919248A (en) * 2015-12-26 2017-07-04 华为技术有限公司 It is applied to the content transmission method and equipment of virtual reality
CN105916022A (en) * 2015-12-28 2016-08-31 乐视致新电子科技(天津)有限公司 Video image processing method and apparatus based on virtual reality technology
CN105455285B (en) * 2015-12-31 2019-02-12 北京小鸟看看科技有限公司 A kind of virtual implementing helmet adaptation method
US10412130B2 (en) * 2016-04-04 2019-09-10 Hanwha Techwin Co., Ltd. Method and apparatus for playing media stream on web browser
US10313417B2 (en) * 2016-04-18 2019-06-04 Qualcomm Incorporated Methods and systems for auto-zoom based adaptive video streaming
CN105898565A (en) * 2016-04-28 2016-08-24 乐视控股(北京)有限公司 Video processing method and device
US9721393B1 (en) * 2016-04-29 2017-08-01 Immersive Enterprises, LLC Method for processing and delivering virtual reality content to a user
CN106060570B (en) * 2016-06-30 2019-06-14 北京奇艺世纪科技有限公司 A kind of full-view video image plays, coding method and device
CN106060515B (en) * 2016-07-14 2018-11-06 腾讯科技(深圳)有限公司 Panorama pushing method for media files and device
CN106231317A (en) * 2016-09-29 2016-12-14 三星电子(中国)研发中心 Video processing, coding/decoding method and device, VR terminal, audio/video player system
US10595069B2 (en) * 2016-12-05 2020-03-17 Adobe Inc. Prioritizing tile-based virtual reality video streaming using adaptive rate allocation
US20180295375A1 (en) * 2017-04-05 2018-10-11 Lyrical Labs Video Compression Technology, LLC Video processing and encoding
CN107087212B (en) * 2017-05-09 2019-10-29 杭州码全信息科技有限公司 Interactive panoramic video transcoding and playback method and system based on spatial scalable coding

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8897370B1 (en) * 2009-11-30 2014-11-25 Google Inc. Bitrate video transcoding based on video coding complexity estimation
US20120254455A1 (en) * 2011-03-30 2012-10-04 Verizon Patent And Licensing Inc. Downloading video using excess bandwidth
US20150208103A1 (en) * 2012-08-08 2015-07-23 National University Of Singapore System and Method for Enabling User Control of Live Video Stream(s)
US20160191585A1 (en) * 2014-12-24 2016-06-30 Intel Corporation Link-aware streaming adaptation

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11190820B2 (en) 2018-06-01 2021-11-30 At&T Intellectual Property I, L.P. Field of view prediction in live panoramic video streaming
US11641499B2 (en) 2018-06-01 2023-05-02 At&T Intellectual Property I, L.P. Field of view prediction in live panoramic video streaming
US10812774B2 (en) 2018-06-06 2020-10-20 At&T Intellectual Property I, L.P. Methods and devices for adapting the rate of video content streaming
US10616621B2 (en) * 2018-06-29 2020-04-07 At&T Intellectual Property I, L.P. Methods and devices for determining multipath routing for panoramic video content
US11019361B2 (en) 2018-08-13 2021-05-25 At&T Intellectual Property I, L.P. Methods, systems and devices for adjusting panoramic view of a camera for capturing video content
US11671623B2 (en) 2018-08-13 2023-06-06 At&T Intellectual Property I, L.P. Methods, systems and devices for adjusting panoramic view of a camera for capturing video content
CN109343518A (en) * 2018-09-03 2019-02-15 浙江大丰实业股份有限公司 Universal ride scene drives platform
US20210329255A1 (en) * 2018-10-22 2021-10-21 Bitmovin, Inc. Video Encoding Based on Customized Bitrate Table
US11563951B2 (en) * 2018-10-22 2023-01-24 Bitmovin, Inc. Video encoding based on customized bitrate table
CN112468806A (en) * 2020-11-12 2021-03-09 中山大学 Panoramic video transmission optimization method for cloud VR platform
CN114979799A (en) * 2022-05-20 2022-08-30 北京字节跳动网络技术有限公司 Panoramic video processing method, device, equipment and storage medium

Also Published As

Publication number Publication date
CN108366293A (en) 2018-08-03
US10334238B2 (en) 2019-06-25
CN108366272A (en) 2018-08-03
US20180192063A1 (en) 2018-07-05
US20180189980A1 (en) 2018-07-05
CN108377381A (en) 2018-08-07
CN108391103A (en) 2018-08-10
US20180192044A1 (en) 2018-07-05
US20180192022A1 (en) 2018-07-05
US20180192026A1 (en) 2018-07-05
CN108419142A (en) 2018-08-17
CN108419093A (en) 2018-08-17

Similar Documents

Publication Publication Date Title
US20180191868A1 (en) Method and System for Downloading Multiple Resolutions Bitrate for Virtual Reality (VR) Video Streaming Optimization
US11303881B2 (en) Method and client for playing back panoramic video
WO2017193576A1 (en) Video resolution adaptation method and apparatus, and virtual reality terminal
JP6310073B2 (en) Drawing system, control method, and storage medium
CN110996097B (en) VR multimedia experience quality determination method and device
US20050271361A1 (en) Image frame processing method and device for displaying moving images to a variety of displays
US11122245B2 (en) Display apparatus, method for controlling the same and image providing apparatus
CN105430471A (en) Method and device for displaying live commenting in video
US20220382053A1 (en) Image processing method and apparatus for head-mounted display device as well as electronic device
CN111665940B (en) Cursor control method and device and related equipment
CN112511849A (en) Game display method, device, equipment, system and storage medium
US7936936B2 (en) Method of visualizing a large still picture on a small-size display
US20120293638A1 (en) Apparatus and method for providing 3d content
JP7550306B2 (en) Automatic Head-Up Display Positioning Based on Eye Tracking
KR20230104602A (en) Dynamic user device upscaling of media streams
US20240098316A1 (en) Video encoding method and apparatus, real-time communication method and apparatus, device, and storage medium
CN106454417A (en) Streaming media speed multiplication playing method and apparatus
US20200311947A1 (en) Image processing apparatus, transmission method, and storage medium
KR20160015136A (en) System for cloud streaming service, method of cloud streaming service using still image compression technique and apparatus for the same
CN115393490A (en) Image rendering method and device, storage medium and electronic equipment
JP6623905B2 (en) Server device, information processing method and program
US8482574B2 (en) System, method, and computer program product for calculating statistics associated with a surface to be rendered utilizing a graphics processor
EP3635958B1 (en) Image processing device and system
WO2024126043A1 (en) Mechanism to control the refresh rate of the real-environment computation for augmented reality (ar) experiences
TWI829350B (en) Method and system for enhancing quality of a frame sequence

Legal Events

Date Code Title Description
AS Assignment

Owner name: BLACK SAILS TECHNOLOGY INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:WANG, ZHUO;TANG, YONGTAO;ZHAO, RUOXI;AND OTHERS;REEL/FRAME:044518/0775

Effective date: 20180102

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION