US20180192044A1 - Method and System for Providing A Viewport Division Scheme for Virtual Reality (VR) Video Streaming - Google Patents

Method and System for Providing A Viewport Division Scheme for Virtual Reality (VR) Video Streaming Download PDF

Info

Publication number
US20180192044A1
US20180192044A1 US15/860,392 US201815860392A US2018192044A1 US 20180192044 A1 US20180192044 A1 US 20180192044A1 US 201815860392 A US201815860392 A US 201815860392A US 2018192044 A1 US2018192044 A1 US 2018192044A1
Authority
US
United States
Prior art keywords
viewport
block
blocks
core
area
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US15/860,392
Inventor
Zhuo Wang
Yongtao Tang
Ruoxi Zhao
Haoyan Zu
Chia-Chi Chang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Black Sails Technology Inc
Original Assignee
Black Sails Technology Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Black Sails Technology Inc filed Critical Black Sails Technology Inc
Priority to US15/860,392 priority Critical patent/US20180192044A1/en
Assigned to Black Sails Technology Inc. reassignment Black Sails Technology Inc. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: CHANG, CHIA-CHI, TANG, YONGTAO, WANG, ZHUO, ZHAO, RUOXI, ZU, HAOYAN
Publication of US20180192044A1 publication Critical patent/US20180192044A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • H04N13/0497
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/398Synchronisation thereof; Control thereof
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00Details of colour television systems
    • H04N9/12Picture reproducers
    • H04N9/31Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H04N9/3179Video signal processing therefor
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012Head tracking input arrangements
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/003D [Three Dimensional] image rendering
    • G06T15/04Texture mapping
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/003D [Three Dimensional] image rendering
    • G06T15/10Geometric effects
    • G06T15/20Perspective computation
    • G06T15/205Image-based rendering
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00Three dimensional [3D] modelling, e.g. data description of 3D objects
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00Image coding
    • G06T9/001Model-based coding, e.g. wire frame
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00Arrangements for monitoring or testing data switching networks
    • H04L43/08Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H04L43/0876Network utilisation, e.g. volume of load or congestion level
    • H04L43/0888Throughput
    • H04L65/601
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60Network streaming of media packets
    • H04L65/75Media network packet handling
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60Network streaming of media packets
    • H04L65/75Media network packet handling
    • H04L65/752Media network packet handling adapting media to network capabilities
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/131Protocols for games, networked simulations or virtual reality
    • H04N13/0429
    • H04N13/0479
    • H04N13/0484
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106Processing image signals
    • H04N13/111Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
    • H04N13/117Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106Processing image signals
    • H04N13/122Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106Processing image signals
    • H04N13/139Format conversion, e.g. of frame-rate or size
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106Processing image signals
    • H04N13/161Encoding, multiplexing or demultiplexing different image signal components
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/189Recording image signals; Reproducing recorded image signals
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/275Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/332Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/332Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H04N13/344Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/366Image reproducers using viewer tracking
    • H04N13/378Image reproducers using viewer tracking for tracking rotational head movements around an axis perpendicular to the screen
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/366Image reproducers using viewer tracking
    • H04N13/383Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/40Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/44Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/70Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/231Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/233Processing of audio elementary streams
    • H04N21/2335Processing of audio elementary streams involving reformatting operations of audio signals, e.g. by converting from one coding standard to another
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234345Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234363Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the spatial resolution, e.g. for clients with a lower screen resolution
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234381Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/238Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams
    • H04N21/2387Stream processing in response to a playback request from an end-user, e.g. for trick-play
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/239Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests
    • H04N21/2393Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests involving handling client requests
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/24Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
    • H04N21/2405Monitoring of the internal components or processes of the server, e.g. server load
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/266Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
    • H04N21/2662Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41Structure of client; Structure of client peripherals
    • H04N21/4104Peripherals receiving signals from specially adapted client devices
    • H04N21/4122Peripherals receiving signals from specially adapted client devices additional display device, e.g. video projector
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44012Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H04N21/440236Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81Monomedia components thereof
    • H04N21/816Monomedia components thereof involving special video data, e.g 3D video
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/845Structuring of content, e.g. decomposing content into time segments
    • H04N21/8456Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/76Television signal recording
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00Details of colour television systems
    • H04N9/12Picture reproducers
    • H04N9/31Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00Details of colour television systems
    • H04N9/12Picture reproducers
    • H04N9/31Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H04N9/3179Video signal processing therefor
    • H04N9/3185Geometric adjustment, e.g. keystone or convergence
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00Details of colour television systems
    • H04N9/79Processing of colour television signals in connection with recording
    • H04N9/87Regeneration of colour television signals
    • H04N9/8715Regeneration of colour television signals involving the mixing of the reproduced video signal with a non-recorded signal, e.g. a text signal
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2215/00Indexing scheme for image rendering
    • G06T2215/16Using real world measurements to influence rendering
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformations in the plane of the image
    • G06T3/40Scaling of whole images or parts thereof, e.g. expanding or contracting

Definitions

  • the present disclosure relates to video processing technology, and more particularly, to a method and a system for providing a viewport division scheme for virtual reality (VR) video streaming BACKGROUND OF THE DISCLOSURE
  • VR virtual reality
  • Virtual Reality is a computer simulation technology for creating and experiencing a virtual world. For example, a three-dimensional real-time image can be presented based on a technology which tracks a user's head, eyes or hands.
  • full-view video data can be pre-stored on a server, and then transmitted to a display device.
  • a display device can be glasses, a head-mounted display, etc.
  • a video is displayed on the display device according to a user's field of view.
  • a high-resolution video data occupies large transmission of bandwidth and requires high computing power from the display devices.
  • Presenting high-resolution VR video on internet is difficult.
  • the existing video streaming technology can not fulfill the virtual reality. Therefore, in order to present VR video smoothly in real-time, it is desirable to further improve the existing video streaming technology to save bandwidth and reduce performance requirements for display devices, by a new way to encode and store the VR video data on the server.
  • the present disclosure provides a method and a system for providing a viewport division scheme for virtual reality (VR) video streaming, which determine a viewport block of the VR video data and process the VR video data by different methods for a viewport area and a non-viewport area so as to have a better viewing experience at a player end through less data transmission.
  • VR virtual reality
  • a method for providing a viewport division scheme for VR video streaming comprising:
  • the step of obtaining a projection area of VR video comprises:
  • the step of dividing the projection area into a plurality of grid blocks comprises:
  • the method further comprises creating identifications for the plurality of grid blocks, the plurality of core blocks and the viewport, and establishing a mapping relationship between the viewport and the plurality of grid blocks by the identifications.
  • the step of creating identifications for the plurality of grid blocks, the plurality of core blocks and the viewport, and establishing a mapping relationship between the viewport and the plurality of grid blocks by the identifications comprises:
  • width block height video n 2 ( 2 )
  • height video represents an image height
  • N + represents a positive integer
  • id core_block ⁇ id block ⁇ ⁇ % ⁇ ⁇ 2 ⁇ n 2 , id block ⁇ Set core null , others ; ( 4 )
  • a viewport block in the core area to have a width determined by a user's field of view and a height equal to that of the core area, where a width width viewport _ block of the viewport block satisfies Equation (5),
  • width viewport_block fov * height video ⁇ , ( 5 )
  • fov represents the user's field of view
  • the identifications of the core blocks in the viewport block can be obtained according to Equation (7),
  • f ⁇ ( id viewport_block ) ⁇ ⁇ id viewport_block , id viewport_block + 1 , ... ⁇ , id viewport_block + i - 2 , id viewport_block + i - 1 ⁇ ; id viewport_block ⁇ [ 0 , 2 ⁇ n 2 - i ] ⁇ id viewport_block , id viewport_block + 1 , ... ⁇ , 2 ⁇ n 2 - 1 , 0 , 1 , ... ⁇ , i - ( 2 ⁇ n 2 - id viewport_block ) - 1 ⁇ ; , id viewport_block ⁇ [ 2 ⁇ n 2 - i + 1 , 2 ⁇ n 2 - 1 ] ; ⁇ ( 7 ) ( 7 )
  • the identification of the viewport block including the core block can be obtained uniquely according to Equation (8),
  • id core _ block _ 1st represents the identification of the first core block that is included in the viewport block.
  • the step of dividing the core area into the plurality of core blocks comprises:
  • the plurality core blocks each have a height equal to of the central and the core area has an area equal to integral times of the area of a grid block.
  • the method further comprises dividing the VR video data into a plurality of sub-data according to the plurality of grid blocks, wherein each sub-data corresponds to one of the grid blocks, and determining a mapping relationship between the plurality of sub-data and the viewport according to the mapping relationship between the plurality of grid blocks and the viewport.
  • the method further comprises creating a plurality of viewport blocks according to a user's field of view.
  • a system for providing a viewport for VR video streaming comprising:
  • a configuration module configured to obtain a projection area of VR video
  • a first division module configured to divide the projection area into a plurality of grid blocks
  • a second division module configured to define a core area in the projection area
  • a third division module configured to divide the core area into a plurality of core blocks
  • a combination module configured to combine the core blocks to provide a plurality of viewport blocks
  • mapping module configured to establish a mapping relationship between the viewport and the plurality of grid blocks, wherein the plurality of viewport blocks are related to a user's field of view.
  • the configuration module prefferably, the configuration module
  • the VR source video data is divided into a plurality of grid blocks, and a mapping table between the viewport blocks and the grid blocks is established according to a specific field of view.
  • the viewport block is retrieved according to the mapping table, and the video data of a viewport area and a non-viewport area are processed by different methods to have a better view experience.
  • FIG. 1 is a diagram illustrating an example the network-based virtual reality playback system
  • FIG. 2 is a flowchart presents the process of the playback system in FIG. 1 ;
  • FIG. 3 is a flowchart presents the method for providing a viewport division scheme for VR video streaming
  • FIG. 4A-4C schematically show a core area, grid blocks and core blocks of the video data.
  • FIG. 5 is a block diagram illustrating a system for providing a viewport for VR video streaming.
  • FIG. 1 is a diagram illustrating an example network of a VR playback system.
  • the VR playback system 10 includes a server 100 and a display device 120 which are coupled with each other through a network 110 , and a VR device.
  • the server 100 may be a stand-alone computer server or a server cluster.
  • the server 100 is used to store various video data and to store various applications that process these video data.
  • various daemons run on the server 100 in real time, so as to process various video data in the server 100 and to respond various requests from VR devices and the display device 120 .
  • the network 110 may be a selected one or selected ones from the group consisting of an internet, a local area network, an internet of things, and the like.
  • the display device 120 may be any of the computing devices, including a computer device having an independent display screen and a processing capability.
  • the display device 120 may be a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a palmtop computer, a personal digital assistant, a smart phone, an intelligent electrical apparatus, a game console, an iPAD/iPhone, a video player, a DVD recorder/player, a television, or a home entertainment system.
  • the display device 120 may store VR player software as a VR player. When the VR player is started, it requests and downloads various video data from the server 100 , and renders and plays the video data in the display device.
  • the VR device 130 is a stand-alone head-mounted device that can interact with the display device 120 and the server 100 , and transmit information to the display device 120 and/or the server 100 by signal, including change of the position of a user's helmet, change of the eye sight, and the like.
  • the display device 120 can process flexibly the video data which is played according to the information.
  • the display device 120 determines that a central viewport of the user has been changed, and starts to play video data with a high resolution in the central viewport.
  • the VR device 130 is a stand-alone head-mounted device.
  • the VR device 130 is not limited thereto, and the VR device 130 may also be an all-in-one head-mounted device.
  • the all-in-one head-mounted device itself has a display screen, so that it is not necessary to connect the all-in-one head-mounted device with the external display device.
  • the display device 120 may be omitted.
  • the all-in-one head-mounted device is configured to obtain video data from the server 100 and to perform playback operation, and the all-in-one head-mounted device is also configured to detect user's field of view and to adjust the playback operation according to the user's field of view.
  • FIG. 2 is a flowchart showing a method used in the VR playback system of FIG. 1 . The method includes the following steps.
  • step S 10 the server performs source video data processing.
  • step S 20 display device side obtains some related information by interacting with the VR device.
  • step S 30 according to the related information, the display device side requests the server side to provide the video data and receives the video data.
  • step S 40 the display device side renders the received video data.
  • the step S 10 is used to process the video data stored on the server side.
  • the video data can be divided into those in a viewport are and those in a non-viewport area, and then can be processed by different methods to reduce data transmission for the non-viewport. Therefore, the present disclosure further provides a method for providing a viewport division scheme to achieve the above functions.
  • FIG. 3 is a flowchart presents the method for providing a viewport division scheme for VR video streaming. Specifically, the method comprises following steps.
  • step S 100 a projection area of VR video is obtained.
  • a projection process may be included in a VR video production process and a rendering process to change the VR video data to two-dimensional spatial data.
  • the original video data may be subjected to the projection process, for example, an equivalent rectangle matrix process with the aspect ratio of 2:1, to obtain a projection area.
  • the projection area can be obtained by decoding the VR video data.
  • Two main VR video formats include the video data with the aspect ratio of 1:1 and the aspect ratio of 2:1, which are well-known in the industry as film sources.
  • the server needs to perform a normalization process in view of the user's requirements.
  • the video data with the aspect ratio of 1:1 is processed into the video data with the aspect ratio of 2:1 which is suitable for the user's VR device.
  • the equivalent rectangle matrix projection for example, in an equally-spaced cylindrical projection, the meridians can be rendered to be vertical straight lines having a constant pitch, or the latitude circles can be rendered to be horizontal straight lines having a constant pitch.
  • the VR video data here refers to original data and/or intermediate data in any VR processing.
  • step S 200 a core area of the projection area is divided.
  • the projection area of the VR video data is obtained in the previous step S 100 .
  • the core area of the projection area is divided.
  • the data after being projected is generally distorted.
  • a full-view image may be rendered by an equivalent rectangle matrix projection process to obtain an area with the aspect ratio of 2:1.
  • stretching characteristics of the equivalent rectangle matrix projection when an image is close to a pole, the image will be stretched and distorted. The degree of pixel distortion will be higher. Therefore, according to stretching characteristics of the equivalent rectangle matrix projection, a central portion will be defined at the pole.
  • step S 300 a block size is defined according to the parameter N.
  • the parameter N is a given parameter.
  • an area of each grid block is obtained according to the N value. For example, assuming that the normalized video data has an area S and N is 32, each grid block has a block size equal to S/32.
  • step S 400 the core area is divided into grid blocks.
  • the core area can be divided into M core blocks based on a given value M.
  • the core blocks may be numbered for identifying each core block. Generally, the number starts from 0.
  • step S 500 the projection area is divided into grid blocks.
  • the projection area is divided into a plurality of equal portions, each of which corresponds to a grid block.
  • the grid block here is a portion of the VR video data.
  • each grid block is numbered. Generally, the number starts from 0.
  • step S 600 the core blocks are combined into a viewport block.
  • the viewport block contains one or more core blocks.
  • the viewport block is also changed. Accordingly, the core blocks in the viewport block are also changed. Therefore, the core blocks may be combined into the viewport block according to the user's field of view.
  • step S 700 a mapping table between the viewport blocks and the grid blocks is established.
  • the mapping table between the viewport blocks and the grid blocks is established.
  • the VR source video data is divided into a plurality of grid blocks, and a viewport block is defined according to a specific field of field of view (fov) to establish the mapping table between the viewport blocks and the grid blocks.
  • fov field of field of view
  • the viewport block is changed and obtained according to the mapping table.
  • the video data is processed by different methods for the viewport area and for the non-viewport area, so as to have a better viewing experience. Further, the user's field of view is proportional to the area of the viewport block.
  • Many viewport blocks can be defined for many fields of views (fov).
  • the grid blocks, the core blocks and the viewport blocks have identifications, and a mapping relationship between the viewport blocks and the grid blocks is established with the identifications.
  • a segmentation step may also be included in which the video data is divided into a plurality of sub-data according to a plurality of grid blocks which are obtained according to the above embodiment.
  • Each sub-data corresponds to a grid block. According to the above mapping relationship between the viewport blocks and the grid blocks, one can determine the mapping relationship between the plurality of sub-data and the plurality of viewport blocks.
  • mathematical equations can be established based on the identifications of the grid blocks, the core blocks and the viewport blocks, so that the identification of the viewport block can be determined according to the identifications of the grid blocks and/or the core blocks, or the identifications of the core blocks can be determined according to the identification of the viewport block. Details will be described in the following paragraphs.
  • each grid unit is a grid block and has the identification id block as follows,
  • width block height video n 2 ( 2 )
  • height video represents an image height
  • N + represents a positive integer
  • the core area is defined as a set of grid blocks Set core which satisfy Equation (3),
  • a core block is defined to have a width equal to that of the grid blocks and a height equal to that of the core are.
  • the identification id core _ block of the core block satisfies Equation (4),
  • id core_block ⁇ id block ⁇ ⁇ % ⁇ ⁇ 2 ⁇ n 2 , id block ⁇ Set core null , others ; ( 4 )
  • a viewport block is defined as having blocks in the core area, which are determined by a user's field of view fov and have a width equal to that of the core area.
  • a width width viewport _ block of the viewport block satisfies Equation (5),
  • width viewport_block fov * height video ⁇ , ( 5 )
  • the number of the core blocks in the viewport block is defined as i, which satisfies equation (6),
  • the identifications of the core blocks in the viewport block can be obtained according to Equation (7),
  • f ⁇ ( id viewport_block ) ⁇ ⁇ id viewport_block , id viewport_block + 1 , ... ⁇ , id viewport_block + i - 2 , id viewport_block + i - 1 ⁇ ; id viewport_block ⁇ [ 0 , 2 ⁇ n 2 - i ] ⁇ id viewport_block , id viewport_block + 1 , ... ⁇ , 2 ⁇ n 2 - 1 , 0 , 1 , ... ⁇ , i - ( 2 ⁇ n 2 - id viewport_block ) - 1 ⁇ ; , id viewport_block ⁇ [ 2 ⁇ n 2 - i + 1 , 2 ⁇ n 2 - 1 ] ; ⁇ ( 7 ) ( 7 )
  • the identification of the viewport block including the core block which represents the identification of the first core block, can be obtained uniquely according to Equation (8),
  • the identification satisfies Equation (8),
  • id core _ block _ 1st represents the identification of the first core block that is included in the viewport block.
  • FIG. 4A-4C schematically show a core area, grid blocks and core blocks of the video data.
  • 400 represents a projection area
  • 401 represents a core area
  • N is equal to 32. That is to say, the projection area is divided into 32 grid blocks, which are numbered as 0-31.
  • the core blocks are numbered as 0-7.
  • one viewport block (not shown) contains two core blocks which are numbered as 1 and 2, the viewport block contains those grid blocks being numbered as 9-10 and 17-18.
  • the identifications of the grid blocks, the core blocks and the viewport blocks may be numbered and be verified by the above equations.
  • this example is given only for the mapping relationship among the grid blocks, the core blocks and the viewport blocks under ideal conditions. That is to say, in this example, the core blocks and the grid blocks and the viewport blocks overlap each other. However, the present disclosure is not limited to this and can be applied in a case that the grid blocks, the core blocks and the viewport blocks partially overlap each other.
  • FIG. 5 is a block diagram illustrating a system for providing a viewport for VR video streaming.
  • the system includes a configuration module 501 , a first division module 503 , a second division module 502 , a third division module 504 , a combination module 505 , and a mapping module 506 .
  • the configuration module 501 is configured to obtain a projection area of VR video.
  • the VR video data can decoded to obtain the projection area of the video data from a parameter.
  • the projection area can be obtained by projecting the VR video into a rectangle by an equivalent rectangle matrix projection process.
  • Two main VR video formats include the video data with the aspect ratio of 1:1 and the aspect ratio of 2:1, which are well-known in the industry as film sources.
  • the original VR video format may be normalized to have the aspect ratio of 2:1, and then further processed.
  • the first division module 503 is configured to divide the projection area into N grid blocks. For example, the projection area is divided into 32 equal portions. As a result, the 32 grid blocks each are a portion of the projection area with equal size.
  • the second division module 502 is configured to divide a core area of the projection area according to the user's field of view.
  • the core area is defined in the projection area. For example, for a rectangle, a central portion in the X-axis direction or the Y-axis direction is defined as the core area.
  • the third division module 504 is configured to divide the core area into a plurality of core blocks.
  • the core area is divided into a plurality of grid units, generally having equal area, but is not limited thereto. For example, if an area of an edge unit is insufficient, the area of the edge unit will be smaller.
  • the combination module 505 is configured to combine the core blocks into a plurality of viewport blocks.
  • the core blocks are combined according to a predetermined rule. For example, two adjacent core blocks are combined into one viewport block. In another example, a core block at a central point and those core blocks surrounding the central point are combined into one viewport block.
  • the mapping module 506 is configured to establish a mapping relationship between the viewport blocks and the grid blocks.
  • Each viewport block contains a plurality of core blocks.
  • Each core block may are mixed with one or more grid blocks.
  • one or more grid blocks may be retrieved according to the core blocks.
  • the system can locate the viewport block and improve viewing experience effect by optimizing the video data of the viewport block.
  • the above system for providing a viewport for VR video streaming may further comprise a segmentation module (not shown).
  • the segmentation module divides the video data into a plurality of sub-data according to the plurality of grid blocks. Each sub-data corresponds to a grid.
  • the mapping relationship between the plurality of sub-data and the plurality of viewport blocks can be determined by the mapping relationship between the plurality of viewport blocks and the plurality of grid blocks.
  • the configuration module projects the VR video into a rectangle with the aspect ratio of 2:1 by equivalent rectangle matrix to obtain the projection area.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Computer Graphics (AREA)
  • Human Computer Interaction (AREA)
  • Geometry (AREA)
  • Environmental & Geological Engineering (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Controls And Circuits For Display Device (AREA)
  • Image Generation (AREA)
  • Processing Or Creating Images (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Image Processing (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)

Abstract

A method and a system for providing a viewport division scheme for VR video streaming are disclosed. A projection area of VR video is obtained. The projection area is divided into a plurality of grid blocks. A core area is defined in the projection area and divided into a plurality of core blocks. A plurality of viewport blocks are obtained by combining the core blocks. A mapping relationship between the plurality of viewport blocks and the grid blocks is established. The plurality of viewport blocks are related to a user's field of view. The plurality of viewport blocks are obtained in accordance with a mapping table. The video data is processed by different methods for a viewport area and a non-viewport area to have a better viewing experience.

Description

    CROSS-REFERENCE TO RELATED APPLICATION
  • This application claims the priority and benefit of U.S. provisional application 62/441,936, filed on Jan. 3, 2017, which is incorporated herein by reference in its entirety.
  • BACKGROUND OF THE DISCLOSURE Field of the Disclosure
  • The present disclosure relates to video processing technology, and more particularly, to a method and a system for providing a viewport division scheme for virtual reality (VR) video streaming BACKGROUND OF THE DISCLOSURE
  • Virtual Reality (VR) is a computer simulation technology for creating and experiencing a virtual world. For example, a three-dimensional real-time image can be presented based on a technology which tracks a user's head, eyes or hands. In the network-based virtual reality technology, full-view video data can be pre-stored on a server, and then transmitted to a display device. A display device can be glasses, a head-mounted display, etc. A video is displayed on the display device according to a user's field of view.
  • However, a high-resolution video data occupies large transmission of bandwidth and requires high computing power from the display devices. Presenting high-resolution VR video on internet is difficult. Precisely, the existing video streaming technology can not fulfill the virtual reality. Therefore, in order to present VR video smoothly in real-time, it is desirable to further improve the existing video streaming technology to save bandwidth and reduce performance requirements for display devices, by a new way to encode and store the VR video data on the server.
  • It is a problem in the art to play back video data on a player end with a better viewing experience through less video data transmission.
  • SUMMARY OF THE DISCLOSURE
  • In view of this, the present disclosure provides a method and a system for providing a viewport division scheme for virtual reality (VR) video streaming, which determine a viewport block of the VR video data and process the VR video data by different methods for a viewport area and a non-viewport area so as to have a better viewing experience at a player end through less data transmission.
  • According to one aspect of the present disclosure, there is provided a method for providing a viewport division scheme for VR video streaming, comprising:
  • obtaining a projection area of VR video;
  • dividing the projection area into a plurality of grid blocks;
  • defining a core area in the projection area;
  • dividing the core area into a plurality of core blocks;
  • combining the core blocks to provide a plurality of viewport blocks; and
  • establishing a mapping relation between the plurality of viewport blocks and the plurality of grid blocks, wherein the plurality of viewport blocks are related to a user's field of view.
  • Preferably, the step of obtaining a projection area of VR video comprises:
  • projecting the VR video into a rectangle with the aspect ratio of 2:1 by equivalent rectangle matrix to obtain the projection area.
  • Preferably, the step of dividing the projection area into a plurality of grid blocks comprises:
  • dividing the projection area into a plurality of equal portions; and
  • dividing each of the plurality of equal portions into the same number of grid blocks.
  • Preferably, the method further comprises creating identifications for the plurality of grid blocks, the plurality of core blocks and the viewport, and establishing a mapping relationship between the viewport and the plurality of grid blocks by the identifications.
  • Preferably, the step of creating identifications for the plurality of grid blocks, the plurality of core blocks and the viewport, and establishing a mapping relationship between the viewport and the plurality of grid blocks by the identifications comprises:
  • setting the identificationidblock of a grid block as a positive integer started from zero, so as to satisfy Equation (1)

  • idblockϵ[0,n−1],idblock ϵN +  (1);
  • setting an edge length of the grid block to satisfy Equation (2),
  • width block = height video n 2 ( 2 )
  • where heightvideo represents an image height, and N+ represents a positive integer;
  • setting the core area to be a set of grid blocks Setcore which satisfy Equation (3),
  • Set core = { id block | id block = { n 4 , n 4 + 1 , , 3 n 4 - 2 , 3 n 4 - 1 } } ; ( 3 )
  • defining a core block to have a width equal to that of the plurality of grid blocks and a height equal to that of the core area, and setting the identification idcore _ block of the core block, so as to satisfy Equation (4),
  • id core_block = { id block % 2 n 2 , id block Set core null , others ; ( 4 )
  • defining a viewport block in the core area to have a width determined by a user's field of view and a height equal to that of the core area, where a width widthviewport _ block of the viewport block satisfies Equation (5),
  • width viewport_block = fov * height video π , ( 5 )
  • where fov represents the user's field of view;
  • defining the number of the core blocks in the viewport block to be i, which satisfies equation (6),
  • i = width viewport_block width block ; ( 6 )
  • for any identification of the viewport block, the identifications of the core blocks in the viewport block can be obtained according to Equation (7),
  • f ( id viewport_block ) = { { id viewport_block , id viewport_block + 1 , , id viewport_block + i - 2 , id viewport_block + i - 1 } ; id viewport_block [ 0 , 2 n 2 - i ] { id viewport_block , id viewport_block + 1 , , 2 n 2 - 1 , 0 , 1 , , i - ( 2 n 2 - id viewport_block ) - 1 } ; , id viewport_block [ 2 n 2 - i + 1 , 2 n 2 - 1 ] ; ( 7 ) ( 7 )
  • for any identification of the core block, the identification of the viewport block including the core block can be obtained uniquely according to Equation (8),

  • idviewport _ block =f(idcore _ block)=idcore _ block _ 1st  (8),
  • where idcore _ block _ 1st represents the identification of the first core block that is included in the viewport block.
  • Preferably, the step of dividing the core area into the plurality of core blocks comprises:
  • dividing the core area into the plurality of core blocks with equal areas.
  • Preferably, the plurality core blocks each have a height equal to of the central and the core area has an area equal to integral times of the area of a grid block.
  • Preferably, the method further comprises dividing the VR video data into a plurality of sub-data according to the plurality of grid blocks, wherein each sub-data corresponds to one of the grid blocks, and determining a mapping relationship between the plurality of sub-data and the viewport according to the mapping relationship between the plurality of grid blocks and the viewport.
  • Preferably, the method further comprises creating a plurality of viewport blocks according to a user's field of view.
  • According to another aspect of the present disclosure, there is provided a system for providing a viewport for VR video streaming, comprising:
  • a configuration module, configured to obtain a projection area of VR video;
  • a first division module, configured to divide the projection area into a plurality of grid blocks;
  • a second division module, configured to define a core area in the projection area;
  • a third division module, configured to divide the core area into a plurality of core blocks;
  • a combination module, configured to combine the core blocks to provide a plurality of viewport blocks;
  • a mapping module, configured to establish a mapping relationship between the viewport and the plurality of grid blocks, wherein the plurality of viewport blocks are related to a user's field of view.
  • Preferably, the configuration module
  • projects the VR video into a rectangle with the aspect ratio of 2:1 by equivalent rectangle matrix to obtain the projection area.
  • In this embodiment, the VR source video data is divided into a plurality of grid blocks, and a mapping table between the viewport blocks and the grid blocks is established according to a specific field of view. When the user's field of view is changed, the viewport block is retrieved according to the mapping table, and the video data of a viewport area and a non-viewport area are processed by different methods to have a better view experience.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • The above and other objects, advantages and features of the present disclosure will become more fully understood from the detailed description given hereinbelow in connection reference with the appended drawings, and wherein:
  • FIG. 1 is a diagram illustrating an example the network-based virtual reality playback system;
  • FIG. 2 is a flowchart presents the process of the playback system in FIG. 1;
  • FIG. 3 is a flowchart presents the method for providing a viewport division scheme for VR video streaming;
  • FIG. 4A-4C schematically show a core area, grid blocks and core blocks of the video data; and
  • FIG. 5 is a block diagram illustrating a system for providing a viewport for VR video streaming.
  • DETAILED DESCRIPTION OF THE DISCLOSURE
  • Exemplary embodiments of the present disclosure will be described in more details below with reference to the accompanying drawings. In the drawings, like reference numerals denote like members. The figures are not drawn to scale, for the sake of clarity. Moreover, some well known parts may not be shown.
  • FIG. 1 is a diagram illustrating an example network of a VR playback system. The VR playback system 10 includes a server 100 and a display device 120 which are coupled with each other through a network 110, and a VR device. For example, the server 100 may be a stand-alone computer server or a server cluster. The server 100 is used to store various video data and to store various applications that process these video data. For example, various daemons run on the server 100 in real time, so as to process various video data in the server 100 and to respond various requests from VR devices and the display device 120. The network 110 may be a selected one or selected ones from the group consisting of an internet, a local area network, an internet of things, and the like. For example, the display device 120 may be any of the computing devices, including a computer device having an independent display screen and a processing capability. The display device 120 may be a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a palmtop computer, a personal digital assistant, a smart phone, an intelligent electrical apparatus, a game console, an iPAD/iPhone, a video player, a DVD recorder/player, a television, or a home entertainment system. The display device 120 may store VR player software as a VR player. When the VR player is started, it requests and downloads various video data from the server 100, and renders and plays the video data in the display device. In this example, the VR device 130 is a stand-alone head-mounted device that can interact with the display device 120 and the server 100, and transmit information to the display device 120 and/or the server 100 by signal, including change of the position of a user's helmet, change of the eye sight, and the like. The display device 120 can process flexibly the video data which is played according to the information. In some embodiments, when a user's position is changed, the display device 120 determines that a central viewport of the user has been changed, and starts to play video data with a high resolution in the central viewport.
  • In the above embodiment, the VR device 130 is a stand-alone head-mounted device. However, those skilled in the art should understand that the VR device 130 is not limited thereto, and the VR device 130 may also be an all-in-one head-mounted device. The all-in-one head-mounted device itself has a display screen, so that it is not necessary to connect the all-in-one head-mounted device with the external display device. For example, in this example, if the all-in-one head-mounted device is used as the VR device, the display device 120 may be omitted. At this point, the all-in-one head-mounted device is configured to obtain video data from the server 100 and to perform playback operation, and the all-in-one head-mounted device is also configured to detect user's field of view and to adjust the playback operation according to the user's field of view.
  • FIG. 2 is a flowchart showing a method used in the VR playback system of FIG. 1. The method includes the following steps.
  • In step S10, the server performs source video data processing.
  • In step S20, display device side obtains some related information by interacting with the VR device.
  • In step S30, according to the related information, the display device side requests the server side to provide the video data and receives the video data.
  • In step S40, the display device side renders the received video data.
  • Wherein, the step S10 is used to process the video data stored on the server side. In this step, the video data can be divided into those in a viewport are and those in a non-viewport area, and then can be processed by different methods to reduce data transmission for the non-viewport. Therefore, the present disclosure further provides a method for providing a viewport division scheme to achieve the above functions.
  • FIG. 3 is a flowchart presents the method for providing a viewport division scheme for VR video streaming. Specifically, the method comprises following steps.
  • In step S100, a projection area of VR video is obtained.
  • Although the VR video data is captured in a 360-degree panoramic view of the real-world environment, a projection process may be included in a VR video production process and a rendering process to change the VR video data to two-dimensional spatial data. The original video data may be subjected to the projection process, for example, an equivalent rectangle matrix process with the aspect ratio of 2:1, to obtain a projection area. In a case that the VR video data has been subjected to the projection process, the projection area can be obtained by decoding the VR video data. Two main VR video formats include the video data with the aspect ratio of 1:1 and the aspect ratio of 2:1, which are well-known in the industry as film sources. In some cases, the server needs to perform a normalization process in view of the user's requirements. For example, the video data with the aspect ratio of 1:1 is processed into the video data with the aspect ratio of 2:1 which is suitable for the user's VR device. In the equivalent rectangle matrix projection, for example, in an equally-spaced cylindrical projection, the meridians can be rendered to be vertical straight lines having a constant pitch, or the latitude circles can be rendered to be horizontal straight lines having a constant pitch. It should be noted that the VR video data here refers to original data and/or intermediate data in any VR processing.
  • In step S200, a core area of the projection area is divided.
  • As described above, the projection area of the VR video data is obtained in the previous step S100. In this step, the core area of the projection area is divided. The data after being projected is generally distorted. For example, a full-view image may be rendered by an equivalent rectangle matrix projection process to obtain an area with the aspect ratio of 2:1. According to stretching characteristics of the equivalent rectangle matrix projection, when an image is close to a pole, the image will be stretched and distorted. The degree of pixel distortion will be higher. Therefore, according to stretching characteristics of the equivalent rectangle matrix projection, a central portion will be defined at the pole.
  • In step S300, a block size is defined according to the parameter N.
  • The parameter N is a given parameter. In this step, an area of each grid block is obtained according to the N value. For example, assuming that the normalized video data has an area S and N is 32, each grid block has a block size equal to S/32.
  • In step S400, the core area is divided into grid blocks.
  • After the core area is determined, the core area can be divided into M core blocks based on a given value M. The core blocks may be numbered for identifying each core block. Generally, the number starts from 0.
  • In step S500, the projection area is divided into grid blocks.
  • In this step, the projection area is divided into a plurality of equal portions, each of which corresponds to a grid block. It should be understood that the grid block here is a portion of the VR video data. After that, each grid block is numbered. Generally, the number starts from 0.
  • In step S600, the core blocks are combined into a viewport block.
  • The viewport block contains one or more core blocks. When a field of view of the VR device is changed, the viewport block is also changed. Accordingly, the core blocks in the viewport block are also changed. Therefore, the core blocks may be combined into the viewport block according to the user's field of view.
  • In step S700, a mapping table between the viewport blocks and the grid blocks is established.
  • Based on the viewport block of each VR video data, which is obtained in step S600, the mapping table between the viewport blocks and the grid blocks is established. In this embodiment, the VR source video data is divided into a plurality of grid blocks, and a viewport block is defined according to a specific field of field of view (fov) to establish the mapping table between the viewport blocks and the grid blocks. When the user's field of view is changed, the viewport block is changed and obtained according to the mapping table. The video data is processed by different methods for the viewport area and for the non-viewport area, so as to have a better viewing experience. Further, the user's field of view is proportional to the area of the viewport block. Many viewport blocks can be defined for many fields of views (fov). In an alternative embodiment, the grid blocks, the core blocks and the viewport blocks have identifications, and a mapping relationship between the viewport blocks and the grid blocks is established with the identifications.
  • It should be noted that in the above embodiments, the video data is not actually segmented. Therefore, in an optional embodiment of the present disclosure, a segmentation step may also be included in which the video data is divided into a plurality of sub-data according to a plurality of grid blocks which are obtained according to the above embodiment. Each sub-data corresponds to a grid block. According to the above mapping relationship between the viewport blocks and the grid blocks, one can determine the mapping relationship between the plurality of sub-data and the plurality of viewport blocks.
  • Under some limitations, mathematical equations can be established based on the identifications of the grid blocks, the core blocks and the viewport blocks, so that the identification of the viewport block can be determined according to the identifications of the grid blocks and/or the core blocks, or the identifications of the core blocks can be determined according to the identification of the viewport block. Details will be described in the following paragraphs.
  • Assuming that an image of the video data has the aspect ratio of 2:1 and is divided into n grid units with the aspect ratio of 1:1, and each grid unit is a grid block and has the identification idblock as follows,

  • idblockϵ[0,n−1],idblock ϵN +  (1);
  • An edge length of the grid block satisfies Equation (1).
  • width block = height video n 2 ( 2 )
  • where heightvideo represents an image height, and N+ represents a positive integer;
  • According characteristics of the equivalent rectangle matrix projection, the core area is defined as a set of grid blocks Setcore which satisfy Equation (3),
  • Set core = { id block id block = { n 4 , n 4 + 1 , , 3 n 4 - 2 , 3 n 4 - 1 } } ; ( 3 )
  • A core block is defined to have a width equal to that of the grid blocks and a height equal to that of the core are. The identification idcore _ block of the core block satisfies Equation (4),
  • id core_block = { id block % 2 n 2 , id block Set core null , others ; ( 4 )
  • A viewport block is defined as having blocks in the core area, which are determined by a user's field of view fov and have a width equal to that of the core area. A width widthviewport _ block of the viewport block satisfies Equation (5),
  • width viewport_block = fov * height video π , ( 5 )
  • The number of the core blocks in the viewport block is defined as i, which satisfies equation (6),
  • i = width viewport_block width block ; ( 6 )
  • For any identification of the viewport block, the identifications of the core blocks in the viewport block can be obtained according to Equation (7),
  • f ( id viewport_block ) = { { id viewport_block , id viewport_block + 1 , , id viewport_block + i - 2 , id viewport_block + i - 1 } ; id viewport_block [ 0 , 2 n 2 - i ] { id viewport_block , id viewport_block + 1 , , 2 n 2 - 1 , 0 , 1 , , i - ( 2 n 2 - id viewport_block ) - 1 } ; , id viewport_block [ 2 n 2 - i + 1 , 2 n 2 - 1 ] ; ( 7 ) ( 7 )
  • For any identification of the core block, the identification of the viewport block including the core block, which represents the identification of the first core block, can be obtained uniquely according to Equation (8), The identification satisfies Equation (8),

  • idviewport _ block =f(idcore _ block)=idcore _ block _ 1st  (8),
  • where idcore _ block _ 1st represents the identification of the first core block that is included in the viewport block.
  • FIG. 4A-4C schematically show a core area, grid blocks and core blocks of the video data.
  • In this example, 400 represents a projection area, 401 represents a core area, and N is equal to 32. That is to say, the projection area is divided into 32 grid blocks, which are numbered as 0-31. The core blocks are numbered as 0-7. Assuming that one viewport block (not shown) contains two core blocks which are numbered as 1 and 2, the viewport block contains those grid blocks being numbered as 9-10 and 17-18. Thus, it is possible to establish a mapping relationship between the viewport blocks and the grid blocks. In this case, the identifications of the grid blocks, the core blocks and the viewport blocks may be numbered and be verified by the above equations.
  • It should be understood that this example is given only for the mapping relationship among the grid blocks, the core blocks and the viewport blocks under ideal conditions. That is to say, in this example, the core blocks and the grid blocks and the viewport blocks overlap each other. However, the present disclosure is not limited to this and can be applied in a case that the grid blocks, the core blocks and the viewport blocks partially overlap each other.
  • FIG. 5 is a block diagram illustrating a system for providing a viewport for VR video streaming.
  • The system includes a configuration module 501, a first division module 503, a second division module 502, a third division module 504, a combination module 505, and a mapping module 506.
  • The configuration module 501 is configured to obtain a projection area of VR video. In a case that the VR video data has been processed into two-dimensional data, the VR video data can decoded to obtain the projection area of the video data from a parameter. In another case that the video data is not processed into two-dimensional data, the projection area can be obtained by projecting the VR video into a rectangle by an equivalent rectangle matrix projection process. Two main VR video formats include the video data with the aspect ratio of 1:1 and the aspect ratio of 2:1, which are well-known in the industry as film sources. The original VR video format may be normalized to have the aspect ratio of 2:1, and then further processed.
  • The first division module 503 is configured to divide the projection area into N grid blocks. For example, the projection area is divided into 32 equal portions. As a result, the 32 grid blocks each are a portion of the projection area with equal size.
  • The second division module 502 is configured to divide a core area of the projection area according to the user's field of view. The core area is defined in the projection area. For example, for a rectangle, a central portion in the X-axis direction or the Y-axis direction is defined as the core area.
  • The third division module 504 is configured to divide the core area into a plurality of core blocks. The core area is divided into a plurality of grid units, generally having equal area, but is not limited thereto. For example, if an area of an edge unit is insufficient, the area of the edge unit will be smaller.
  • The combination module 505 is configured to combine the core blocks into a plurality of viewport blocks. The core blocks are combined according to a predetermined rule. For example, two adjacent core blocks are combined into one viewport block. In another example, a core block at a central point and those core blocks surrounding the central point are combined into one viewport block.
  • The mapping module 506 is configured to establish a mapping relationship between the viewport blocks and the grid blocks. Each viewport block contains a plurality of core blocks. Each core block may are mixed with one or more grid blocks. Correspondingly, one or more grid blocks may be retrieved according to the core blocks.
  • By establishing the mapping relationship between the viewport block and the grid blocks, the system can locate the viewport block and improve viewing experience effect by optimizing the video data of the viewport block.
  • Preferably, the above system for providing a viewport for VR video streaming may further comprise a segmentation module (not shown). The segmentation module divides the video data into a plurality of sub-data according to the plurality of grid blocks. Each sub-data corresponds to a grid. The mapping relationship between the plurality of sub-data and the plurality of viewport blocks can be determined by the mapping relationship between the plurality of viewport blocks and the plurality of grid blocks.
  • The configuration module projects the VR video into a rectangle with the aspect ratio of 2:1 by equivalent rectangle matrix to obtain the projection area.
  • The embodiments were chosen and described in order to best explain the principles of the disclosure and its practical applications, to thereby enable others skilled in the art to best utilize the disclosure. The disclosure is intended to cover alternatives, modifications and equivalents that may be included within the spirit and scope of the disclosure as defined by the appended claims.
  • The foregoing descriptions of specific embodiments of the present disclosure have been presented, but are not intended to limit the disclosure to the precise forms disclosed. It will be readily apparent to one skilled in the art that many modifications and changes may be made in the present disclosure. Any modifications, equivalence, variations of the preferred embodiments can be made without departing from the doctrine and spirit of the present disclosure.

Claims (11)

1. A method for providing a viewport division scheme for VR video streaming, comprising:
obtaining a projection area of VR video;
dividing the projection area into a plurality of grid blocks;
defining a core area in the projection area;
dividing the core area into a plurality of core blocks;
obtaining a plurality of viewport blocks by combining the plurality of core blocks; and
establishing a mapping relation between the plurality of viewport blocks and the plurality of grid blocks, wherein the plurality of viewport blocks is related to a user's field of view.
2. The method according to claim 1, wherein the step of obtaining the projection area of VR video comprises:
projecting the VR video into a rectangle with the aspect ratio of 2:1 by equivalent rectangle matrix to obtain the projection area.
3. The method according to claim 1, wherein the step of dividing the projection area into the plurality of grid blocks comprises:
dividing the projection area into a plurality of equal portions.
4. The method according to claim 1, further comprising: creating identifications for the plurality of grid blocks, the plurality of core blocks and the plurality of viewport blocks, and establishing a mapping relationship between the plurality of viewport blocks and the plurality of grid blocks by the identifications.
5. The method according to claim 4, wherein the step of creating identifications for the plurality of grid blocks, the plurality of core blocks and the viewport, and establishing the mapping relationship between the plurality of viewport blocks and the plurality of grid blocks by the identifications comprises:
setting the identification idblock of a grid block as a positive integer started from zero, so as to satisfy Equation (1)

idblockϵ[0,n−1],idblock ϵN +  (1);
setting an edge length of the grid block to satisfy Equation (2),
width block = height video n 2 ( 2 )
where heightvideo represents an image height, and N+ represents a positive integer;
setting the core area to be a set of grid blocks Setcore which satisfy Equation (3),
Set core = { id block id block = { n 4 , n 4 + 1 , , 3 n 4 - 2 , 3 n 4 - 1 } } ; ( 3 )
defining a core block to have a width equal to that of the grid blocks and a height equal to that of the core area, and setting the identification idcore _ block of the core block, so as to satisfy Equation (4),
id core_block = { id block % 2 n 2 , id block Set core null , others ; ( 4 )
defining a viewport block in the core area to have a width determined by a user's field of view and a height equal to that of the core area, where a width widthviewport _ block of the viewport block satisfies Equation (5),
width viewport_block = fov * height video π , ( 5 )
where fov represents the user's field of view;
defining the number of the core blocks in the viewport block to be i, which satisfies equation (6),
i = width viewport_block width block ; ( 6 )
for any identification of the viewport block, the identifications of the core blocks in the viewport block can be obtained according to Equation (7),
f ( id viewport_block ) = { { id viewport_block , id viewport_block + 1 , , id viewport_block + i - 2 , id viewport_block + i - 1 } ; id viewport_block [ 0 , 2 n 2 - i ] { id viewport_block , id viewport_block + 1 , , 2 n 2 - 1 , 0 , 1 , , i - ( 2 n 2 - id viewport_block ) - 1 } ; , id viewport_block [ 2 n 2 - i + 1 , 2 n 2 - 1 ] ; ( 7 ) ( 7 )
for any identification of the core block, the identification of the viewport block can be obtained uniquely according to Equation (8),

idviewport _ block =f(idcore _ block)=idcore _ block _ 1st  (8),
where idcore _ block _ 1st represents the identification of the first core block that is included in the viewport block.
6. The method according to claim 1, wherein the step of dividing the core area into the plurality of core blocks comprises:
dividing the core area into the plurality of core blocks with equal areas.
7. The method according to claim 1, wherein the plurality of core blocks each have a height equal to that of the core area, and the core area has an area equal to an integral times that of the area of a grid block.
8. The method according to claim 1, further comprising: dividing the VR video data into a plurality of sub-data according to the plurality of grid blocks, wherein each sub-data corresponds to one of the grid blocks, and determining a mapping relationship between the plurality of sub-data and the viewport according to the mapping relationship between the plurality of grid blocks and the viewport.
9. The method according to claim 1, further comprising: creating a plurality of viewport blocks according to a user's field of view.
10. A system for providing a viewport division scheme for VR video streaming, comprising:
a configuration module, configured to obtain a projection area of VR video;
a first division module, configured to divide the projection area into a plurality of grid blocks;
a second division module, configured to define a core area in the projection area;
a third division module, configured to divide the core area into a plurality of core blocks;
a combination module, configured to combine the core blocks to provide a plurality of viewport blocks; and
a mapping module, configured to establish a mapping relationship between the plurality of viewport blocks and the grid blocks, wherein the plurality of viewport blocks are related to a user's field of view.
11. The system according to claim 1, wherein the configuration module projects the VR video into a rectangle with the aspect ratio of 2:1 by equivalent rectangle matrix to obtain the projection area.
US15/860,392 2017-01-03 2018-01-02 Method and System for Providing A Viewport Division Scheme for Virtual Reality (VR) Video Streaming Abandoned US20180192044A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US15/860,392 US20180192044A1 (en) 2017-01-03 2018-01-02 Method and System for Providing A Viewport Division Scheme for Virtual Reality (VR) Video Streaming

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201762441936P 2017-01-03 2017-01-03
US15/860,392 US20180192044A1 (en) 2017-01-03 2018-01-02 Method and System for Providing A Viewport Division Scheme for Virtual Reality (VR) Video Streaming

Publications (1)

Publication Number Publication Date
US20180192044A1 true US20180192044A1 (en) 2018-07-05

Family

ID=62711388

Family Applications (6)

Application Number Title Priority Date Filing Date
US15/860,449 Expired - Fee Related US10334238B2 (en) 2017-01-03 2018-01-02 Method and system for real-time rendering displaying high resolution virtual reality (VR) video
US15/860,392 Abandoned US20180192044A1 (en) 2017-01-03 2018-01-02 Method and System for Providing A Viewport Division Scheme for Virtual Reality (VR) Video Streaming
US15/860,494 Abandoned US20180189980A1 (en) 2017-01-03 2018-01-02 Method and System for Providing Virtual Reality (VR) Video Transcoding and Broadcasting
US15/860,358 Abandoned US20180192063A1 (en) 2017-01-03 2018-01-02 Method and System for Virtual Reality (VR) Video Transcode By Extracting Residual From Different Resolutions
US15/860,471 Abandoned US20180192022A1 (en) 2017-01-03 2018-01-02 Method and System for Real-time Rendering Displaying Virtual Reality (VR) On Mobile Using Head-Up Display Devices
US15/860,430 Abandoned US20180191868A1 (en) 2017-01-03 2018-01-02 Method and System for Downloading Multiple Resolutions Bitrate for Virtual Reality (VR) Video Streaming Optimization

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US15/860,449 Expired - Fee Related US10334238B2 (en) 2017-01-03 2018-01-02 Method and system for real-time rendering displaying high resolution virtual reality (VR) video

Family Applications After (4)

Application Number Title Priority Date Filing Date
US15/860,494 Abandoned US20180189980A1 (en) 2017-01-03 2018-01-02 Method and System for Providing Virtual Reality (VR) Video Transcoding and Broadcasting
US15/860,358 Abandoned US20180192063A1 (en) 2017-01-03 2018-01-02 Method and System for Virtual Reality (VR) Video Transcode By Extracting Residual From Different Resolutions
US15/860,471 Abandoned US20180192022A1 (en) 2017-01-03 2018-01-02 Method and System for Real-time Rendering Displaying Virtual Reality (VR) On Mobile Using Head-Up Display Devices
US15/860,430 Abandoned US20180191868A1 (en) 2017-01-03 2018-01-02 Method and System for Downloading Multiple Resolutions Bitrate for Virtual Reality (VR) Video Streaming Optimization

Country Status (2)

Country Link
US (6) US10334238B2 (en)
CN (6) CN108419142A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10331862B2 (en) * 2017-04-20 2019-06-25 Cisco Technology, Inc. Viewport decryption

Families Citing this family (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10291910B2 (en) * 2016-02-12 2019-05-14 Gopro, Inc. Systems and methods for spatially adaptive video encoding
CN115842907A (en) * 2018-03-27 2023-03-24 京东方科技集团股份有限公司 Rendering method, computer product and display device
US11232532B2 (en) * 2018-05-30 2022-01-25 Sony Interactive Entertainment LLC Multi-server cloud virtual reality (VR) streaming
US10623791B2 (en) 2018-06-01 2020-04-14 At&T Intellectual Property I, L.P. Field of view prediction in live panoramic video streaming
US10812774B2 (en) 2018-06-06 2020-10-20 At&T Intellectual Property I, L.P. Methods and devices for adapting the rate of video content streaming
US10616621B2 (en) * 2018-06-29 2020-04-07 At&T Intellectual Property I, L.P. Methods and devices for determining multipath routing for panoramic video content
US11019361B2 (en) 2018-08-13 2021-05-25 At&T Intellectual Property I, L.P. Methods, systems and devices for adjusting panoramic view of a camera for capturing video content
CN109343518B (en) * 2018-09-03 2021-07-02 浙江大丰实业股份有限公司 On-spot drive platform of universal ride
US11128869B1 (en) * 2018-10-22 2021-09-21 Bitmovin, Inc. Video encoding based on customized bitrate table
CN109375369B (en) * 2018-11-23 2021-05-18 国网天津市电力公司 Distortion preprocessing method in VR (virtual reality) large-screen cinema mode
CN111510777B (en) * 2019-01-30 2021-11-23 上海哔哩哔哩科技有限公司 Method and device for measuring network speed, computer equipment and readable storage medium
CN111669666A (en) * 2019-03-08 2020-09-15 北京京东尚科信息技术有限公司 Method, device and system for simulating reality
CN111866485A (en) * 2019-04-25 2020-10-30 中国移动通信有限公司研究院 Stereoscopic picture projection and transmission method, device and computer readable storage medium
CN110336994B (en) * 2019-07-04 2021-06-22 上海索倍信息科技有限公司 Naked eye 3D display system
CN110381331A (en) * 2019-07-23 2019-10-25 深圳市道通智能航空技术有限公司 A kind of image processing method, device, equipment of taking photo by plane and storage medium
CN110490962B (en) * 2019-08-20 2023-09-15 武汉邦拓信息科技有限公司 Remote rendering method based on video stream
CN110544425A (en) * 2019-09-13 2019-12-06 广州城市职业学院 ancient building VR display system
CN111489428B (en) * 2020-04-20 2023-06-30 北京字节跳动网络技术有限公司 Image generation method, device, electronic equipment and computer readable storage medium
US11245911B1 (en) * 2020-05-12 2022-02-08 Whirlwind 3D, LLC Video encoder/decoder (codec) for real-time applications and size/b and width reduction
CN111754614A (en) * 2020-06-30 2020-10-09 平安国际智慧城市科技股份有限公司 Video rendering method and device based on VR (virtual reality), electronic equipment and storage medium
CN112468806B (en) * 2020-11-12 2022-07-26 中山大学 Panoramic video transmission optimization method for cloud VR platform
CN114286142B (en) * 2021-01-18 2023-03-28 海信视像科技股份有限公司 Virtual reality equipment and VR scene screen capturing method
CN113347402A (en) * 2021-06-28 2021-09-03 筑友建筑装饰装修工程有限公司 Improved method, device and storage medium for rendering immersive content based on Unity
CN114466220A (en) * 2022-01-29 2022-05-10 维沃移动通信有限公司 Video downloading method and electronic equipment
CN115396731A (en) * 2022-08-10 2022-11-25 北京势也网络技术有限公司 Panoramic video playing method and device, electronic equipment and readable storage medium
CN116880723B (en) * 2023-09-08 2023-11-17 江西格如灵科技股份有限公司 3D scene display method and system

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070292113A1 (en) * 2005-06-30 2007-12-20 Meng-Nan Tsou Video decoding apparatus, video decoding method, and digital audio/video playback system capable of controlling presentation of sub-pictures
US20130044108A1 (en) * 2011-03-31 2013-02-21 Panasonic Corporation Image rendering device, image rendering method, and image rendering program for rendering stereoscopic panoramic images
US20140225816A1 (en) * 2013-02-12 2014-08-14 Pixtronix, Inc. Display Having Staggered Display Element Arrangement
US20150339833A1 (en) * 2014-05-23 2015-11-26 Seiko Epson Corporation Control apparatus, robot, and control method

Family Cites Families (36)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3196889B2 (en) * 1996-09-05 2001-08-06 株式会社アルファ Three-dimensional image processing method and computer-readable recording medium storing a program for causing a computer to execute the three-dimensional image processing method
US8897370B1 (en) * 2009-11-30 2014-11-25 Google Inc. Bitrate video transcoding based on video coding complexity estimation
US8862763B2 (en) * 2011-03-30 2014-10-14 Verizon Patent And Licensing Inc. Downloading video using excess bandwidth
US8810598B2 (en) * 2011-04-08 2014-08-19 Nant Holdings Ip, Llc Interference based augmented reality hosting platforms
GB2501929B (en) * 2012-05-11 2015-06-24 Sony Comp Entertainment Europe Apparatus and method for augmented reality
SG11201500943PA (en) * 2012-08-08 2015-03-30 Univ Singapore System and method for enabling user control of live video stream(s)
US9129429B2 (en) * 2012-10-24 2015-09-08 Exelis, Inc. Augmented reality on wireless mobile devices
GB2509953B (en) * 2013-01-18 2015-05-20 Canon Kk Method of displaying a region of interest in a video stream
CN103702139B (en) * 2013-12-13 2017-02-01 华中科技大学 Video-on-demand system based on scalable coding under mobile environment
US9398250B2 (en) * 2014-01-06 2016-07-19 Arun Sobti & Associates, Llc System and apparatus for smart devices based conferencing
CN105025351B (en) * 2014-04-30 2018-06-29 深圳Tcl新技术有限公司 The method and device of DST PLAYER buffering
US9918136B2 (en) * 2014-05-29 2018-03-13 Nextvr Inc. Methods and apparatus for delivering content and/or playing back content
CN104240281A (en) * 2014-08-28 2014-12-24 东华大学 Virtual reality head-mounted device based on Unity3D engine
CN104268922B (en) * 2014-09-03 2017-06-06 广州博冠信息科技有限公司 A kind of image rendering method and image rendering device
US10812546B2 (en) * 2014-12-24 2020-10-20 Intel IP Corporation Link-aware streaming adaptation
CN104616243B (en) * 2015-01-20 2018-02-27 北京道和汇通科技发展有限公司 A kind of efficient GPU 3 D videos fusion method for drafting
US20160261908A1 (en) * 2015-03-05 2016-09-08 Htc Corporation Media streaming system and control method thereof
CN104735464A (en) * 2015-03-31 2015-06-24 华为技术有限公司 Panorama video interactive transmission method, server and client end
CN104717507A (en) * 2015-03-31 2015-06-17 北京奇艺世纪科技有限公司 Video transcoding method and device
US10083363B2 (en) * 2015-05-26 2018-09-25 Nbcuniversal Media, Llc System and method for customizing content for a user
JP6866297B2 (en) * 2015-06-12 2021-04-28 グーグル エルエルシーGoogle LLC Electronic display stabilization of head-mounted display
US9956054B2 (en) * 2015-06-25 2018-05-01 EchoPixel, Inc. Dynamic minimally invasive surgical-aware assistant
CN108476324B (en) * 2015-10-08 2021-10-29 皇家Kpn公司 Method, computer and medium for enhancing regions of interest in video frames of a video stream
CN106919248A (en) * 2015-12-26 2017-07-04 华为技术有限公司 It is applied to the content transmission method and equipment of virtual reality
CN105916022A (en) * 2015-12-28 2016-08-31 乐视致新电子科技(天津)有限公司 Video image processing method and apparatus based on virtual reality technology
CN105455285B (en) * 2015-12-31 2019-02-12 北京小鸟看看科技有限公司 A kind of virtual implementing helmet adaptation method
US10412130B2 (en) * 2016-04-04 2019-09-10 Hanwha Techwin Co., Ltd. Method and apparatus for playing media stream on web browser
US10313417B2 (en) * 2016-04-18 2019-06-04 Qualcomm Incorporated Methods and systems for auto-zoom based adaptive video streaming
CN105898565A (en) * 2016-04-28 2016-08-24 乐视控股(北京)有限公司 Video processing method and device
US9721393B1 (en) * 2016-04-29 2017-08-01 Immersive Enterprises, LLC Method for processing and delivering virtual reality content to a user
CN106060570B (en) * 2016-06-30 2019-06-14 北京奇艺世纪科技有限公司 A kind of full-view video image plays, coding method and device
CN106060515B (en) * 2016-07-14 2018-11-06 腾讯科技(深圳)有限公司 Panorama pushing method for media files and device
CN106231317A (en) * 2016-09-29 2016-12-14 三星电子(中国)研发中心 Video processing, coding/decoding method and device, VR terminal, audio/video player system
US10595069B2 (en) * 2016-12-05 2020-03-17 Adobe Inc. Prioritizing tile-based virtual reality video streaming using adaptive rate allocation
US20180295375A1 (en) * 2017-04-05 2018-10-11 Lyrical Labs Video Compression Technology, LLC Video processing and encoding
CN107087212B (en) * 2017-05-09 2019-10-29 杭州码全信息科技有限公司 Interactive panoramic video transcoding and playback method and system based on spatial scalable coding

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070292113A1 (en) * 2005-06-30 2007-12-20 Meng-Nan Tsou Video decoding apparatus, video decoding method, and digital audio/video playback system capable of controlling presentation of sub-pictures
US20130044108A1 (en) * 2011-03-31 2013-02-21 Panasonic Corporation Image rendering device, image rendering method, and image rendering program for rendering stereoscopic panoramic images
US20140225816A1 (en) * 2013-02-12 2014-08-14 Pixtronix, Inc. Display Having Staggered Display Element Arrangement
US20150339833A1 (en) * 2014-05-23 2015-11-26 Seiko Epson Corporation Control apparatus, robot, and control method

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10331862B2 (en) * 2017-04-20 2019-06-25 Cisco Technology, Inc. Viewport decryption

Also Published As

Publication number Publication date
US20180192063A1 (en) 2018-07-05
US20180192022A1 (en) 2018-07-05
US20180191868A1 (en) 2018-07-05
CN108377381A (en) 2018-08-07
US20180192026A1 (en) 2018-07-05
CN108419093A (en) 2018-08-17
CN108419142A (en) 2018-08-17
CN108366293A (en) 2018-08-03
US20180189980A1 (en) 2018-07-05
CN108366272A (en) 2018-08-03
US10334238B2 (en) 2019-06-25
CN108391103A (en) 2018-08-10

Similar Documents

Publication Publication Date Title
US20180192044A1 (en) Method and System for Providing A Viewport Division Scheme for Virtual Reality (VR) Video Streaming
WO2017193576A1 (en) Video resolution adaptation method and apparatus, and virtual reality terminal
US10506223B2 (en) Method, apparatus, and device for realizing virtual stereoscopic scene
US9485493B2 (en) Method and system for displaying multi-viewpoint images and non-transitory computer readable storage medium thereof
US20170153866A1 (en) Audiovisual Surround Augmented Reality (ASAR)
US20050052623A1 (en) Projecting system
CN109074678A (en) A kind of processing method and processing device of information
EP4231650A1 (en) Picture display method and apparatus, and electronic device
WO2020248900A1 (en) Panoramic video processing method and apparatus, and storage medium
US20210327093A1 (en) Image processing method of virtual reality and apparatus thereof
US20170150212A1 (en) Method and electronic device for adjusting video
CN111198610A (en) Method, device and equipment for controlling field of view of panoramic video and storage medium
CN109246477B (en) Panoramic video frame interpolation method and device
US20220382053A1 (en) Image processing method and apparatus for head-mounted display device as well as electronic device
CN108401163B (en) Method and device for realizing VR live broadcast and OTT service system
CN111292280B (en) Method and device for outputting information
CN100498840C (en) Method of and scaling unit for scaling a three-dimensional model
CN111200758B (en) Multi-view-field control method and device for panoramic video, electronic equipment and storage medium
CN110662099B (en) Method and device for displaying bullet screen
KR101773929B1 (en) System for processing video with wide viewing angle, methods for transmitting and displaying vide with wide viewing angle and computer programs for the same
WO2022134962A1 (en) Method and apparatus for presenting point cloud window, computer-readable medium, and electronic device
CN111937043A (en) Associating file format objects with dynamic adaptive streaming over hypertext transfer protocol (DASH) objects
CN117173378B (en) CAVE environment-based WebVR panoramic data display method, device, equipment and medium
CN111277886B (en) Panoramic video view field control method and device, electronic equipment and storage medium
CN106998461A (en) One kind wears display system

Legal Events

Date Code Title Description
AS Assignment

Owner name: BLACK SAILS TECHNOLOGY INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:WANG, ZHUO;TANG, YONGTAO;ZHAO, RUOXI;AND OTHERS;REEL/FRAME:044518/0618

Effective date: 20180102

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION