US20180192044A1 - Method and System for Providing A Viewport Division Scheme for Virtual Reality (VR) Video Streaming - Google Patents
Method and System for Providing A Viewport Division Scheme for Virtual Reality (VR) Video Streaming Download PDFInfo
- Publication number
- US20180192044A1 US20180192044A1 US15/860,392 US201815860392A US2018192044A1 US 20180192044 A1 US20180192044 A1 US 20180192044A1 US 201815860392 A US201815860392 A US 201815860392A US 2018192044 A1 US2018192044 A1 US 2018192044A1
- Authority
- US
- United States
- Prior art keywords
- viewport
- block
- blocks
- core
- area
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000013507 mapping Methods 0.000 claims abstract description 33
- 239000011159 matrix material Substances 0.000 claims description 12
- 230000008569 process Effects 0.000 description 14
- 230000005540 biological transmission Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000005094 computer simulation Methods 0.000 description 1
- 230000004438 eyesight Effects 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 210000003128 head Anatomy 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
Images
Classifications
-
- H04N13/0497—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/398—Synchronisation thereof; Control thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/12—Picture reproducers
- H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
- H04N9/3179—Video signal processing therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/012—Head tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/04—Texture mapping
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/10—Geometric effects
- G06T15/20—Perspective computation
- G06T15/205—Image-based rendering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
- G06T9/001—Model-based coding, e.g. wire frame
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
- H04L43/0888—Throughput
-
- H04L65/601—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/75—Media network packet handling
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/75—Media network packet handling
- H04L65/752—Media network packet handling adapting media to network capabilities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/131—Protocols for games, networked simulations or virtual reality
-
- H04N13/0429—
-
- H04N13/0479—
-
- H04N13/0484—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
- H04N13/117—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/122—Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/139—Format conversion, e.g. of frame-rate or size
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/189—Recording image signals; Reproducing recorded image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/275—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
- H04N13/344—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/366—Image reproducers using viewer tracking
- H04N13/378—Image reproducers using viewer tracking for tracking rotational head movements around an axis perpendicular to the screen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/366—Image reproducers using viewer tracking
- H04N13/383—Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/40—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/44—Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/233—Processing of audio elementary streams
- H04N21/2335—Processing of audio elementary streams involving reformatting operations of audio signals, e.g. by converting from one coding standard to another
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234345—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234363—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the spatial resolution, e.g. for clients with a lower screen resolution
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234381—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/238—Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams
- H04N21/2387—Stream processing in response to a playback request from an end-user, e.g. for trick-play
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/239—Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests
- H04N21/2393—Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests involving handling client requests
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
- H04N21/2405—Monitoring of the internal components or processes of the server, e.g. server load
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
- H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
- H04N21/2662—Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/4104—Peripherals receiving signals from specially adapted client devices
- H04N21/4122—Peripherals receiving signals from specially adapted client devices additional display device, e.g. video projector
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44012—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440236—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/845—Structuring of content, e.g. decomposing content into time segments
- H04N21/8456—Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/76—Television signal recording
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/12—Picture reproducers
- H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/12—Picture reproducers
- H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
- H04N9/3179—Video signal processing therefor
- H04N9/3185—Geometric adjustment, e.g. keystone or convergence
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/79—Processing of colour television signals in connection with recording
- H04N9/87—Regeneration of colour television signals
- H04N9/8715—Regeneration of colour television signals involving the mixing of the reproduced video signal with a non-recorded signal, e.g. a text signal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2215/00—Indexing scheme for image rendering
- G06T2215/16—Using real world measurements to influence rendering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
Definitions
- the present disclosure relates to video processing technology, and more particularly, to a method and a system for providing a viewport division scheme for virtual reality (VR) video streaming BACKGROUND OF THE DISCLOSURE
- VR virtual reality
- Virtual Reality is a computer simulation technology for creating and experiencing a virtual world. For example, a three-dimensional real-time image can be presented based on a technology which tracks a user's head, eyes or hands.
- full-view video data can be pre-stored on a server, and then transmitted to a display device.
- a display device can be glasses, a head-mounted display, etc.
- a video is displayed on the display device according to a user's field of view.
- a high-resolution video data occupies large transmission of bandwidth and requires high computing power from the display devices.
- Presenting high-resolution VR video on internet is difficult.
- the existing video streaming technology can not fulfill the virtual reality. Therefore, in order to present VR video smoothly in real-time, it is desirable to further improve the existing video streaming technology to save bandwidth and reduce performance requirements for display devices, by a new way to encode and store the VR video data on the server.
- the present disclosure provides a method and a system for providing a viewport division scheme for virtual reality (VR) video streaming, which determine a viewport block of the VR video data and process the VR video data by different methods for a viewport area and a non-viewport area so as to have a better viewing experience at a player end through less data transmission.
- VR virtual reality
- a method for providing a viewport division scheme for VR video streaming comprising:
- the step of obtaining a projection area of VR video comprises:
- the step of dividing the projection area into a plurality of grid blocks comprises:
- the method further comprises creating identifications for the plurality of grid blocks, the plurality of core blocks and the viewport, and establishing a mapping relationship between the viewport and the plurality of grid blocks by the identifications.
- the step of creating identifications for the plurality of grid blocks, the plurality of core blocks and the viewport, and establishing a mapping relationship between the viewport and the plurality of grid blocks by the identifications comprises:
- width block height video n 2 ( 2 )
- height video represents an image height
- N + represents a positive integer
- id core_block ⁇ id block ⁇ ⁇ % ⁇ ⁇ 2 ⁇ n 2 , id block ⁇ Set core null , others ; ( 4 )
- a viewport block in the core area to have a width determined by a user's field of view and a height equal to that of the core area, where a width width viewport _ block of the viewport block satisfies Equation (5),
- width viewport_block fov * height video ⁇ , ( 5 )
- fov represents the user's field of view
- the identifications of the core blocks in the viewport block can be obtained according to Equation (7),
- f ⁇ ( id viewport_block ) ⁇ ⁇ id viewport_block , id viewport_block + 1 , ... ⁇ , id viewport_block + i - 2 , id viewport_block + i - 1 ⁇ ; id viewport_block ⁇ [ 0 , 2 ⁇ n 2 - i ] ⁇ id viewport_block , id viewport_block + 1 , ... ⁇ , 2 ⁇ n 2 - 1 , 0 , 1 , ... ⁇ , i - ( 2 ⁇ n 2 - id viewport_block ) - 1 ⁇ ; , id viewport_block ⁇ [ 2 ⁇ n 2 - i + 1 , 2 ⁇ n 2 - 1 ] ; ⁇ ( 7 ) ( 7 )
- the identification of the viewport block including the core block can be obtained uniquely according to Equation (8),
- id core _ block _ 1st represents the identification of the first core block that is included in the viewport block.
- the step of dividing the core area into the plurality of core blocks comprises:
- the plurality core blocks each have a height equal to of the central and the core area has an area equal to integral times of the area of a grid block.
- the method further comprises dividing the VR video data into a plurality of sub-data according to the plurality of grid blocks, wherein each sub-data corresponds to one of the grid blocks, and determining a mapping relationship between the plurality of sub-data and the viewport according to the mapping relationship between the plurality of grid blocks and the viewport.
- the method further comprises creating a plurality of viewport blocks according to a user's field of view.
- a system for providing a viewport for VR video streaming comprising:
- a configuration module configured to obtain a projection area of VR video
- a first division module configured to divide the projection area into a plurality of grid blocks
- a second division module configured to define a core area in the projection area
- a third division module configured to divide the core area into a plurality of core blocks
- a combination module configured to combine the core blocks to provide a plurality of viewport blocks
- mapping module configured to establish a mapping relationship between the viewport and the plurality of grid blocks, wherein the plurality of viewport blocks are related to a user's field of view.
- the configuration module prefferably, the configuration module
- the VR source video data is divided into a plurality of grid blocks, and a mapping table between the viewport blocks and the grid blocks is established according to a specific field of view.
- the viewport block is retrieved according to the mapping table, and the video data of a viewport area and a non-viewport area are processed by different methods to have a better view experience.
- FIG. 1 is a diagram illustrating an example the network-based virtual reality playback system
- FIG. 2 is a flowchart presents the process of the playback system in FIG. 1 ;
- FIG. 3 is a flowchart presents the method for providing a viewport division scheme for VR video streaming
- FIG. 4A-4C schematically show a core area, grid blocks and core blocks of the video data.
- FIG. 5 is a block diagram illustrating a system for providing a viewport for VR video streaming.
- FIG. 1 is a diagram illustrating an example network of a VR playback system.
- the VR playback system 10 includes a server 100 and a display device 120 which are coupled with each other through a network 110 , and a VR device.
- the server 100 may be a stand-alone computer server or a server cluster.
- the server 100 is used to store various video data and to store various applications that process these video data.
- various daemons run on the server 100 in real time, so as to process various video data in the server 100 and to respond various requests from VR devices and the display device 120 .
- the network 110 may be a selected one or selected ones from the group consisting of an internet, a local area network, an internet of things, and the like.
- the display device 120 may be any of the computing devices, including a computer device having an independent display screen and a processing capability.
- the display device 120 may be a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a palmtop computer, a personal digital assistant, a smart phone, an intelligent electrical apparatus, a game console, an iPAD/iPhone, a video player, a DVD recorder/player, a television, or a home entertainment system.
- the display device 120 may store VR player software as a VR player. When the VR player is started, it requests and downloads various video data from the server 100 , and renders and plays the video data in the display device.
- the VR device 130 is a stand-alone head-mounted device that can interact with the display device 120 and the server 100 , and transmit information to the display device 120 and/or the server 100 by signal, including change of the position of a user's helmet, change of the eye sight, and the like.
- the display device 120 can process flexibly the video data which is played according to the information.
- the display device 120 determines that a central viewport of the user has been changed, and starts to play video data with a high resolution in the central viewport.
- the VR device 130 is a stand-alone head-mounted device.
- the VR device 130 is not limited thereto, and the VR device 130 may also be an all-in-one head-mounted device.
- the all-in-one head-mounted device itself has a display screen, so that it is not necessary to connect the all-in-one head-mounted device with the external display device.
- the display device 120 may be omitted.
- the all-in-one head-mounted device is configured to obtain video data from the server 100 and to perform playback operation, and the all-in-one head-mounted device is also configured to detect user's field of view and to adjust the playback operation according to the user's field of view.
- FIG. 2 is a flowchart showing a method used in the VR playback system of FIG. 1 . The method includes the following steps.
- step S 10 the server performs source video data processing.
- step S 20 display device side obtains some related information by interacting with the VR device.
- step S 30 according to the related information, the display device side requests the server side to provide the video data and receives the video data.
- step S 40 the display device side renders the received video data.
- the step S 10 is used to process the video data stored on the server side.
- the video data can be divided into those in a viewport are and those in a non-viewport area, and then can be processed by different methods to reduce data transmission for the non-viewport. Therefore, the present disclosure further provides a method for providing a viewport division scheme to achieve the above functions.
- FIG. 3 is a flowchart presents the method for providing a viewport division scheme for VR video streaming. Specifically, the method comprises following steps.
- step S 100 a projection area of VR video is obtained.
- a projection process may be included in a VR video production process and a rendering process to change the VR video data to two-dimensional spatial data.
- the original video data may be subjected to the projection process, for example, an equivalent rectangle matrix process with the aspect ratio of 2:1, to obtain a projection area.
- the projection area can be obtained by decoding the VR video data.
- Two main VR video formats include the video data with the aspect ratio of 1:1 and the aspect ratio of 2:1, which are well-known in the industry as film sources.
- the server needs to perform a normalization process in view of the user's requirements.
- the video data with the aspect ratio of 1:1 is processed into the video data with the aspect ratio of 2:1 which is suitable for the user's VR device.
- the equivalent rectangle matrix projection for example, in an equally-spaced cylindrical projection, the meridians can be rendered to be vertical straight lines having a constant pitch, or the latitude circles can be rendered to be horizontal straight lines having a constant pitch.
- the VR video data here refers to original data and/or intermediate data in any VR processing.
- step S 200 a core area of the projection area is divided.
- the projection area of the VR video data is obtained in the previous step S 100 .
- the core area of the projection area is divided.
- the data after being projected is generally distorted.
- a full-view image may be rendered by an equivalent rectangle matrix projection process to obtain an area with the aspect ratio of 2:1.
- stretching characteristics of the equivalent rectangle matrix projection when an image is close to a pole, the image will be stretched and distorted. The degree of pixel distortion will be higher. Therefore, according to stretching characteristics of the equivalent rectangle matrix projection, a central portion will be defined at the pole.
- step S 300 a block size is defined according to the parameter N.
- the parameter N is a given parameter.
- an area of each grid block is obtained according to the N value. For example, assuming that the normalized video data has an area S and N is 32, each grid block has a block size equal to S/32.
- step S 400 the core area is divided into grid blocks.
- the core area can be divided into M core blocks based on a given value M.
- the core blocks may be numbered for identifying each core block. Generally, the number starts from 0.
- step S 500 the projection area is divided into grid blocks.
- the projection area is divided into a plurality of equal portions, each of which corresponds to a grid block.
- the grid block here is a portion of the VR video data.
- each grid block is numbered. Generally, the number starts from 0.
- step S 600 the core blocks are combined into a viewport block.
- the viewport block contains one or more core blocks.
- the viewport block is also changed. Accordingly, the core blocks in the viewport block are also changed. Therefore, the core blocks may be combined into the viewport block according to the user's field of view.
- step S 700 a mapping table between the viewport blocks and the grid blocks is established.
- the mapping table between the viewport blocks and the grid blocks is established.
- the VR source video data is divided into a plurality of grid blocks, and a viewport block is defined according to a specific field of field of view (fov) to establish the mapping table between the viewport blocks and the grid blocks.
- fov field of field of view
- the viewport block is changed and obtained according to the mapping table.
- the video data is processed by different methods for the viewport area and for the non-viewport area, so as to have a better viewing experience. Further, the user's field of view is proportional to the area of the viewport block.
- Many viewport blocks can be defined for many fields of views (fov).
- the grid blocks, the core blocks and the viewport blocks have identifications, and a mapping relationship between the viewport blocks and the grid blocks is established with the identifications.
- a segmentation step may also be included in which the video data is divided into a plurality of sub-data according to a plurality of grid blocks which are obtained according to the above embodiment.
- Each sub-data corresponds to a grid block. According to the above mapping relationship between the viewport blocks and the grid blocks, one can determine the mapping relationship between the plurality of sub-data and the plurality of viewport blocks.
- mathematical equations can be established based on the identifications of the grid blocks, the core blocks and the viewport blocks, so that the identification of the viewport block can be determined according to the identifications of the grid blocks and/or the core blocks, or the identifications of the core blocks can be determined according to the identification of the viewport block. Details will be described in the following paragraphs.
- each grid unit is a grid block and has the identification id block as follows,
- width block height video n 2 ( 2 )
- height video represents an image height
- N + represents a positive integer
- the core area is defined as a set of grid blocks Set core which satisfy Equation (3),
- a core block is defined to have a width equal to that of the grid blocks and a height equal to that of the core are.
- the identification id core _ block of the core block satisfies Equation (4),
- id core_block ⁇ id block ⁇ ⁇ % ⁇ ⁇ 2 ⁇ n 2 , id block ⁇ Set core null , others ; ( 4 )
- a viewport block is defined as having blocks in the core area, which are determined by a user's field of view fov and have a width equal to that of the core area.
- a width width viewport _ block of the viewport block satisfies Equation (5),
- width viewport_block fov * height video ⁇ , ( 5 )
- the number of the core blocks in the viewport block is defined as i, which satisfies equation (6),
- the identifications of the core blocks in the viewport block can be obtained according to Equation (7),
- f ⁇ ( id viewport_block ) ⁇ ⁇ id viewport_block , id viewport_block + 1 , ... ⁇ , id viewport_block + i - 2 , id viewport_block + i - 1 ⁇ ; id viewport_block ⁇ [ 0 , 2 ⁇ n 2 - i ] ⁇ id viewport_block , id viewport_block + 1 , ... ⁇ , 2 ⁇ n 2 - 1 , 0 , 1 , ... ⁇ , i - ( 2 ⁇ n 2 - id viewport_block ) - 1 ⁇ ; , id viewport_block ⁇ [ 2 ⁇ n 2 - i + 1 , 2 ⁇ n 2 - 1 ] ; ⁇ ( 7 ) ( 7 )
- the identification of the viewport block including the core block which represents the identification of the first core block, can be obtained uniquely according to Equation (8),
- the identification satisfies Equation (8),
- id core _ block _ 1st represents the identification of the first core block that is included in the viewport block.
- FIG. 4A-4C schematically show a core area, grid blocks and core blocks of the video data.
- 400 represents a projection area
- 401 represents a core area
- N is equal to 32. That is to say, the projection area is divided into 32 grid blocks, which are numbered as 0-31.
- the core blocks are numbered as 0-7.
- one viewport block (not shown) contains two core blocks which are numbered as 1 and 2, the viewport block contains those grid blocks being numbered as 9-10 and 17-18.
- the identifications of the grid blocks, the core blocks and the viewport blocks may be numbered and be verified by the above equations.
- this example is given only for the mapping relationship among the grid blocks, the core blocks and the viewport blocks under ideal conditions. That is to say, in this example, the core blocks and the grid blocks and the viewport blocks overlap each other. However, the present disclosure is not limited to this and can be applied in a case that the grid blocks, the core blocks and the viewport blocks partially overlap each other.
- FIG. 5 is a block diagram illustrating a system for providing a viewport for VR video streaming.
- the system includes a configuration module 501 , a first division module 503 , a second division module 502 , a third division module 504 , a combination module 505 , and a mapping module 506 .
- the configuration module 501 is configured to obtain a projection area of VR video.
- the VR video data can decoded to obtain the projection area of the video data from a parameter.
- the projection area can be obtained by projecting the VR video into a rectangle by an equivalent rectangle matrix projection process.
- Two main VR video formats include the video data with the aspect ratio of 1:1 and the aspect ratio of 2:1, which are well-known in the industry as film sources.
- the original VR video format may be normalized to have the aspect ratio of 2:1, and then further processed.
- the first division module 503 is configured to divide the projection area into N grid blocks. For example, the projection area is divided into 32 equal portions. As a result, the 32 grid blocks each are a portion of the projection area with equal size.
- the second division module 502 is configured to divide a core area of the projection area according to the user's field of view.
- the core area is defined in the projection area. For example, for a rectangle, a central portion in the X-axis direction or the Y-axis direction is defined as the core area.
- the third division module 504 is configured to divide the core area into a plurality of core blocks.
- the core area is divided into a plurality of grid units, generally having equal area, but is not limited thereto. For example, if an area of an edge unit is insufficient, the area of the edge unit will be smaller.
- the combination module 505 is configured to combine the core blocks into a plurality of viewport blocks.
- the core blocks are combined according to a predetermined rule. For example, two adjacent core blocks are combined into one viewport block. In another example, a core block at a central point and those core blocks surrounding the central point are combined into one viewport block.
- the mapping module 506 is configured to establish a mapping relationship between the viewport blocks and the grid blocks.
- Each viewport block contains a plurality of core blocks.
- Each core block may are mixed with one or more grid blocks.
- one or more grid blocks may be retrieved according to the core blocks.
- the system can locate the viewport block and improve viewing experience effect by optimizing the video data of the viewport block.
- the above system for providing a viewport for VR video streaming may further comprise a segmentation module (not shown).
- the segmentation module divides the video data into a plurality of sub-data according to the plurality of grid blocks. Each sub-data corresponds to a grid.
- the mapping relationship between the plurality of sub-data and the plurality of viewport blocks can be determined by the mapping relationship between the plurality of viewport blocks and the plurality of grid blocks.
- the configuration module projects the VR video into a rectangle with the aspect ratio of 2:1 by equivalent rectangle matrix to obtain the projection area.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computer Graphics (AREA)
- Human Computer Interaction (AREA)
- Geometry (AREA)
- Environmental & Geological Engineering (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Controls And Circuits For Display Device (AREA)
- Image Generation (AREA)
- Processing Or Creating Images (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
- Image Processing (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
A method and a system for providing a viewport division scheme for VR video streaming are disclosed. A projection area of VR video is obtained. The projection area is divided into a plurality of grid blocks. A core area is defined in the projection area and divided into a plurality of core blocks. A plurality of viewport blocks are obtained by combining the core blocks. A mapping relationship between the plurality of viewport blocks and the grid blocks is established. The plurality of viewport blocks are related to a user's field of view. The plurality of viewport blocks are obtained in accordance with a mapping table. The video data is processed by different methods for a viewport area and a non-viewport area to have a better viewing experience.
Description
- This application claims the priority and benefit of U.S. provisional application 62/441,936, filed on Jan. 3, 2017, which is incorporated herein by reference in its entirety.
- The present disclosure relates to video processing technology, and more particularly, to a method and a system for providing a viewport division scheme for virtual reality (VR) video streaming BACKGROUND OF THE DISCLOSURE
- Virtual Reality (VR) is a computer simulation technology for creating and experiencing a virtual world. For example, a three-dimensional real-time image can be presented based on a technology which tracks a user's head, eyes or hands. In the network-based virtual reality technology, full-view video data can be pre-stored on a server, and then transmitted to a display device. A display device can be glasses, a head-mounted display, etc. A video is displayed on the display device according to a user's field of view.
- However, a high-resolution video data occupies large transmission of bandwidth and requires high computing power from the display devices. Presenting high-resolution VR video on internet is difficult. Precisely, the existing video streaming technology can not fulfill the virtual reality. Therefore, in order to present VR video smoothly in real-time, it is desirable to further improve the existing video streaming technology to save bandwidth and reduce performance requirements for display devices, by a new way to encode and store the VR video data on the server.
- It is a problem in the art to play back video data on a player end with a better viewing experience through less video data transmission.
- In view of this, the present disclosure provides a method and a system for providing a viewport division scheme for virtual reality (VR) video streaming, which determine a viewport block of the VR video data and process the VR video data by different methods for a viewport area and a non-viewport area so as to have a better viewing experience at a player end through less data transmission.
- According to one aspect of the present disclosure, there is provided a method for providing a viewport division scheme for VR video streaming, comprising:
- obtaining a projection area of VR video;
- dividing the projection area into a plurality of grid blocks;
- defining a core area in the projection area;
- dividing the core area into a plurality of core blocks;
- combining the core blocks to provide a plurality of viewport blocks; and
- establishing a mapping relation between the plurality of viewport blocks and the plurality of grid blocks, wherein the plurality of viewport blocks are related to a user's field of view.
- Preferably, the step of obtaining a projection area of VR video comprises:
- projecting the VR video into a rectangle with the aspect ratio of 2:1 by equivalent rectangle matrix to obtain the projection area.
- Preferably, the step of dividing the projection area into a plurality of grid blocks comprises:
- dividing the projection area into a plurality of equal portions; and
- dividing each of the plurality of equal portions into the same number of grid blocks.
- Preferably, the method further comprises creating identifications for the plurality of grid blocks, the plurality of core blocks and the viewport, and establishing a mapping relationship between the viewport and the plurality of grid blocks by the identifications.
- Preferably, the step of creating identifications for the plurality of grid blocks, the plurality of core blocks and the viewport, and establishing a mapping relationship between the viewport and the plurality of grid blocks by the identifications comprises:
- setting the identificationidblock of a grid block as a positive integer started from zero, so as to satisfy Equation (1)
-
idblockϵ[0,n−1],idblock ϵN + (1); - setting an edge length of the grid block to satisfy Equation (2),
-
- where heightvideo represents an image height, and N+ represents a positive integer;
- setting the core area to be a set of grid blocks Setcore which satisfy Equation (3),
-
- defining a core block to have a width equal to that of the plurality of grid blocks and a height equal to that of the core area, and setting the identification idcore _ block of the core block, so as to satisfy Equation (4),
-
- defining a viewport block in the core area to have a width determined by a user's field of view and a height equal to that of the core area, where a width widthviewport _ block of the viewport block satisfies Equation (5),
-
- where fov represents the user's field of view;
- defining the number of the core blocks in the viewport block to be i, which satisfies equation (6),
-
- for any identification of the viewport block, the identifications of the core blocks in the viewport block can be obtained according to Equation (7),
-
- for any identification of the core block, the identification of the viewport block including the core block can be obtained uniquely according to Equation (8),
-
idviewport _ block =f(idcore _ block)=idcore _ block _ 1st (8), - where idcore _ block _ 1st represents the identification of the first core block that is included in the viewport block.
- Preferably, the step of dividing the core area into the plurality of core blocks comprises:
- dividing the core area into the plurality of core blocks with equal areas.
- Preferably, the plurality core blocks each have a height equal to of the central and the core area has an area equal to integral times of the area of a grid block.
- Preferably, the method further comprises dividing the VR video data into a plurality of sub-data according to the plurality of grid blocks, wherein each sub-data corresponds to one of the grid blocks, and determining a mapping relationship between the plurality of sub-data and the viewport according to the mapping relationship between the plurality of grid blocks and the viewport.
- Preferably, the method further comprises creating a plurality of viewport blocks according to a user's field of view.
- According to another aspect of the present disclosure, there is provided a system for providing a viewport for VR video streaming, comprising:
- a configuration module, configured to obtain a projection area of VR video;
- a first division module, configured to divide the projection area into a plurality of grid blocks;
- a second division module, configured to define a core area in the projection area;
- a third division module, configured to divide the core area into a plurality of core blocks;
- a combination module, configured to combine the core blocks to provide a plurality of viewport blocks;
- a mapping module, configured to establish a mapping relationship between the viewport and the plurality of grid blocks, wherein the plurality of viewport blocks are related to a user's field of view.
- Preferably, the configuration module
- projects the VR video into a rectangle with the aspect ratio of 2:1 by equivalent rectangle matrix to obtain the projection area.
- In this embodiment, the VR source video data is divided into a plurality of grid blocks, and a mapping table between the viewport blocks and the grid blocks is established according to a specific field of view. When the user's field of view is changed, the viewport block is retrieved according to the mapping table, and the video data of a viewport area and a non-viewport area are processed by different methods to have a better view experience.
- The above and other objects, advantages and features of the present disclosure will become more fully understood from the detailed description given hereinbelow in connection reference with the appended drawings, and wherein:
-
FIG. 1 is a diagram illustrating an example the network-based virtual reality playback system; -
FIG. 2 is a flowchart presents the process of the playback system inFIG. 1 ; -
FIG. 3 is a flowchart presents the method for providing a viewport division scheme for VR video streaming; -
FIG. 4A-4C schematically show a core area, grid blocks and core blocks of the video data; and -
FIG. 5 is a block diagram illustrating a system for providing a viewport for VR video streaming. - Exemplary embodiments of the present disclosure will be described in more details below with reference to the accompanying drawings. In the drawings, like reference numerals denote like members. The figures are not drawn to scale, for the sake of clarity. Moreover, some well known parts may not be shown.
-
FIG. 1 is a diagram illustrating an example network of a VR playback system. TheVR playback system 10 includes aserver 100 and adisplay device 120 which are coupled with each other through anetwork 110, and a VR device. For example, theserver 100 may be a stand-alone computer server or a server cluster. Theserver 100 is used to store various video data and to store various applications that process these video data. For example, various daemons run on theserver 100 in real time, so as to process various video data in theserver 100 and to respond various requests from VR devices and thedisplay device 120. Thenetwork 110 may be a selected one or selected ones from the group consisting of an internet, a local area network, an internet of things, and the like. For example, thedisplay device 120 may be any of the computing devices, including a computer device having an independent display screen and a processing capability. Thedisplay device 120 may be a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a palmtop computer, a personal digital assistant, a smart phone, an intelligent electrical apparatus, a game console, an iPAD/iPhone, a video player, a DVD recorder/player, a television, or a home entertainment system. Thedisplay device 120 may store VR player software as a VR player. When the VR player is started, it requests and downloads various video data from theserver 100, and renders and plays the video data in the display device. In this example, theVR device 130 is a stand-alone head-mounted device that can interact with thedisplay device 120 and theserver 100, and transmit information to thedisplay device 120 and/or theserver 100 by signal, including change of the position of a user's helmet, change of the eye sight, and the like. Thedisplay device 120 can process flexibly the video data which is played according to the information. In some embodiments, when a user's position is changed, thedisplay device 120 determines that a central viewport of the user has been changed, and starts to play video data with a high resolution in the central viewport. - In the above embodiment, the
VR device 130 is a stand-alone head-mounted device. However, those skilled in the art should understand that theVR device 130 is not limited thereto, and theVR device 130 may also be an all-in-one head-mounted device. The all-in-one head-mounted device itself has a display screen, so that it is not necessary to connect the all-in-one head-mounted device with the external display device. For example, in this example, if the all-in-one head-mounted device is used as the VR device, thedisplay device 120 may be omitted. At this point, the all-in-one head-mounted device is configured to obtain video data from theserver 100 and to perform playback operation, and the all-in-one head-mounted device is also configured to detect user's field of view and to adjust the playback operation according to the user's field of view. -
FIG. 2 is a flowchart showing a method used in the VR playback system ofFIG. 1 . The method includes the following steps. - In step S10, the server performs source video data processing.
- In step S20, display device side obtains some related information by interacting with the VR device.
- In step S30, according to the related information, the display device side requests the server side to provide the video data and receives the video data.
- In step S40, the display device side renders the received video data.
- Wherein, the step S10 is used to process the video data stored on the server side. In this step, the video data can be divided into those in a viewport are and those in a non-viewport area, and then can be processed by different methods to reduce data transmission for the non-viewport. Therefore, the present disclosure further provides a method for providing a viewport division scheme to achieve the above functions.
-
FIG. 3 is a flowchart presents the method for providing a viewport division scheme for VR video streaming. Specifically, the method comprises following steps. - In step S100, a projection area of VR video is obtained.
- Although the VR video data is captured in a 360-degree panoramic view of the real-world environment, a projection process may be included in a VR video production process and a rendering process to change the VR video data to two-dimensional spatial data. The original video data may be subjected to the projection process, for example, an equivalent rectangle matrix process with the aspect ratio of 2:1, to obtain a projection area. In a case that the VR video data has been subjected to the projection process, the projection area can be obtained by decoding the VR video data. Two main VR video formats include the video data with the aspect ratio of 1:1 and the aspect ratio of 2:1, which are well-known in the industry as film sources. In some cases, the server needs to perform a normalization process in view of the user's requirements. For example, the video data with the aspect ratio of 1:1 is processed into the video data with the aspect ratio of 2:1 which is suitable for the user's VR device. In the equivalent rectangle matrix projection, for example, in an equally-spaced cylindrical projection, the meridians can be rendered to be vertical straight lines having a constant pitch, or the latitude circles can be rendered to be horizontal straight lines having a constant pitch. It should be noted that the VR video data here refers to original data and/or intermediate data in any VR processing.
- In step S200, a core area of the projection area is divided.
- As described above, the projection area of the VR video data is obtained in the previous step S100. In this step, the core area of the projection area is divided. The data after being projected is generally distorted. For example, a full-view image may be rendered by an equivalent rectangle matrix projection process to obtain an area with the aspect ratio of 2:1. According to stretching characteristics of the equivalent rectangle matrix projection, when an image is close to a pole, the image will be stretched and distorted. The degree of pixel distortion will be higher. Therefore, according to stretching characteristics of the equivalent rectangle matrix projection, a central portion will be defined at the pole.
- In step S300, a block size is defined according to the parameter N.
- The parameter N is a given parameter. In this step, an area of each grid block is obtained according to the N value. For example, assuming that the normalized video data has an area S and N is 32, each grid block has a block size equal to S/32.
- In step S400, the core area is divided into grid blocks.
- After the core area is determined, the core area can be divided into M core blocks based on a given value M. The core blocks may be numbered for identifying each core block. Generally, the number starts from 0.
- In step S500, the projection area is divided into grid blocks.
- In this step, the projection area is divided into a plurality of equal portions, each of which corresponds to a grid block. It should be understood that the grid block here is a portion of the VR video data. After that, each grid block is numbered. Generally, the number starts from 0.
- In step S600, the core blocks are combined into a viewport block.
- The viewport block contains one or more core blocks. When a field of view of the VR device is changed, the viewport block is also changed. Accordingly, the core blocks in the viewport block are also changed. Therefore, the core blocks may be combined into the viewport block according to the user's field of view.
- In step S700, a mapping table between the viewport blocks and the grid blocks is established.
- Based on the viewport block of each VR video data, which is obtained in step S600, the mapping table between the viewport blocks and the grid blocks is established. In this embodiment, the VR source video data is divided into a plurality of grid blocks, and a viewport block is defined according to a specific field of field of view (fov) to establish the mapping table between the viewport blocks and the grid blocks. When the user's field of view is changed, the viewport block is changed and obtained according to the mapping table. The video data is processed by different methods for the viewport area and for the non-viewport area, so as to have a better viewing experience. Further, the user's field of view is proportional to the area of the viewport block. Many viewport blocks can be defined for many fields of views (fov). In an alternative embodiment, the grid blocks, the core blocks and the viewport blocks have identifications, and a mapping relationship between the viewport blocks and the grid blocks is established with the identifications.
- It should be noted that in the above embodiments, the video data is not actually segmented. Therefore, in an optional embodiment of the present disclosure, a segmentation step may also be included in which the video data is divided into a plurality of sub-data according to a plurality of grid blocks which are obtained according to the above embodiment. Each sub-data corresponds to a grid block. According to the above mapping relationship between the viewport blocks and the grid blocks, one can determine the mapping relationship between the plurality of sub-data and the plurality of viewport blocks.
- Under some limitations, mathematical equations can be established based on the identifications of the grid blocks, the core blocks and the viewport blocks, so that the identification of the viewport block can be determined according to the identifications of the grid blocks and/or the core blocks, or the identifications of the core blocks can be determined according to the identification of the viewport block. Details will be described in the following paragraphs.
- Assuming that an image of the video data has the aspect ratio of 2:1 and is divided into n grid units with the aspect ratio of 1:1, and each grid unit is a grid block and has the identification idblock as follows,
-
idblockϵ[0,n−1],idblock ϵN + (1); - An edge length of the grid block satisfies Equation (1).
-
- where heightvideo represents an image height, and N+ represents a positive integer;
- According characteristics of the equivalent rectangle matrix projection, the core area is defined as a set of grid blocks Setcore which satisfy Equation (3),
-
- A core block is defined to have a width equal to that of the grid blocks and a height equal to that of the core are. The identification idcore _ block of the core block satisfies Equation (4),
-
- A viewport block is defined as having blocks in the core area, which are determined by a user's field of view fov and have a width equal to that of the core area. A width widthviewport _ block of the viewport block satisfies Equation (5),
-
- The number of the core blocks in the viewport block is defined as i, which satisfies equation (6),
-
- For any identification of the viewport block, the identifications of the core blocks in the viewport block can be obtained according to Equation (7),
-
- For any identification of the core block, the identification of the viewport block including the core block, which represents the identification of the first core block, can be obtained uniquely according to Equation (8), The identification satisfies Equation (8),
-
idviewport _ block =f(idcore _ block)=idcore _ block _ 1st (8), - where idcore _ block _ 1st represents the identification of the first core block that is included in the viewport block.
-
FIG. 4A-4C schematically show a core area, grid blocks and core blocks of the video data. - In this example, 400 represents a projection area, 401 represents a core area, and N is equal to 32. That is to say, the projection area is divided into 32 grid blocks, which are numbered as 0-31. The core blocks are numbered as 0-7. Assuming that one viewport block (not shown) contains two core blocks which are numbered as 1 and 2, the viewport block contains those grid blocks being numbered as 9-10 and 17-18. Thus, it is possible to establish a mapping relationship between the viewport blocks and the grid blocks. In this case, the identifications of the grid blocks, the core blocks and the viewport blocks may be numbered and be verified by the above equations.
- It should be understood that this example is given only for the mapping relationship among the grid blocks, the core blocks and the viewport blocks under ideal conditions. That is to say, in this example, the core blocks and the grid blocks and the viewport blocks overlap each other. However, the present disclosure is not limited to this and can be applied in a case that the grid blocks, the core blocks and the viewport blocks partially overlap each other.
-
FIG. 5 is a block diagram illustrating a system for providing a viewport for VR video streaming. - The system includes a
configuration module 501, afirst division module 503, asecond division module 502, athird division module 504, acombination module 505, and amapping module 506. - The
configuration module 501 is configured to obtain a projection area of VR video. In a case that the VR video data has been processed into two-dimensional data, the VR video data can decoded to obtain the projection area of the video data from a parameter. In another case that the video data is not processed into two-dimensional data, the projection area can be obtained by projecting the VR video into a rectangle by an equivalent rectangle matrix projection process. Two main VR video formats include the video data with the aspect ratio of 1:1 and the aspect ratio of 2:1, which are well-known in the industry as film sources. The original VR video format may be normalized to have the aspect ratio of 2:1, and then further processed. - The
first division module 503 is configured to divide the projection area into N grid blocks. For example, the projection area is divided into 32 equal portions. As a result, the 32 grid blocks each are a portion of the projection area with equal size. - The
second division module 502 is configured to divide a core area of the projection area according to the user's field of view. The core area is defined in the projection area. For example, for a rectangle, a central portion in the X-axis direction or the Y-axis direction is defined as the core area. - The
third division module 504 is configured to divide the core area into a plurality of core blocks. The core area is divided into a plurality of grid units, generally having equal area, but is not limited thereto. For example, if an area of an edge unit is insufficient, the area of the edge unit will be smaller. - The
combination module 505 is configured to combine the core blocks into a plurality of viewport blocks. The core blocks are combined according to a predetermined rule. For example, two adjacent core blocks are combined into one viewport block. In another example, a core block at a central point and those core blocks surrounding the central point are combined into one viewport block. - The
mapping module 506 is configured to establish a mapping relationship between the viewport blocks and the grid blocks. Each viewport block contains a plurality of core blocks. Each core block may are mixed with one or more grid blocks. Correspondingly, one or more grid blocks may be retrieved according to the core blocks. - By establishing the mapping relationship between the viewport block and the grid blocks, the system can locate the viewport block and improve viewing experience effect by optimizing the video data of the viewport block.
- Preferably, the above system for providing a viewport for VR video streaming may further comprise a segmentation module (not shown). The segmentation module divides the video data into a plurality of sub-data according to the plurality of grid blocks. Each sub-data corresponds to a grid. The mapping relationship between the plurality of sub-data and the plurality of viewport blocks can be determined by the mapping relationship between the plurality of viewport blocks and the plurality of grid blocks.
- The configuration module projects the VR video into a rectangle with the aspect ratio of 2:1 by equivalent rectangle matrix to obtain the projection area.
- The embodiments were chosen and described in order to best explain the principles of the disclosure and its practical applications, to thereby enable others skilled in the art to best utilize the disclosure. The disclosure is intended to cover alternatives, modifications and equivalents that may be included within the spirit and scope of the disclosure as defined by the appended claims.
- The foregoing descriptions of specific embodiments of the present disclosure have been presented, but are not intended to limit the disclosure to the precise forms disclosed. It will be readily apparent to one skilled in the art that many modifications and changes may be made in the present disclosure. Any modifications, equivalence, variations of the preferred embodiments can be made without departing from the doctrine and spirit of the present disclosure.
Claims (11)
1. A method for providing a viewport division scheme for VR video streaming, comprising:
obtaining a projection area of VR video;
dividing the projection area into a plurality of grid blocks;
defining a core area in the projection area;
dividing the core area into a plurality of core blocks;
obtaining a plurality of viewport blocks by combining the plurality of core blocks; and
establishing a mapping relation between the plurality of viewport blocks and the plurality of grid blocks, wherein the plurality of viewport blocks is related to a user's field of view.
2. The method according to claim 1 , wherein the step of obtaining the projection area of VR video comprises:
projecting the VR video into a rectangle with the aspect ratio of 2:1 by equivalent rectangle matrix to obtain the projection area.
3. The method according to claim 1 , wherein the step of dividing the projection area into the plurality of grid blocks comprises:
dividing the projection area into a plurality of equal portions.
4. The method according to claim 1 , further comprising: creating identifications for the plurality of grid blocks, the plurality of core blocks and the plurality of viewport blocks, and establishing a mapping relationship between the plurality of viewport blocks and the plurality of grid blocks by the identifications.
5. The method according to claim 4 , wherein the step of creating identifications for the plurality of grid blocks, the plurality of core blocks and the viewport, and establishing the mapping relationship between the plurality of viewport blocks and the plurality of grid blocks by the identifications comprises:
setting the identification idblock of a grid block as a positive integer started from zero, so as to satisfy Equation (1)
idblockϵ[0,n−1],idblock ϵN + (1);
idblockϵ[0,n−1],idblock ϵN + (1);
setting an edge length of the grid block to satisfy Equation (2),
where heightvideo represents an image height, and N+ represents a positive integer;
setting the core area to be a set of grid blocks Setcore which satisfy Equation (3),
defining a core block to have a width equal to that of the grid blocks and a height equal to that of the core area, and setting the identification idcore _ block of the core block, so as to satisfy Equation (4),
defining a viewport block in the core area to have a width determined by a user's field of view and a height equal to that of the core area, where a width widthviewport _ block of the viewport block satisfies Equation (5),
where fov represents the user's field of view;
defining the number of the core blocks in the viewport block to be i, which satisfies equation (6),
for any identification of the viewport block, the identifications of the core blocks in the viewport block can be obtained according to Equation (7),
for any identification of the core block, the identification of the viewport block can be obtained uniquely according to Equation (8),
idviewport _ block =f(idcore _ block)=idcore _ block _ 1st (8),
idviewport _ block =f(idcore _ block)=idcore _ block _ 1st (8),
where idcore _ block _ 1st represents the identification of the first core block that is included in the viewport block.
6. The method according to claim 1 , wherein the step of dividing the core area into the plurality of core blocks comprises:
dividing the core area into the plurality of core blocks with equal areas.
7. The method according to claim 1 , wherein the plurality of core blocks each have a height equal to that of the core area, and the core area has an area equal to an integral times that of the area of a grid block.
8. The method according to claim 1 , further comprising: dividing the VR video data into a plurality of sub-data according to the plurality of grid blocks, wherein each sub-data corresponds to one of the grid blocks, and determining a mapping relationship between the plurality of sub-data and the viewport according to the mapping relationship between the plurality of grid blocks and the viewport.
9. The method according to claim 1 , further comprising: creating a plurality of viewport blocks according to a user's field of view.
10. A system for providing a viewport division scheme for VR video streaming, comprising:
a configuration module, configured to obtain a projection area of VR video;
a first division module, configured to divide the projection area into a plurality of grid blocks;
a second division module, configured to define a core area in the projection area;
a third division module, configured to divide the core area into a plurality of core blocks;
a combination module, configured to combine the core blocks to provide a plurality of viewport blocks; and
a mapping module, configured to establish a mapping relationship between the plurality of viewport blocks and the grid blocks, wherein the plurality of viewport blocks are related to a user's field of view.
11. The system according to claim 1 , wherein the configuration module projects the VR video into a rectangle with the aspect ratio of 2:1 by equivalent rectangle matrix to obtain the projection area.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/860,392 US20180192044A1 (en) | 2017-01-03 | 2018-01-02 | Method and System for Providing A Viewport Division Scheme for Virtual Reality (VR) Video Streaming |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201762441936P | 2017-01-03 | 2017-01-03 | |
US15/860,392 US20180192044A1 (en) | 2017-01-03 | 2018-01-02 | Method and System for Providing A Viewport Division Scheme for Virtual Reality (VR) Video Streaming |
Publications (1)
Publication Number | Publication Date |
---|---|
US20180192044A1 true US20180192044A1 (en) | 2018-07-05 |
Family
ID=62711388
Family Applications (6)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/860,449 Expired - Fee Related US10334238B2 (en) | 2017-01-03 | 2018-01-02 | Method and system for real-time rendering displaying high resolution virtual reality (VR) video |
US15/860,392 Abandoned US20180192044A1 (en) | 2017-01-03 | 2018-01-02 | Method and System for Providing A Viewport Division Scheme for Virtual Reality (VR) Video Streaming |
US15/860,494 Abandoned US20180189980A1 (en) | 2017-01-03 | 2018-01-02 | Method and System for Providing Virtual Reality (VR) Video Transcoding and Broadcasting |
US15/860,358 Abandoned US20180192063A1 (en) | 2017-01-03 | 2018-01-02 | Method and System for Virtual Reality (VR) Video Transcode By Extracting Residual From Different Resolutions |
US15/860,471 Abandoned US20180192022A1 (en) | 2017-01-03 | 2018-01-02 | Method and System for Real-time Rendering Displaying Virtual Reality (VR) On Mobile Using Head-Up Display Devices |
US15/860,430 Abandoned US20180191868A1 (en) | 2017-01-03 | 2018-01-02 | Method and System for Downloading Multiple Resolutions Bitrate for Virtual Reality (VR) Video Streaming Optimization |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/860,449 Expired - Fee Related US10334238B2 (en) | 2017-01-03 | 2018-01-02 | Method and system for real-time rendering displaying high resolution virtual reality (VR) video |
Family Applications After (4)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/860,494 Abandoned US20180189980A1 (en) | 2017-01-03 | 2018-01-02 | Method and System for Providing Virtual Reality (VR) Video Transcoding and Broadcasting |
US15/860,358 Abandoned US20180192063A1 (en) | 2017-01-03 | 2018-01-02 | Method and System for Virtual Reality (VR) Video Transcode By Extracting Residual From Different Resolutions |
US15/860,471 Abandoned US20180192022A1 (en) | 2017-01-03 | 2018-01-02 | Method and System for Real-time Rendering Displaying Virtual Reality (VR) On Mobile Using Head-Up Display Devices |
US15/860,430 Abandoned US20180191868A1 (en) | 2017-01-03 | 2018-01-02 | Method and System for Downloading Multiple Resolutions Bitrate for Virtual Reality (VR) Video Streaming Optimization |
Country Status (2)
Country | Link |
---|---|
US (6) | US10334238B2 (en) |
CN (6) | CN108419142A (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10331862B2 (en) * | 2017-04-20 | 2019-06-25 | Cisco Technology, Inc. | Viewport decryption |
Families Citing this family (26)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10291910B2 (en) * | 2016-02-12 | 2019-05-14 | Gopro, Inc. | Systems and methods for spatially adaptive video encoding |
CN115842907A (en) * | 2018-03-27 | 2023-03-24 | 京东方科技集团股份有限公司 | Rendering method, computer product and display device |
US11232532B2 (en) * | 2018-05-30 | 2022-01-25 | Sony Interactive Entertainment LLC | Multi-server cloud virtual reality (VR) streaming |
US10623791B2 (en) | 2018-06-01 | 2020-04-14 | At&T Intellectual Property I, L.P. | Field of view prediction in live panoramic video streaming |
US10812774B2 (en) | 2018-06-06 | 2020-10-20 | At&T Intellectual Property I, L.P. | Methods and devices for adapting the rate of video content streaming |
US10616621B2 (en) * | 2018-06-29 | 2020-04-07 | At&T Intellectual Property I, L.P. | Methods and devices for determining multipath routing for panoramic video content |
US11019361B2 (en) | 2018-08-13 | 2021-05-25 | At&T Intellectual Property I, L.P. | Methods, systems and devices for adjusting panoramic view of a camera for capturing video content |
CN109343518B (en) * | 2018-09-03 | 2021-07-02 | 浙江大丰实业股份有限公司 | On-spot drive platform of universal ride |
US11128869B1 (en) * | 2018-10-22 | 2021-09-21 | Bitmovin, Inc. | Video encoding based on customized bitrate table |
CN109375369B (en) * | 2018-11-23 | 2021-05-18 | 国网天津市电力公司 | Distortion preprocessing method in VR (virtual reality) large-screen cinema mode |
CN111510777B (en) * | 2019-01-30 | 2021-11-23 | 上海哔哩哔哩科技有限公司 | Method and device for measuring network speed, computer equipment and readable storage medium |
CN111669666A (en) * | 2019-03-08 | 2020-09-15 | 北京京东尚科信息技术有限公司 | Method, device and system for simulating reality |
CN111866485A (en) * | 2019-04-25 | 2020-10-30 | 中国移动通信有限公司研究院 | Stereoscopic picture projection and transmission method, device and computer readable storage medium |
CN110336994B (en) * | 2019-07-04 | 2021-06-22 | 上海索倍信息科技有限公司 | Naked eye 3D display system |
CN110381331A (en) * | 2019-07-23 | 2019-10-25 | 深圳市道通智能航空技术有限公司 | A kind of image processing method, device, equipment of taking photo by plane and storage medium |
CN110490962B (en) * | 2019-08-20 | 2023-09-15 | 武汉邦拓信息科技有限公司 | Remote rendering method based on video stream |
CN110544425A (en) * | 2019-09-13 | 2019-12-06 | 广州城市职业学院 | ancient building VR display system |
CN111489428B (en) * | 2020-04-20 | 2023-06-30 | 北京字节跳动网络技术有限公司 | Image generation method, device, electronic equipment and computer readable storage medium |
US11245911B1 (en) * | 2020-05-12 | 2022-02-08 | Whirlwind 3D, LLC | Video encoder/decoder (codec) for real-time applications and size/b and width reduction |
CN111754614A (en) * | 2020-06-30 | 2020-10-09 | 平安国际智慧城市科技股份有限公司 | Video rendering method and device based on VR (virtual reality), electronic equipment and storage medium |
CN112468806B (en) * | 2020-11-12 | 2022-07-26 | 中山大学 | Panoramic video transmission optimization method for cloud VR platform |
CN114286142B (en) * | 2021-01-18 | 2023-03-28 | 海信视像科技股份有限公司 | Virtual reality equipment and VR scene screen capturing method |
CN113347402A (en) * | 2021-06-28 | 2021-09-03 | 筑友建筑装饰装修工程有限公司 | Improved method, device and storage medium for rendering immersive content based on Unity |
CN114466220A (en) * | 2022-01-29 | 2022-05-10 | 维沃移动通信有限公司 | Video downloading method and electronic equipment |
CN115396731A (en) * | 2022-08-10 | 2022-11-25 | 北京势也网络技术有限公司 | Panoramic video playing method and device, electronic equipment and readable storage medium |
CN116880723B (en) * | 2023-09-08 | 2023-11-17 | 江西格如灵科技股份有限公司 | 3D scene display method and system |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070292113A1 (en) * | 2005-06-30 | 2007-12-20 | Meng-Nan Tsou | Video decoding apparatus, video decoding method, and digital audio/video playback system capable of controlling presentation of sub-pictures |
US20130044108A1 (en) * | 2011-03-31 | 2013-02-21 | Panasonic Corporation | Image rendering device, image rendering method, and image rendering program for rendering stereoscopic panoramic images |
US20140225816A1 (en) * | 2013-02-12 | 2014-08-14 | Pixtronix, Inc. | Display Having Staggered Display Element Arrangement |
US20150339833A1 (en) * | 2014-05-23 | 2015-11-26 | Seiko Epson Corporation | Control apparatus, robot, and control method |
Family Cites Families (36)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3196889B2 (en) * | 1996-09-05 | 2001-08-06 | 株式会社アルファ | Three-dimensional image processing method and computer-readable recording medium storing a program for causing a computer to execute the three-dimensional image processing method |
US8897370B1 (en) * | 2009-11-30 | 2014-11-25 | Google Inc. | Bitrate video transcoding based on video coding complexity estimation |
US8862763B2 (en) * | 2011-03-30 | 2014-10-14 | Verizon Patent And Licensing Inc. | Downloading video using excess bandwidth |
US8810598B2 (en) * | 2011-04-08 | 2014-08-19 | Nant Holdings Ip, Llc | Interference based augmented reality hosting platforms |
GB2501929B (en) * | 2012-05-11 | 2015-06-24 | Sony Comp Entertainment Europe | Apparatus and method for augmented reality |
SG11201500943PA (en) * | 2012-08-08 | 2015-03-30 | Univ Singapore | System and method for enabling user control of live video stream(s) |
US9129429B2 (en) * | 2012-10-24 | 2015-09-08 | Exelis, Inc. | Augmented reality on wireless mobile devices |
GB2509953B (en) * | 2013-01-18 | 2015-05-20 | Canon Kk | Method of displaying a region of interest in a video stream |
CN103702139B (en) * | 2013-12-13 | 2017-02-01 | 华中科技大学 | Video-on-demand system based on scalable coding under mobile environment |
US9398250B2 (en) * | 2014-01-06 | 2016-07-19 | Arun Sobti & Associates, Llc | System and apparatus for smart devices based conferencing |
CN105025351B (en) * | 2014-04-30 | 2018-06-29 | 深圳Tcl新技术有限公司 | The method and device of DST PLAYER buffering |
US9918136B2 (en) * | 2014-05-29 | 2018-03-13 | Nextvr Inc. | Methods and apparatus for delivering content and/or playing back content |
CN104240281A (en) * | 2014-08-28 | 2014-12-24 | 东华大学 | Virtual reality head-mounted device based on Unity3D engine |
CN104268922B (en) * | 2014-09-03 | 2017-06-06 | 广州博冠信息科技有限公司 | A kind of image rendering method and image rendering device |
US10812546B2 (en) * | 2014-12-24 | 2020-10-20 | Intel IP Corporation | Link-aware streaming adaptation |
CN104616243B (en) * | 2015-01-20 | 2018-02-27 | 北京道和汇通科技发展有限公司 | A kind of efficient GPU 3 D videos fusion method for drafting |
US20160261908A1 (en) * | 2015-03-05 | 2016-09-08 | Htc Corporation | Media streaming system and control method thereof |
CN104735464A (en) * | 2015-03-31 | 2015-06-24 | 华为技术有限公司 | Panorama video interactive transmission method, server and client end |
CN104717507A (en) * | 2015-03-31 | 2015-06-17 | 北京奇艺世纪科技有限公司 | Video transcoding method and device |
US10083363B2 (en) * | 2015-05-26 | 2018-09-25 | Nbcuniversal Media, Llc | System and method for customizing content for a user |
JP6866297B2 (en) * | 2015-06-12 | 2021-04-28 | グーグル エルエルシーGoogle LLC | Electronic display stabilization of head-mounted display |
US9956054B2 (en) * | 2015-06-25 | 2018-05-01 | EchoPixel, Inc. | Dynamic minimally invasive surgical-aware assistant |
CN108476324B (en) * | 2015-10-08 | 2021-10-29 | 皇家Kpn公司 | Method, computer and medium for enhancing regions of interest in video frames of a video stream |
CN106919248A (en) * | 2015-12-26 | 2017-07-04 | 华为技术有限公司 | It is applied to the content transmission method and equipment of virtual reality |
CN105916022A (en) * | 2015-12-28 | 2016-08-31 | 乐视致新电子科技(天津)有限公司 | Video image processing method and apparatus based on virtual reality technology |
CN105455285B (en) * | 2015-12-31 | 2019-02-12 | 北京小鸟看看科技有限公司 | A kind of virtual implementing helmet adaptation method |
US10412130B2 (en) * | 2016-04-04 | 2019-09-10 | Hanwha Techwin Co., Ltd. | Method and apparatus for playing media stream on web browser |
US10313417B2 (en) * | 2016-04-18 | 2019-06-04 | Qualcomm Incorporated | Methods and systems for auto-zoom based adaptive video streaming |
CN105898565A (en) * | 2016-04-28 | 2016-08-24 | 乐视控股(北京)有限公司 | Video processing method and device |
US9721393B1 (en) * | 2016-04-29 | 2017-08-01 | Immersive Enterprises, LLC | Method for processing and delivering virtual reality content to a user |
CN106060570B (en) * | 2016-06-30 | 2019-06-14 | 北京奇艺世纪科技有限公司 | A kind of full-view video image plays, coding method and device |
CN106060515B (en) * | 2016-07-14 | 2018-11-06 | 腾讯科技(深圳)有限公司 | Panorama pushing method for media files and device |
CN106231317A (en) * | 2016-09-29 | 2016-12-14 | 三星电子(中国)研发中心 | Video processing, coding/decoding method and device, VR terminal, audio/video player system |
US10595069B2 (en) * | 2016-12-05 | 2020-03-17 | Adobe Inc. | Prioritizing tile-based virtual reality video streaming using adaptive rate allocation |
US20180295375A1 (en) * | 2017-04-05 | 2018-10-11 | Lyrical Labs Video Compression Technology, LLC | Video processing and encoding |
CN107087212B (en) * | 2017-05-09 | 2019-10-29 | 杭州码全信息科技有限公司 | Interactive panoramic video transcoding and playback method and system based on spatial scalable coding |
-
2018
- 2018-01-02 US US15/860,449 patent/US10334238B2/en not_active Expired - Fee Related
- 2018-01-02 US US15/860,392 patent/US20180192044A1/en not_active Abandoned
- 2018-01-02 US US15/860,494 patent/US20180189980A1/en not_active Abandoned
- 2018-01-02 US US15/860,358 patent/US20180192063A1/en not_active Abandoned
- 2018-01-02 US US15/860,471 patent/US20180192022A1/en not_active Abandoned
- 2018-01-02 US US15/860,430 patent/US20180191868A1/en not_active Abandoned
- 2018-01-03 CN CN201810005019.3A patent/CN108419142A/en active Pending
- 2018-01-03 CN CN201810005787.9A patent/CN108366272A/en active Pending
- 2018-01-03 CN CN201810005786.4A patent/CN108377381A/en active Pending
- 2018-01-03 CN CN201810005788.3A patent/CN108391103A/en active Pending
- 2018-01-03 CN CN201810005392.9A patent/CN108366293A/en active Pending
- 2018-01-03 CN CN201810004987.2A patent/CN108419093A/en active Pending
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070292113A1 (en) * | 2005-06-30 | 2007-12-20 | Meng-Nan Tsou | Video decoding apparatus, video decoding method, and digital audio/video playback system capable of controlling presentation of sub-pictures |
US20130044108A1 (en) * | 2011-03-31 | 2013-02-21 | Panasonic Corporation | Image rendering device, image rendering method, and image rendering program for rendering stereoscopic panoramic images |
US20140225816A1 (en) * | 2013-02-12 | 2014-08-14 | Pixtronix, Inc. | Display Having Staggered Display Element Arrangement |
US20150339833A1 (en) * | 2014-05-23 | 2015-11-26 | Seiko Epson Corporation | Control apparatus, robot, and control method |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10331862B2 (en) * | 2017-04-20 | 2019-06-25 | Cisco Technology, Inc. | Viewport decryption |
Also Published As
Publication number | Publication date |
---|---|
US20180192063A1 (en) | 2018-07-05 |
US20180192022A1 (en) | 2018-07-05 |
US20180191868A1 (en) | 2018-07-05 |
CN108377381A (en) | 2018-08-07 |
US20180192026A1 (en) | 2018-07-05 |
CN108419093A (en) | 2018-08-17 |
CN108419142A (en) | 2018-08-17 |
CN108366293A (en) | 2018-08-03 |
US20180189980A1 (en) | 2018-07-05 |
CN108366272A (en) | 2018-08-03 |
US10334238B2 (en) | 2019-06-25 |
CN108391103A (en) | 2018-08-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20180192044A1 (en) | Method and System for Providing A Viewport Division Scheme for Virtual Reality (VR) Video Streaming | |
WO2017193576A1 (en) | Video resolution adaptation method and apparatus, and virtual reality terminal | |
US10506223B2 (en) | Method, apparatus, and device for realizing virtual stereoscopic scene | |
US9485493B2 (en) | Method and system for displaying multi-viewpoint images and non-transitory computer readable storage medium thereof | |
US20170153866A1 (en) | Audiovisual Surround Augmented Reality (ASAR) | |
US20050052623A1 (en) | Projecting system | |
CN109074678A (en) | A kind of processing method and processing device of information | |
EP4231650A1 (en) | Picture display method and apparatus, and electronic device | |
WO2020248900A1 (en) | Panoramic video processing method and apparatus, and storage medium | |
US20210327093A1 (en) | Image processing method of virtual reality and apparatus thereof | |
US20170150212A1 (en) | Method and electronic device for adjusting video | |
CN111198610A (en) | Method, device and equipment for controlling field of view of panoramic video and storage medium | |
CN109246477B (en) | Panoramic video frame interpolation method and device | |
US20220382053A1 (en) | Image processing method and apparatus for head-mounted display device as well as electronic device | |
CN108401163B (en) | Method and device for realizing VR live broadcast and OTT service system | |
CN111292280B (en) | Method and device for outputting information | |
CN100498840C (en) | Method of and scaling unit for scaling a three-dimensional model | |
CN111200758B (en) | Multi-view-field control method and device for panoramic video, electronic equipment and storage medium | |
CN110662099B (en) | Method and device for displaying bullet screen | |
KR101773929B1 (en) | System for processing video with wide viewing angle, methods for transmitting and displaying vide with wide viewing angle and computer programs for the same | |
WO2022134962A1 (en) | Method and apparatus for presenting point cloud window, computer-readable medium, and electronic device | |
CN111937043A (en) | Associating file format objects with dynamic adaptive streaming over hypertext transfer protocol (DASH) objects | |
CN117173378B (en) | CAVE environment-based WebVR panoramic data display method, device, equipment and medium | |
CN111277886B (en) | Panoramic video view field control method and device, electronic equipment and storage medium | |
CN106998461A (en) | One kind wears display system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: BLACK SAILS TECHNOLOGY INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:WANG, ZHUO;TANG, YONGTAO;ZHAO, RUOXI;AND OTHERS;REEL/FRAME:044518/0618 Effective date: 20180102 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |