CN108366272A - Vr视频渲染方法及装置 - Google Patents

Vr视频渲染方法及装置 Download PDF

Info

Publication number
CN108366272A
CN108366272A CN201810005787.9A CN201810005787A CN108366272A CN 108366272 A CN108366272 A CN 108366272A CN 201810005787 A CN201810005787 A CN 201810005787A CN 108366272 A CN108366272 A CN 108366272A
Authority
CN
China
Prior art keywords
video
enhancing
textures
base
model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201810005787.9A
Other languages
English (en)
Inventor
王焯
唐泳滔
赵若熹
张家齐
祖浩焱
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Black Sail Technology Co Ltd
Original Assignee
Black Sail Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Black Sail Technology Co Ltd filed Critical Black Sail Technology Co Ltd
Publication of CN108366272A publication Critical patent/CN108366272A/zh
Pending legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/398—Synchronisation thereof; Control thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H04N9/3179—Video signal processing therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/04—Texture mapping
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/10—Geometric effects
    • G06T15/20—Perspective computation
    • G06T15/205—Image-based rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/001—Model-based coding, e.g. wire frame
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H04L43/0876—Network utilisation, e.g. volume of load or congestion level
    • H04L43/0888—Throughput
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/75—Media network packet handling
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/75—Media network packet handling
    • H04L65/752—Media network packet handling adapting media to network capabilities
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/131—Protocols for games, networked simulations or virtual reality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
    • H04N13/117—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/122—Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/139—Format conversion, e.g. of frame-rate or size
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/189—Recording image signals; Reproducing recorded image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/275—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H04N13/344—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/366—Image reproducers using viewer tracking
    • H04N13/378—Image reproducers using viewer tracking for tracking rotational head movements around an axis perpendicular to the screen
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/366—Image reproducers using viewer tracking
    • H04N13/383—Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/40—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/44—Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/233—Processing of audio elementary streams
    • H04N21/2335—Processing of audio elementary streams involving reformatting operations of audio signals, e.g. by converting from one coding standard to another
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234345—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234363—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the spatial resolution, e.g. for clients with a lower screen resolution
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234381—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/238—Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams
    • H04N21/2387—Stream processing in response to a playback request from an end-user, e.g. for trick-play
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/239—Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests
    • H04N21/2393—Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests involving handling client requests
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
    • H04N21/2405—Monitoring of the internal components or processes of the server, e.g. server load
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
    • H04N21/2662—Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/4104—Peripherals receiving signals from specially adapted client devices
    • H04N21/4122—Peripherals receiving signals from specially adapted client devices additional display device, e.g. video projector
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44012—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H04N21/440236—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/845—Structuring of content, e.g. decomposing content into time segments
    • H04N21/8456—Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/76—Television signal recording
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H04N9/3179—Video signal processing therefor
    • H04N9/3185—Geometric adjustment, e.g. keystone or convergence
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/79—Processing of colour television signals in connection with recording
    • H04N9/87—Regeneration of colour television signals
    • H04N9/8715—Regeneration of colour television signals involving the mixing of the reproduced video signal with a non-recorded signal, e.g. a text signal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2215/00—Indexing scheme for image rendering
    • G06T2215/16—Using real world measurements to influence rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Computer Graphics (AREA)
  • Human Computer Interaction (AREA)
  • Geometry (AREA)
  • Environmental & Geological Engineering (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Controls And Circuits For Display Device (AREA)
  • Image Generation (AREA)
  • Processing Or Creating Images (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Image Processing (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)

Abstract

公开了一种VR视频渲染方法，包括：分别创建基础视频模型与增强视频模型；初始化基础视频模型和增强视频模型上的UV坐标；根据用户当前的视场角度获得基础视频片段和增强视频片段；解码获得基础视频片段和增强视频片段的像素信息；分别根据基础视频片段的像素信息和基础视频模型的UV坐标生成基础视频贴图，以及根据增强视频片段的像素信息和增强视频模型的UV坐标生成增强视频贴图；确定增强视频贴图的UV对齐坐标；根据UV对齐坐标生成基础视频贴图与增强视频贴图叠加后的像素信息；以及根据叠加后的像素信息绘出图像。通过叠加基础视频和增强视频以播放VR视频，降低了数据下载量，提升了渲染效率。同时公开对应的装置。

Description

VR视频渲染方法及装置
技术领域
本发明涉及视频处理技术，更具体地，涉及一种VR视频渲染方法及装置。
背景技术
虚拟现实(VR)技术是一种创建和体验虚拟世界的计算机仿系统，例如，可以根据观察者头、眼和手的跟踪技术呈现相对应的三维实时图像。在基于网络的虚拟现实技术中，预先将全视角的视频数据存储在服务器上，然后将全视角的视频数据传送至诸如眼镜之类的显示装置上，根据观察者的观看位置将相应视角的视频数据显示在显示装置上。
然而，高分辨率的视频数据需要占据大量的传输带宽，并且需要显示装置具备强大的数据处理能力。由于数据传输带宽和显示装置处理能力的需求高，因此，现有的视频处理技术对网络和设备终端设置的要求高。
因此，期望在网络以及设备终端进一步改进VR视频处理过程，以提升VR视频的观影体验。
发明内容
有鉴于此，本申请提供VR视频渲染方法及装置，以解决上述问题。
根据本发明的第一方面，提供一种VR视频渲染方法，包括：
分别创建基础视频模型与增强视频模型；
初始化所述基础视频模型和所述增强视频模型上的UV坐标；
根据用户当前的视场角度获得基础视频片段和增强视频片段；
解码获得所述基础视频片段和所述增强视频片段的像素信息；
分别根据所述基础视频片段的像素信息和所述基础视频模型的UV 坐标生成基础视频贴图，以及根据所述增强视频片段的像素信息和所述增强视频模型的UV坐标生成增强视频贴图；
确定所述增强视频贴图的UV对齐坐标；
根据所述UV对齐坐标生成所述基础视频贴图与所述增强视频贴图叠加后的像素信息；以及
根据叠加后的像素信息绘出图像。
优选地，所述基础视频模型和增强视频模型为相同的多边形球面。
优选地，通过获取设备陀螺仪的滚转参数获得所述用户当前的视场角度。
优选地，根据对齐前的增强视频贴图的UV对齐坐标计算对齐后的增强视频贴图的UV对齐坐标。
优选地，所述根据对齐前的增强视频贴图的UV对齐坐标计算对齐后的增强视频贴图的UV对齐坐标包括以下步骤：
设定视场角度缩放因子i与fov 换算公式为：则
对齐前的U值和对齐后的U值可以通过以下公式进行转换：
对齐前的V值和对齐后的V值通过以下公式进行转换：
优选地，所述根据所述UV对齐坐标生成所述基础视频贴图与所述增强视频贴图叠加后的像素信息包括：
采用下述公式生成所述叠加后的像素信息，
其中，Px,yReconstructed对应于相应坐标(x,y)的叠加后的像素信息，Px,yScaleBase对应于相应坐标(x,y)的基础视频贴图的像素信息， Px,yNormalizedResidual对应于相应坐标(x,y)的增强视频贴图的像素信息，H,L对应于像素信息的最大和最小值。
优选地，调用OpenGL和/或WebGL的相关接口实现相应功能。
根据本发明的第二方面，提供一种VR视频渲染装置，包括：
初始化模块，用于分别创建基础视频模型与增强视频模型，并且初始化所述基础视频模型和所述增强视频模型上的UV坐标；
下载解码模块，用于根据用户当前的视场角度获得基础视频片段和增强视频片段，并且解码获得所述基础视频片段和所述增强视频片段的像素信息；
贴图生成模块，用于分别根据所述基础视频片段的像素信息和所述基础视频模型的UV坐标生成基础视频贴图，以及根据所述增强视频片段的像素信息和所述增强视频模型的UV坐标生成增强视频贴图；
对齐坐标确定模块，用于确定所述增强视频贴图的UV对齐坐标；
像素计算模块，用于根据所述UV对齐坐标生成所述基础视频贴图与所述增强视频贴图叠加后的像素信息；
绘制模块，用于根据叠加后的像素信息绘出图像。
优选地，所述基础视频模型和增强视频模型为相同的多边形球面。
优选地，通过获取设备陀螺仪的滚转参数获得所述用户当前的视场角度。
本发明提供的VR视频渲染方法，由于高分辨率的视频数据在服务器被处理成增强视频集和基础视频集，则在设备终端可以通过叠加基础视频和增强视频以实现VR视频数据播放，其中根据视场角度确定基础视频和增强视频的叠加位置，从而在不影响用户的观影体验的条件下，降低了数据下载量，提升了渲染效率。
附图说明
通过参照以下附图对本发明实施例的描述，本发明的上述以及其它目的、特征和优点将更为清楚，在附图中：
图1是VR播放系统的网络示意图；
图2是适用于图1的VR播放系统的方法的流程图；
图3示出了本发明实施例的VR视频渲染方法的流程图；
图4是本发明实施例的VR视频渲染装置的结构图。
具体实施方式
以下将参照附图更详细地描述本发明。在各个附图中，相同的元件采用类似的附图标记来表示。为了清楚起见，附图中的各个部分没有按比例绘制。此外，可能未示出某些公知的部分。
图1是VR播放系统的网络示意图。设备网络10包括通过网络110 耦接的服务器100和显示设备120以及VR设备。服务器100可以例如独立的计算机服务器或者服务器集群，服务器100用于存储各种视频数据以及存储处理这些视频数据的各种应用程序。服务器100上例如实时运行着各种守护进程，一方面，用于处理服务器100上的各种视频数据，另一方面，用于响应VR设备和显示设备120的各种请求。网络110可以是任意(一个或多个)网络，例如因特网、局域网和物联网等。显示设备120例如是一个具有独立显示屏的和一定处理能力的计算机设备，可以是：个人计算机、膝上型计算机、计算机工作站、服务器、大型主机计算机、掌上电脑、个人数字助理、智能手机、智能电器、游戏控制台、 iPAD/iPhone、视频播放器、DVD刻录机/播放器、电视机、家庭娱乐系统或任意其它合适的计算设备。显示设备120可以存储VR播放器软件，当VR播放器启动后，播放器向服务器100请求和接收各种视频数据，并且将下载到显示设备端的视频数据在播放器内进行渲染播放。在本例中， VR设备130是一个外接式头显设备，可以和显示设备120以及服务器100 进行各种交互，通过信号传递向显示设备120和/或服务器100传递一些用户当前的观影角度变化信息，例如，用户头盔位置的改变、眼睛视线的变化等等，显示设备120可以根据这些信息能够灵活处理当前播放的视频数据。在一些实施例里，当用户头盔位置发生改变时，显示设备120 判断用户的核心观影区域发生改变，并在核心观影区域播放高分辨率的视频数据。
在上述实施例，VR设备130是一个外接式头显设备，但是本领域的技术人员可以理解到，VR设备130不仅限于此，VR设备130还可以一体式头显设备。一体式头显设备本身具有显示屏的功能，因此可以不需要外部的显示设备，例如本例中，如果采用一体式头显设备，即可以省略显示120。此时，一体式头显设备承担了从服务器100上获取视频数据以及播放的工作，同时，一体式头显设备检测到用户当前的一些观影角度变化信息并基于这些观影角度变化信息调整播放工作。
图2是适用于图1的VR播放系统的方法的流程图。该方法包括以下步骤。
在步骤S10中，服务器端进行源视频数据处理。
在步骤S20中，显示设备端基于和VR设备的交互获得用户当前的视场角度。
在步骤S30中，显示设备端根据用户当前的视场角度向服务器端请求视频数据以及接收视频数据。
在步骤S40中，显示设备端基于接收到的视频数据进行渲染处理。
其中，步骤S40将从服务器获取的视频数据按照设定的模型绘制出来，即实现播放。基于此，图3示出了适用于步骤S40的VR视频渲染方法。
图3是本发明实施例的VR视频渲染方法的流程图。具体包括以下步骤。
在步骤S100中，根据用户当前的视场角度获得基础视频片段和增强视频片段。
当VR视频播放时，检测设备，例如设置在显示设备或VR设备上的设备陀螺仪，能够检测用户的一些位置变化，并将位置变化信息发送给设备端(VR设备、显示设备或服务器)，进而根据位置变化信息确定用户当前的观影角度。在服务器端，在视频转码过程中，可以将原始视频数据处理成需要叠加播放的基础视频片段和增强视频片段。例如，对于源视频进行时间和/或空间上的切割获得多个视频片段。而本步骤中，根据当前的视场角度从服务器端获得对应视频的基础视频片段和增强视频片段。
在步骤S200中，解码获得基础视频和增强视频的像素信息。
在本步骤中，通过合适的解码器解码基础视频片段和增强视频片段，获得各自的像素信息。解码器解码过程可能还包括解压缩过程，用于解压缩压缩后的视频数据。对应于各个不同的颜色空间，解出不同的像素分量，例如解出R、G、B分量。
在步骤S300中，分别创建基础视频模型与增强视频模型。
本步骤中，可以根据需求选择创建三维模型，例如，可以创建两个多边形球面作为基础视频模型和增强视频模型。
在步骤S400中，初始化模型上的点UV坐标。
UV坐标这里指u,v纹理贴图坐标的简称,类似于空间模型的X,Y,Z 轴。它定义了和3D模型相互对应的平面上的每个点的位置信息。通过 UV坐标能够将图像上的每一点都精确对应到三维模型上。在本步骤中，建立基础视频模型和增强视频模型上的每个UV坐标点。
在步骤S500中，生成基础视频贴图和增强视频贴图。
在本步骤中，根据基础视频片段的像素信息和基础视频模型的UV坐标生成基础视频贴图，并且，根据增强视频片段的像素信息和增强视频模型的UV坐标生成增强视频贴图。
在步骤S600中，确定增强视频模型的UV对齐坐标。
在本步骤中，可以通过用户当前的视场角度确定基础视频模型和增强视频模型的UV对齐坐标。
在步骤S700中，根据UV对齐坐标生成基础视频贴图与增强视频贴图叠加后的像素信息。
在本步骤中，基于基础视频片段和增强视频片段之间的关系，计算获得叠加后的的像素信息。
为了便于理解，采用以下述示例进一步说明。
假设像素Px,yOriginal＝(r,g,b)T是源视频数据中，坐标为x,y的像素，r,g,b∈[L,H]，Px,yScaledBase＝(r′,g′,b′)T是基础视频中，坐标为 x,y的像素，r′,g′,b′∈[L,H]，那么对于所有的x,y有以下差值生成方程式(1)：
Px,yNormalizedResidual表示像素差值，
对于所有的像素x,y有以下的差值还原公式为：
在步骤S800，根据叠加后的像素信息绘出图像。
本发明提供的VR视频渲染方法，从服务器端分别下载基础视频片段和增强视频片段，通过叠加基础视频和增强视频以实现VR视频数据播放，其中，根据用户当前的视场角度确定基础视频和增强视频的叠加位置的UV对齐坐标，从而在不影响用户的观影体验的条件下，降低了数据下载量，提高了渲染效率。
应该指出的是，在上述实施例中，针对一个具体的视场角度，可能对应多个增强视频片段(由视频片段的分割方式决定)，则对于每一个增强视频片段，重复执行上述步骤，实现和基础视频片段的叠加渲染。
在可选的实施例中，调用openGL和/或WebGL的接口实现相关功能。 OpenGL(OpenGraphics Library)定义了一个图形程序接口，用于二维或三维图像的绘制。WebGL(WebGraphics Library)是一种3D绘图协议。
另外，对于步骤S600，可以基于对齐前的增强视频片段的UV对齐坐标计算对齐后的增强视频片段的UV对齐坐标。具体如下。
表格1
设定视场角度缩放因子i与fov 换算公式为：则
和可以通过以下公式进行转换：
和通过以下公式进行转换：
其中，id为视频转码过程中，将视频切割成基础视频片段和增强视频片段时获得的的增强视频片段的编号，其值为从0开始的正整数。
在上述示例中，通过设定视场角度的枚举值，并且获得最初的增强视频贴图UV坐标，则可以在视场角度发生变化时，计算出变化后的增强视频贴图的UV坐标。当然，本领域的技术人员也可以采用其他方式获得增强视频贴图的UV坐标。
图4是本发明实施例的VR视频渲染装置的结构图。
该VR视频渲染装置400，包括初始化模块401、下载解码模块402、贴图生成模块403、对齐坐标确定模块404、像素计算模块405和绘制模块406。
初始化模块401用于分别创建基础视频模型与增强视频模型并且初始化基础视频模型和增强视频模型上的UV坐标。根据需求创球形、圆柱形或其他形状的三维模型，并初始化三维模型中的UV坐标。
下载解码模块402用于根据用户当前的视场角度获得基础视频片段和/或增强视频片段，并且解码获得基础视频片段和增强视频片段的像素信息。可以通过获取设备陀螺仪的滚转参数获得所述用户当前的视场角度，进而根据当前播放的视频以及用户的视场角度确定下载一个和多个基础和/或增强视频片段。
贴图生成模块403用于分别根据基础视频片段的像素信息和基础视频模型的UV坐标生成基础视频贴图，以及根据增强视频片段的像素信息和增强视频模型的UV坐标生成增强视频贴图。
对齐坐标确定模块404用于根据用户当前的视场角度确定增强视频贴图的UV对齐坐标。
像素计算模块405用于根据UV对齐坐标生成基础视频贴图与增强视频贴图叠加后的像素信息。
绘制模块406用于根据叠加后的像素信息绘出图像。
优选地，所述基础视频模型和增强视频模型为相同的多边形球面。
本发明实施例虽然以较佳实施例公开如上，但其并不是用来限定权利要求，任何本领域技术人员在不脱离本发明的精神和范围内，都可以做出可能的变动和修改，因此本发明的保护范围应当以本发明权利要求所界定的范围为准。
以上所述仅为本发明的优选实施例，并不用于限制本发明，对于本领域技术人员而言，本发明可以有各种改动和变化。凡在本发明的精神和原理之内所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。

Claims (10)

1.一种VR视频渲染方法，包括：
分别创建基础视频模型与增强视频模型；
初始化所述基础视频模型和所述增强视频模型上的UV坐标；
根据用户当前的视场角度获得基础视频片段和增强视频片段；
解码获得所述基础视频片段和所述增强视频片段的像素信息；
分别根据所述基础视频片段的像素信息和所述基础视频模型的UV坐标生成基础视频贴图，以及根据所述增强视频片段的像素信息和所述增强视频模型的UV坐标生成增强视频贴图；以及
确定所述增强视频贴图的UV对齐坐标；
根据所述UV对齐坐标生成所述基础视频贴图与所述增强视频贴图叠加后的像素信息；以及
根据叠加后的像素信息绘出图像。
2.根据权利要求1所述的VR视频渲染方法，其中，所述基础视频模型和增强视频模型为相同的多边形球面。
3.根据权利要求1所述的VR视频渲染方法，其中，通过获取设备陀螺仪的滚转参数获得所述用户当前的视场角度。
4.根据权利要求1所述的VR视频渲染方法，其中，根据对齐前的增强视频贴图的UV对齐坐标计算对齐后的增强视频贴图的UV对齐坐标。
5.根据权利要求4所述的VR视频渲染方法，其中，所述根据对齐前的增强视频贴图的UV对齐坐标计算对齐后的增强视频贴图的UV对齐坐标包括以下步骤：
设定视场角度缩放因子i与fov换算公式为：则
对齐前的增强视频贴图的U值和对齐后的增强视频贴图的U值可以通过以下公式进行转换：
id为对应的增强视频片段编号，id*Δ∈[0,1]；
对齐前的增强视频贴图的V值和对齐后的增强视频贴图的V值通过以下公式进行转换：
6.根据权利要求1所述的VR视频渲染方法，其中，所述根据所述UV对齐坐标生成所述基础视频贴图与所述增强视频贴图叠加后的像素信息包括：
采用下述公式生成所述叠加后的像素信息，
其中，Px,yReconstructed对应于相应坐标(x,y)的叠加后的像素信息，Px,yScaleBase对应于相应坐标(x,y)的基础视频贴图的像素信息，Px,yNormalizedResidual对应于相应坐标(x,y)的增强视频贴图的像素信息，H,L对应于像素信息的最大和最小值。
7.根据权利要求1所述的VR视频渲染方法，其中，调用OpenGL和/或WebGL的相关接口实现相关功能。
8.一种VR视频渲染装置，包括：
初始化模块，用于分别创建基础视频模型与增强视频模型，并且初始化所述基础视频模型和所述增强视频模型上的UV坐标；
下载解码模块，用于根据用户当前的视场角度获得基础视频片段和增强视频片段，并且解码获得所述基础视频片段和所述增强视频片段的像素信息；
贴图生成模块，用于分别根据所述基础视频片段的像素信息和所述基础视频模型的UV坐标生成基础视频贴图，以及根据所述增强视频片段的像素信息和所述增强视频模型的UV坐标生成增强视频贴图；
对齐坐标确定模块，用于确定所述增强视频贴图的UV对齐坐标；
像素计算模块，用于根据所述UV对齐坐标生成所述基础视频贴图与所述增强视频贴图叠加后的像素信息；
绘制模块，用于根据叠加后的像素信息绘出图像。
9.根据权利要求8所述的VR视频渲染装置，其中，所述基础视频模型和增强视频模型为相同的多边形球面。
10.根据权利要求1所述的VR视频渲染装置，其中，通过获取设备陀螺仪的滚转参数获得所述用户当前的视场角度。
CN201810005787.9A 2017-01-03 2018-01-03 Vr视频渲染方法及装置 Pending CN108366272A (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201762441936P 2017-01-03 2017-01-03
USUS62/441,936 2017-01-03

Publications (1)

Publication Number Publication Date
CN108366272A true CN108366272A (zh) 2018-08-03

Family

ID=62711388

Family Applications (6)

Application Number Title Priority Date Filing Date
CN201810005019.3A Pending CN108419142A (zh) 2017-01-03 2018-01-03 Vr视频播放方法及装置
CN201810005787.9A Pending CN108366272A (zh) 2017-01-03 2018-01-03 Vr视频渲染方法及装置
CN201810005786.4A Pending CN108377381A (zh) 2017-01-03 2018-01-03 沉浸式vr视频渲染方法及装置
CN201810005788.3A Pending CN108391103A (zh) 2017-01-03 2018-01-03 Vr视频观影区域划分方法及装置
CN201810005392.9A Pending CN108366293A (zh) 2017-01-03 2018-01-03 Vr视频转码方法及装置
CN201810004987.2A Pending CN108419093A (zh) 2017-01-03 2018-01-03 Vr视频下载方法及装置

Family Applications Before (1)

Application Number Title Priority Date Filing Date
CN201810005019.3A Pending CN108419142A (zh) 2017-01-03 2018-01-03 Vr视频播放方法及装置

Family Applications After (4)

Application Number Title Priority Date Filing Date
CN201810005786.4A Pending CN108377381A (zh) 2017-01-03 2018-01-03 沉浸式vr视频渲染方法及装置
CN201810005788.3A Pending CN108391103A (zh) 2017-01-03 2018-01-03 Vr视频观影区域划分方法及装置
CN201810005392.9A Pending CN108366293A (zh) 2017-01-03 2018-01-03 Vr视频转码方法及装置
CN201810004987.2A Pending CN108419093A (zh) 2017-01-03 2018-01-03 Vr视频下载方法及装置

Country Status (2)

Country Link
US (6) US10334238B2 (zh)
CN (6) CN108419142A (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111489428A (zh) * 2020-04-20 2020-08-04 北京字节跳动网络技术有限公司 图像生成方法、装置、电子设备及计算机可读存储介质
CN111754614A (zh) * 2020-06-30 2020-10-09 平安国际智慧城市科技股份有限公司 基于vr视频渲染方法、装置、电子设备及存储介质
CN115396731A (zh) * 2022-08-10 2022-11-25 北京势也网络技术有限公司 全景视频的播放方法、装置、电子设备及可读存储介质

Families Citing this family (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10291910B2 (en) * 2016-02-12 2019-05-14 Gopro, Inc. Systems and methods for spatially adaptive video encoding
US10331862B2 (en) * 2017-04-20 2019-06-25 Cisco Technology, Inc. Viewport decryption
CN115842907A (zh) * 2018-03-27 2023-03-24 京东方科技集团股份有限公司 渲染方法、计算机产品及显示装置
US11232532B2 (en) * 2018-05-30 2022-01-25 Sony Interactive Entertainment LLC Multi-server cloud virtual reality (VR) streaming
US10623791B2 (en) 2018-06-01 2020-04-14 At&T Intellectual Property I, L.P. Field of view prediction in live panoramic video streaming
US10812774B2 (en) 2018-06-06 2020-10-20 At&T Intellectual Property I, L.P. Methods and devices for adapting the rate of video content streaming
US10616621B2 (en) * 2018-06-29 2020-04-07 At&T Intellectual Property I, L.P. Methods and devices for determining multipath routing for panoramic video content
US11019361B2 (en) 2018-08-13 2021-05-25 At&T Intellectual Property I, L.P. Methods, systems and devices for adjusting panoramic view of a camera for capturing video content
CN109343518B (zh) * 2018-09-03 2021-07-02 浙江大丰实业股份有限公司 万向车台现场驱动平台
US11128869B1 (en) * 2018-10-22 2021-09-21 Bitmovin, Inc. Video encoding based on customized bitrate table
CN109375369B (zh) * 2018-11-23 2021-05-18 国网天津市电力公司 一种vr巨幕影院模式下的畸变预处理方法
CN111510777B (zh) * 2019-01-30 2021-11-23 上海哔哩哔哩科技有限公司 测量网速的方法、装置、计算机设备及可读存储介质
CN111669666A (zh) * 2019-03-08 2020-09-15 北京京东尚科信息技术有限公司 模拟现实的方法、装置和系统
CN111866485A (zh) * 2019-04-25 2020-10-30 中国移动通信有限公司研究院 立体画面投影和传输方法、装置和计算机可读存储介质
CN110336994B (zh) * 2019-07-04 2021-06-22 上海索倍信息科技有限公司 一种裸眼3d显示系统
CN110381331A (zh) * 2019-07-23 2019-10-25 深圳市道通智能航空技术有限公司 一种图像处理方法、装置、航拍设备及存储介质
CN110490962B (zh) * 2019-08-20 2023-09-15 武汉邦拓信息科技有限公司 一种基于视频流的远程渲染方法
CN110544425A (zh) * 2019-09-13 2019-12-06 广州城市职业学院 一种古建筑vr展示系统
US11245911B1 (en) * 2020-05-12 2022-02-08 Whirlwind 3D, LLC Video encoder/decoder (codec) for real-time applications and size/b and width reduction
CN112468806B (zh) * 2020-11-12 2022-07-26 中山大学 一种用于云vr平台的全景视频传输优化方法
CN114286142B (zh) * 2021-01-18 2023-03-28 海信视像科技股份有限公司 一种虚拟现实设备及vr场景截屏方法
CN113347402A (zh) * 2021-06-28 2021-09-03 筑友建筑装饰装修工程有限公司 基于Unity的渲染沉浸式内容的改良方法、装置和存储介质
CN114466220A (zh) * 2022-01-29 2022-05-10 维沃移动通信有限公司 视频下载方法和电子设备
CN116880723B (zh) * 2023-09-08 2023-11-17 江西格如灵科技股份有限公司 一种3d场景显示方法及系统

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120256954A1 (en) * 2011-04-08 2012-10-11 Patrick Soon-Shiong Interference Based Augmented Reality Hosting Platforms
CN103390287A (zh) * 2012-05-11 2013-11-13 索尼电脑娱乐欧洲有限公司 用于增强现实的装置和方法
CN104268922A (zh) * 2014-09-03 2015-01-07 广州博冠信息科技有限公司 一种图像渲染方法及图像渲染装置
CN104616243A (zh) * 2015-01-20 2015-05-13 北京大学 一种高效的gpu三维视频融合绘制方法
US20150195489A1 (en) * 2014-01-06 2015-07-09 Arun Sobti & Associates, Llc System and apparatus for smart devices based conferencing
US20150356790A1 (en) * 2012-10-24 2015-12-10 Exelis Inc. Augmented Reality Control Systems
CN105916022A (zh) * 2015-12-28 2016-08-31 乐视致新电子科技（天津）有限公司 一种基于虚拟现实技术的视频图像处理方法及装置

Family Cites Families (33)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3196889B2 (ja) * 1996-09-05 2001-08-06 株式会社アルファ 三次元画像処理方法および三次元画像処理方法をコンピュータに実行させるプログラムを記録したコンピュータ読み取り可能な記録媒体
TWI262725B (en) * 2005-06-30 2006-09-21 Cheertek Inc Video decoding apparatus and digital audio and video display system capable of controlling presentation of subtitles and method thereof
US8897370B1 (en) * 2009-11-30 2014-11-25 Google Inc. Bitrate video transcoding based on video coding complexity estimation
US8862763B2 (en) * 2011-03-30 2014-10-14 Verizon Patent And Licensing Inc. Downloading video using excess bandwidth
JP5891426B2 (ja) * 2011-03-31 2016-03-23 パナソニックＩｐマネジメント株式会社 全周囲立体画像の描画を行う画像描画装置、画像描画方法、画像描画プログラム
SG11201500943PA (en) * 2012-08-08 2015-03-30 Univ Singapore System and method for enabling user control of live video stream(s)
GB2509953B (en) * 2013-01-18 2015-05-20 Canon Kk Method of displaying a region of interest in a video stream
US9196199B2 (en) * 2013-02-12 2015-11-24 Pixtronix, Inc. Display having staggered display element arrangement
CN103702139B (zh) * 2013-12-13 2017-02-01 华中科技大学 一种移动环境下基于可扩展编码的视频点播系统
CN105025351B (zh) * 2014-04-30 2018-06-29 深圳Tcl新技术有限公司 流媒体播放器缓冲的方法及装置
JP6337614B2 (ja) * 2014-05-23 2018-06-06 セイコーエプソン株式会社 制御装置、ロボット、及び制御方法
US9918136B2 (en) * 2014-05-29 2018-03-13 Nextvr Inc. Methods and apparatus for delivering content and/or playing back content
CN104240281A (zh) * 2014-08-28 2014-12-24 东华大学 一种基于Unity3D引擎的虚拟现实头戴式设备
US10812546B2 (en) * 2014-12-24 2020-10-20 Intel IP Corporation Link-aware streaming adaptation
US20160261908A1 (en) * 2015-03-05 2016-09-08 Htc Corporation Media streaming system and control method thereof
CN104735464A (zh) * 2015-03-31 2015-06-24 华为技术有限公司 一种全景视频交互传输方法、服务器和客户端
CN104717507A (zh) * 2015-03-31 2015-06-17 北京奇艺世纪科技有限公司 一种视频转码方法及装置
US10083363B2 (en) * 2015-05-26 2018-09-25 Nbcuniversal Media, Llc System and method for customizing content for a user
JP6866297B2 (ja) * 2015-06-12 2021-04-28 グーグル エルエルシーＧｏｏｇｌｅ ＬＬＣ ヘッドマウントディスプレイの電子ディスプレイ安定化
US9956054B2 (en) * 2015-06-25 2018-05-01 EchoPixel, Inc. Dynamic minimally invasive surgical-aware assistant
CN108476324B (zh) * 2015-10-08 2021-10-29 皇家Kpn公司 增强视频流的视频帧中的感兴趣区域的方法、计算机和介质
CN106919248A (zh) * 2015-12-26 2017-07-04 华为技术有限公司 应用于虚拟现实的内容传输方法以及设备
CN105455285B (zh) * 2015-12-31 2019-02-12 北京小鸟看看科技有限公司 一种虚拟现实头盔适配方法
US10412130B2 (en) * 2016-04-04 2019-09-10 Hanwha Techwin Co., Ltd. Method and apparatus for playing media stream on web browser
US10313417B2 (en) * 2016-04-18 2019-06-04 Qualcomm Incorporated Methods and systems for auto-zoom based adaptive video streaming
CN105898565A (zh) * 2016-04-28 2016-08-24 乐视控股（北京）有限公司 一种视频处理方法及设备
US9721393B1 (en) * 2016-04-29 2017-08-01 Immersive Enterprises, LLC Method for processing and delivering virtual reality content to a user
CN106060570B (zh) * 2016-06-30 2019-06-14 北京奇艺世纪科技有限公司 一种全景视频图像播放、编码方法及装置
CN106060515B (zh) * 2016-07-14 2018-11-06 腾讯科技（深圳）有限公司 全景媒体文件推送方法及装置
CN106231317A (zh) * 2016-09-29 2016-12-14 三星电子（中国）研发中心 视频处理、解码方法和装置、vr终端、视频播放系统
US10595069B2 (en) * 2016-12-05 2020-03-17 Adobe Inc. Prioritizing tile-based virtual reality video streaming using adaptive rate allocation
US20180295375A1 (en) * 2017-04-05 2018-10-11 Lyrical Labs Video Compression Technology, LLC Video processing and encoding
CN107087212B (zh) * 2017-05-09 2019-10-29 杭州码全信息科技有限公司 基于空间可伸缩编码的交互式全景视频转码与播放方法及系统

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120256954A1 (en) * 2011-04-08 2012-10-11 Patrick Soon-Shiong Interference Based Augmented Reality Hosting Platforms
CN103390287A (zh) * 2012-05-11 2013-11-13 索尼电脑娱乐欧洲有限公司 用于增强现实的装置和方法
US20150356790A1 (en) * 2012-10-24 2015-12-10 Exelis Inc. Augmented Reality Control Systems
US20150195489A1 (en) * 2014-01-06 2015-07-09 Arun Sobti & Associates, Llc System and apparatus for smart devices based conferencing
CN104268922A (zh) * 2014-09-03 2015-01-07 广州博冠信息科技有限公司 一种图像渲染方法及图像渲染装置
CN104616243A (zh) * 2015-01-20 2015-05-13 北京大学 一种高效的gpu三维视频融合绘制方法
CN105916022A (zh) * 2015-12-28 2016-08-31 乐视致新电子科技（天津）有限公司 一种基于虚拟现实技术的视频图像处理方法及装置

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111489428A (zh) * 2020-04-20 2020-08-04 北京字节跳动网络技术有限公司 图像生成方法、装置、电子设备及计算机可读存储介质
CN111489428B (zh) * 2020-04-20 2023-06-30 北京字节跳动网络技术有限公司 图像生成方法、装置、电子设备及计算机可读存储介质
CN111754614A (zh) * 2020-06-30 2020-10-09 平安国际智慧城市科技股份有限公司 基于vr视频渲染方法、装置、电子设备及存储介质
CN115396731A (zh) * 2022-08-10 2022-11-25 北京势也网络技术有限公司 全景视频的播放方法、装置、电子设备及可读存储介质

Also Published As

Publication number Publication date
US20180192063A1 (en) 2018-07-05
US20180192022A1 (en) 2018-07-05
US20180191868A1 (en) 2018-07-05
US20180192044A1 (en) 2018-07-05
CN108377381A (zh) 2018-08-07
US20180192026A1 (en) 2018-07-05
CN108419093A (zh) 2018-08-17
CN108419142A (zh) 2018-08-17
CN108366293A (zh) 2018-08-03
US20180189980A1 (en) 2018-07-05
US10334238B2 (en) 2019-06-25
CN108391103A (zh) 2018-08-10

Similar Documents

Publication Publication Date Title
CN108366272A (zh) Vr视频渲染方法及装置
CN107590771B (zh) 具有用于在建模3d空间中投影观看的选项的2d视频
EP1883052B1 (en) Generating images combining real and virtual images
CN107362532B (zh) 视频游戏的方向输入
CN108616557A (zh) 一种全景视频传输方法、装置、终端、服务器及系统
GB2481366A (en) 3D interactive display and pointer control
CN103918012A (zh) 渲染系统、渲染服务器、其控制方法、程序及记录媒体
US20180330648A1 (en) Multi-window smart content rendering and optimizing method and projection method based on cave system
TW202029742A (zh) 影像合成
US20230290043A1 (en) Picture generation method and apparatus, device, and medium
TW202101374A (zh) 影像的深度圖之處理
CN109328462A (zh) 一种用于流视频内容的方法及装置
JP2008077372A (ja) 画像処理装置、画像処理装置の制御方法及びプログラム
CN113206993A (zh) 一种调整显示屏幕的方法及显示设备
CN111080704A (zh) 视频增强现实的方法及装置
CN115187729A (zh) 三维模型生成方法、装置、设备和存储介质
WO2019137821A1 (en) Apparatus and method for generating view images
CN108230430B (zh) 云层遮罩图的处理方法及装置
JP7479386B2 (ja) シーンを表す画像信号
KR101773929B1 (ko) 광 시야각 영상 처리 시스템, 광 시야각 영상의 전송 및 재생 방법, 및 이를 위한 컴퓨터 프로그램
JP2001283244A (ja) ３次元画像合成装置及び方法、情報記憶媒体、プログラム配信装置及び方法
KR102491939B1 (ko) 전방향 이미지를 프로세싱하기 위한 방법 및 장치
US11900136B2 (en) AoT compiler for a legacy game
CN114779981B (zh) 全景视频中可拖拽式热点交互方法、系统及存储介质
US20060267979A1 (en) Apparatus and method for processing three-dimensional images

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20180803

WD01 Invention patent application deemed withdrawn after publication