GB2557416B - Prioritizing tile-based virtual reality video streaming using adaptive rate allocation - Google Patents

Prioritizing tile-based virtual reality video streaming using adaptive rate allocation Download PDF

Info

Publication number
GB2557416B
GB2557416B GB1716286.8A GB201716286A GB2557416B GB 2557416 B GB2557416 B GB 2557416B GB 201716286 A GB201716286 A GB 201716286A GB 2557416 B GB2557416 B GB 2557416B
Authority
GB
United Kingdom
Prior art keywords
prioritizing
tile
virtual reality
video streaming
based virtual
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
GB1716286.8A
Other versions
GB2557416A (en
GB201716286D0 (en
Inventor
Swaminathan Viswanathan
Hosseini Mohammad
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Adobe Inc
Original Assignee
Adobe Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US15/593,050 external-priority patent/US10595069B2/en
Application filed by Adobe Inc filed Critical Adobe Inc
Publication of GB201716286D0 publication Critical patent/GB201716286D0/en
Publication of GB2557416A publication Critical patent/GB2557416A/en
Application granted granted Critical
Publication of GB2557416B publication Critical patent/GB2557416B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
    • H04N21/2662—Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/70—Admission control; Resource allocation
    • H04L47/80—Actions related to the user profile or the type of traffic
    • H04L47/805—QOS or priority aware
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/61—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio
    • H04L65/612—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio for unicast
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/70—Media network packetisation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/75—Media network packet handling
    • H04L65/752—Media network packet handling adapting media to network capabilities
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/75—Media network packet handling
    • H04L65/765—Media network packet handling intermediate
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/80—Responding to QoS
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/194—Transmission of image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/115—Selection of the code volume for a coding unit prior to coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
    • H04N19/33—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability in the spatial domain
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
    • H04N19/37—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability with arrangements for assigning different transmission priorities to video input data or to video coded data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/21805—Source of audio or video content, e.g. local disk arrays enabling multiple viewpoints, e.g. using a plurality of cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/23439—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements for generating different versions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/262—Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists
    • H04N21/26208—Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists the scheduling operation being performed under constraints
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
    • H04N21/462—Content or additional data management, e.g. creating a master electronic program guide from data received from the Internet and a Head-end, controlling the complexity of a video stream by scaling the resolution or bit-rate based on the client capabilities
    • H04N21/4621—Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
    • H04N21/4728—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • H04N21/647—Control signaling between network components and server or clients; Network processes for video distribution between server and clients, e.g. controlling the quality of the video stream, by dropping packets, protecting content from unauthorised alteration within the network, monitoring of network load, bridging between two different networks, e.g. between IP and wireless
    • H04N21/64723—Monitoring of network processes or resources, e.g. monitoring of network load
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/65—Transmission of management data between client and server
    • H04N21/658—Transmission by the client directed to the server
    • H04N21/6587—Control parameters, e.g. trick play commands, viewpoint selection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/845—Structuring of content, e.g. decomposing content into time segments
    • H04N21/8456—Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Databases & Information Systems (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • Computer Security & Cryptography (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Information Transfer Between Computers (AREA)
GB1716286.8A 2016-12-05 2017-10-05 Prioritizing tile-based virtual reality video streaming using adaptive rate allocation Active GB2557416B (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201662430313P 2016-12-05 2016-12-05
US15/593,050 US10595069B2 (en) 2016-12-05 2017-05-11 Prioritizing tile-based virtual reality video streaming using adaptive rate allocation

Publications (3)

Publication Number Publication Date
GB201716286D0 GB201716286D0 (en) 2017-11-22
GB2557416A GB2557416A (en) 2018-06-20
GB2557416B true GB2557416B (en) 2021-08-04

Family

ID=60326931

Family Applications (1)

Application Number Title Priority Date Filing Date
GB1716286.8A Active GB2557416B (en) 2016-12-05 2017-10-05 Prioritizing tile-based virtual reality video streaming using adaptive rate allocation

Country Status (3)

Country Link
CN (1) CN108156484B (en)
DE (1) DE102017009121A1 (en)
GB (1) GB2557416B (en)

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10567780B2 (en) 2018-06-14 2020-02-18 Telefonaktiebolaget Lm Ericsson (Publ) System and method for encoding 360° immersive video
US10419738B1 (en) 2018-06-14 2019-09-17 Telefonaktiebolaget Lm Ericsson (Publ) System and method for providing 360° immersive video based on gaze vector information
US10623736B2 (en) 2018-06-14 2020-04-14 Telefonaktiebolaget Lm Ericsson (Publ) Tile selection and bandwidth optimization for providing 360° immersive video
US10523914B1 (en) * 2018-07-26 2019-12-31 Telefonaktiebolaget Lm Ericsson (Publ) System and method for providing multiple 360° immersive video sessions in a network
US10841662B2 (en) 2018-07-27 2020-11-17 Telefonaktiebolaget Lm Ericsson (Publ) System and method for inserting advertisement content in 360° immersive video
US10735778B2 (en) 2018-08-23 2020-08-04 At&T Intellectual Property I, L.P. Proxy assisted panoramic video streaming at mobile edge
CN108924629B (en) * 2018-08-28 2021-01-05 恒信东方文化股份有限公司 VR image processing method
US10757389B2 (en) 2018-10-01 2020-08-25 Telefonaktiebolaget Lm Ericsson (Publ) Client optimization for providing quality control in 360° immersive video during pause
US10939139B2 (en) 2018-11-29 2021-03-02 Apple Inc. Adaptive coding and streaming of multi-directional video
US11956295B2 (en) 2019-09-27 2024-04-09 Apple Inc. Client-end enhanced view prediction for multi-view video streaming exploiting pre-fetched data and side information
US20230011586A1 (en) * 2019-12-09 2023-01-12 Telefonaktiebolaget Lm Ericsson (Publ) Electronic device, server and methods for viewport prediction based on head and eye gaze
CN111586414B (en) * 2020-04-07 2022-04-15 南京师范大学 SVC and DASH-based 360-degree video stream scheduling method
EP3926441B1 (en) * 2020-06-15 2024-02-21 Nokia Technologies Oy Output of virtual content
CN112584119B (en) * 2020-11-24 2022-07-22 鹏城实验室 Self-adaptive panoramic video transmission method and system based on reinforcement learning
CN113362438A (en) * 2021-06-30 2021-09-07 北京百度网讯科技有限公司 Panorama rendering method, device, electronic apparatus, medium, and program

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150249813A1 (en) * 2014-03-03 2015-09-03 Next3D, Inc. Methods and apparatus for streaming content
WO2015184416A1 (en) * 2014-05-29 2015-12-03 Nextvr Inc. Methods and apparatus for delivering content and/or playing back content
US20150346832A1 (en) * 2014-05-29 2015-12-03 Nextvr Inc. Methods and apparatus for delivering content and/or playing back content
WO2016138043A1 (en) * 2015-02-24 2016-09-01 NextVR, Inc. Calibration for immersive content systems

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN100542298C (en) * 2007-09-29 2009-09-16 中国科学院计算技术研究所 A kind of block size computing method of cylinder panorama video code and transmission method
US20120293607A1 (en) * 2011-05-17 2012-11-22 Apple Inc. Panorama Processing
CN105791882B (en) * 2016-03-22 2018-09-18 腾讯科技（深圳）有限公司 Method for video coding and device
CN106162207B (en) * 2016-08-25 2019-02-12 北京字节跳动科技有限公司 A kind of panoramic video parallel encoding method and apparatus

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150249813A1 (en) * 2014-03-03 2015-09-03 Next3D, Inc. Methods and apparatus for streaming content
WO2015184416A1 (en) * 2014-05-29 2015-12-03 Nextvr Inc. Methods and apparatus for delivering content and/or playing back content
US20150346832A1 (en) * 2014-05-29 2015-12-03 Nextvr Inc. Methods and apparatus for delivering content and/or playing back content
WO2016138043A1 (en) * 2015-02-24 2016-09-01 NextVR, Inc. Calibration for immersive content systems

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
M Hosseini et al, "Adaptive 360 VR Video Streaming: Divide and Conquer!", Published September 2016, arXiv, Available from https://arxiv.org/abs/1609.08729v1 *

Also Published As

Publication number Publication date
CN108156484A (en) 2018-06-12
GB2557416A (en) 2018-06-20
GB201716286D0 (en) 2017-11-22
DE102017009121A1 (en) 2018-06-07
CN108156484B (en) 2022-01-14

Similar Documents

Publication Publication Date Title
GB2557416B (en) Prioritizing tile-based virtual reality video streaming using adaptive rate allocation
SG11202004402XA (en) Interactive video game system
IL271902A (en) Virtual reality apparatus
EP3459253C0 (en) Circular fisheye video in virtual reality
GB2553446B (en) Entity based temporal segmentation of video streams
HK1225539A1 (en) Interactive video conferencing
GB201702693D0 (en) Virtual reality
SG11201709253YA (en) Virtual reality experience device
SG11202004400RA (en) Interactive video game system
GB2563276B (en) Virtual reality systems
GB2560004B (en) Virtual reality
SG11201911331QA (en) Enhanced high-level signaling for fisheye virtual reality video
GB201712569D0 (en) Virtual reality video processing
EP3164963A4 (en) Priority of uplink streams in video switching
GB201620985D0 (en) Virtual reality
GB2546589B (en) Virtual training system
GB2558284B (en) Virtual reality
EP3170092A4 (en) Virtual high definition video player
GB201503831D0 (en) Virtual trying on experience
SG11202000031RA (en) Virtual office
GB201802888D0 (en) Virtual reality
GB201706701D0 (en) Virtual reality viewpoint adaption
HK1211403A1 (en) Virtual video patrol system and components therefor
PL3107275T3 (en) Virtual queuing system
GB202003904D0 (en) Status streaming pop-up map