WO2019115153A1 - Method, apparatus, system and computer readable medium to transmit data stream - Google Patents

Method, apparatus, system and computer readable medium to transmit data stream Download PDF

Info

Publication number
WO2019115153A1
WO2019115153A1 PCT/EP2018/081634 EP2018081634W WO2019115153A1 WO 2019115153 A1 WO2019115153 A1 WO 2019115153A1 EP 2018081634 W EP2018081634 W EP 2018081634W WO 2019115153 A1 WO2019115153 A1 WO 2019115153A1
Authority
WO
WIPO (PCT)
Prior art keywords
data stream
view
union
requested
requested field
Prior art date
Application number
PCT/EP2018/081634
Other languages
French (fr)
Inventor
Elangovan MANICKAM
Sandeep RK
Original Assignee
Nokia Solutions And Networks Oy
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nokia Solutions And Networks Oy filed Critical Nokia Solutions And Networks Oy
Publication of WO2019115153A1 publication Critical patent/WO2019115153A1/en

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
    • H04N21/4728—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/61—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio
    • H04L65/612—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio for unicast
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/21805—Source of audio or video content, e.g. local disk arrays enabling multiple viewpoints, e.g. using a plurality of cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234345—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/65—Transmission of management data between client and server
    • H04N21/658—Transmission by the client directed to the server
    • H04N21/6587—Control parameters, e.g. trick play commands, viewpoint selection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding

Definitions

  • the present subject matter relates to the transmission of 360 AR/VR data from a server to an end-user apparatus.
  • AR we mean augmented reality, who is a live direct or indirect view of a physical, real-world environment whose elements are "augmented” by computer generated or extracted real-world sensory input such as sound, video, graphics, haptics or GPS data.
  • VR virtual reality who is a computer technology that uses virtual reality headsets or multi -projected environments, sometimes in combination with physical environments or props, to generate realistic images, sounds and other sensations that simulate a user's physical presence in a virtual or imaginary environment.
  • 360 video also known as immersive videos or spherical videos we mean video recordings where a view in every direction is recorded at the same time, shot using an omnidirectional camera or a collection of cameras. During playback, the viewer has control of the viewing direction like a panorama.
  • the present subject matter has the advantages of optimizing and synchronizing the transmission of 360 AR/VR Data traffic from a server to end user apparatus. These optimization and synchronization allow the reduction of this overhead of data and reduce the throughput of the data transmitted between the server and the end user apparatus.
  • a method to transmit data stream comprises a step of receiving at least two requests of the data stream, the requests being associated to a requested field of view and a step of determining a union of at least two of the requested field of view and a step of requesting the data stream presented according to the union of the requested field of view and a step of extracting at least one relevant part of the received data stream corresponding to one of the requested field of view and a step of transmitting the relevant part of the received data stream.
  • an apparatus comprising a module for receiving, at least two requests of the data stream, the requests being associated to a requested field of view and a module for determining a union of at least two of the requested field of view and a module for requesting the data stream presented according to the union of the requested field of view and a module for extracting at least one relevant part of the received data stream corresponding to one of the requested field of view and a module for transmitting the relevant part of the received data stream.
  • a system is described. The system comprises an end user apparatus, the apparatus described in the section above.
  • a computer-readable medium has embodied thereon a computer program configured to realize a method to transmit a data stream.
  • This method comprises a step of receiving at least two requests of the data stream, the requests being associated to a requested field of view and a step of determining a union of at least two of the requested field of view and a step of requesting the data stream presented according to the union of the requested field of view and a step of extracting at least one relevant part of the received data stream corresponding to one of the requested field of view and a step of transmitting the relevant part of the received data stream.
  • Figure 1 presents a first embodiment of the method of the present subject matter.
  • Figures 2-a and 2-b present a 360/AR/VR field of view
  • Figure 3 presents an apparatus of the present subject matter.
  • Figure 4 presents a system of the present subject matter.
  • the figure 1 presents a first embodiment of the method of the present subject matter. This method allows the transmission of a data stream. This method comprises:
  • a step 105 of transmitting the relevant part of the received data stream is configured to transmit the data stream between a server and at least two end user apparatus.
  • the data stream is a 360 AR/VR data stream.
  • the step 101 of receiving the two requests of the data stream is configured to receive these requests from at least two end user apparatus, furthermore within this embodiment each request being associated to one of the end user apparatus.
  • the step 103 of requesting the data stream is configured to request this data stream to a server and this step is further configured to receive, from the server, this data stream presented according the union of the requested field of view.
  • the step 105 of transmitting the relevant part of the received data stream is configured to transmit this relevant part of the received data stream to the end user apparatus associated to the requested field of view.
  • the figure 2 presents a 360 Field of View (FOV). This Field of View can be divided in three parts: ⁇ Primary field of view (201): This covers the area of the primary focus and vision
  • Blind spot (203) This area forms the area which is not visible to the user at a given time.
  • the user shall view the one direction of the data. Based on the above field of view definition it is not required to transmit the data for the entire spectrum. This can be optimized by differential grading across the field of view range.
  • the Primary FoV can have the content with the highest quality as it is discernible to the user.
  • the secondary FoV can be of lower resolution as this just forms the peripheral vision of the user. And the Blind spot information can be totally omitted.
  • the data stream and more precisely the 360/VR/AR data is represented according to a plurality of segments.
  • Each segment presents a part of the view in every direction recorded at the same time within the data stream.
  • Each segment is parameter by a center axial and an angle range.
  • the Field of view of a user is constituted by a sub set of the segments of the complete set of segment presenting the view in all direction. More precisely this sub set of segments data which forms the area of interest of the user.
  • the step 102 of determining a union of at least two of the requested field of view is configured to determine segments constituting the two requested fields of view and wherein the union of the fields of view is constituting of the previously determined segments.
  • the union of the requested field of views is constituted of the segments, that belong at least to one of the field of view requested by the user.
  • the step 104 of extracting a relevant part of the received data stream is configured to determine segments constituting the requested field of view and wherein the relevant part of the received data stream according to the concatenation of these determined segments.
  • the extraction of the relevant video segments is also done based on the segments requested from each user as part of the Field of view requests.
  • the overlapping data is replicated towards each of the users and the union data is only buffered at the edge.
  • the step 102 of determining the union of the requested field of view is configured to use meta data indicating the requested data streams and/or meta data indicating the requested fields of view.
  • the step 103 of requesting and receiving the data stream is configured to use meta data to indicate the union of the requested field of view.
  • a method of predictive data flow optimization is proposed. With increased understanding of the content of the data and machine learning aspects of user movement, it shall be possible to predict user movement. This information which can be provided as meta data with the video and/or leant from the user behavior is used to pre-sync data based on predicted user movement. Such optimization increases user experience by providing a seamless experience without lag.
  • the data agents will perform predictive analysis at the device and the network based on the machine learning concepts of user behavior [head rotation etc.] and the meta information from the content itself. [0031] This information will to determine the most optimized segment of Field of view to be transmitted to the user equipment. Hence the next field of view of the user is already predicted and transmitted to the Edge and to the user equipment.
  • FIG 3 an embodiment of an apparatus of the present subject matter.
  • This apparatus comprises:
  • the apparatus is configured to transmit the data stream between a server and at least two end user apparatus.
  • the data stream is a 360 AR/VR data stream.
  • the module 301 for receiving the two requests ofthe data stream is configured to receive these requests from at least two end user apparatus, furthermore within this embodiment each request being associated to one of the end user apparatus.
  • the module 303 for requesting the data stream is configured to request this data stream to a server and this step is further configured to receive, from the server, this data stream presented according the union of the requested field of view.
  • the module 305 for transmitting the relevant part of the received data stream is configured to transmit this relevant part of the received data stream to the end user apparatus associated to the requested field of view.
  • the module 302 for determining the union of the requested field of view is configured to use meta data indicating the requested data streams and/or meta data indicating the requested fields of view.
  • the module 303 for requesting and receiving the data stream is configured to use meta data to indicate the union of the requested field of view.
  • the apparatus is an edge data agent.
  • the module 302 for determining a union of at least two of the requested field of view is configured to determine segments constituting the two requested fields of view and wherein the union of the fields of view is constituting of the previously determined segments.
  • the union of the requested field of views is constituted of the segments, that belong at least to one of the field of view requested by the user.
  • the module 304 for extracting a relevant part of the received data stream is configured to determine segments constituting the requested field of view and wherein the relevant part of the received data stream according to the concatenation of these determined segments.
  • the extraction of the relevant video segments is also done based on the segments requested from each user as part of the Field of view requests.
  • the overlapping data is replicated towards each of the users and the union data is only buffered at the edge.
  • This apparatus comprises a processor or a processor circuitry or a microprocessor.
  • This apparatus also comprises memory and a software integrated within this memory, this memory plus software configure to, with the processer or processor circuitry or microprocessor, cause the apparatus to receive, at least two requests of the data stream the requests being associated to a requested field of view and
  • the modules will be realized by one or more processors and one or more memory.
  • the module may also be realized using dedicated hardware as well as hardware capable of executing software in association with appropriate software.
  • the modules When provided by a processor, the modules may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared.
  • explicit use of the term“processor” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), read only memory (ROM) for storing software, random access memory (RAM), and non- volatile storage.
  • DSP digital signal processor
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • ROM read only memory
  • RAM random access memory
  • non- volatile storage Other hardware, conventional and/or custom, may also be included.
  • the memory may include any computer-readable medium known in the art including, for example, volatile memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM), and/or non-volatile memory, such as read only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes.
  • volatile memory such as static random access memory (SRAM) and dynamic random access memory (DRAM)
  • non-volatile memory such as read only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes.
  • the memory includes modules and data.
  • the modules include routines, programs, objects, components, data structures, etc., which perform tasks or implement abstract data types.
  • the data serves as a repository for storing data processed, received, and generated by one or more of the modules.
  • This cloud data server comprises a module for receiving a request of data stream presented per the union of the requested field of view and module for transmitting this data stream presented per the union of the requested field of view.
  • this cloud data server comprises a processor or a processor circuitry or a microprocessor.
  • the cloud data server also comprises memory and a software integrated within this memory, this memory plus software configure to, with the processer or processor circuitry or microprocessor, cause the cloud data server to:
  • This end user apparatus could be for example a virtual reality head set.
  • This end user apparatus comprises a module for transmitting a request of a data stream and an associated indication of a requested field of view and a module for receiving a relevant part of a data stream.
  • this end user apparatus comprises a processor or a processor circuitry or a microprocessor.
  • the end user apparatus also comprises memory and a software integrated within this memory, this memory plus software configure to, with the processer or processor circuitry or microprocessor, cause the end user apparatus to
  • the figure 4 presents an embodiment of a system of the present subject matter.
  • This system comprises end user apparatus, for example virtual reality headset, edge data agents, configured to realize the method of the figure 2 and cloud data server, comprising a set of data stream.
  • the edge Data agents are configured to obtain form the could data server an aggregation of streamed content. To realize this if a video stream is being viewed by multiple end user apparatus under a common edge data agent the edge date agent uses field of view information from multiple sources and to realize an aggregate request at destination of the cloud data server.
  • This aggregation allows a reduction of the data which is being requested from the cloud data server. Indeed, the edge data agent downloads a single copy of the data which is restricted to a union of the field of view requests.
  • the edge data agent is also configured to split this video based on the request streams towards individual devices.
  • a movement of the field of view of a user is handled locally from the edge data agent itself.
  • the edge data agent shall request the cloud data server for this not covered part of the field of view, or vice versa.
  • two end user apparatus are requesting segment of 360 data based on their field of view.
  • the end user apparatus send this request for the field of view to the Edge Data Agent.
  • the Edge Data Agent uses the meta data to determine the commonality of the request and commonality in the field of view.
  • This unified field of view which a subset of the 360 data is requested from the backend cloud which has the complete data.
  • the Backend Data Agent checks the meta data to determine the field of view to be returned. It extracts the relevant section and streams it to the Edge until it receives a change in the field of view.
  • the Edge Data agent splits the field of view with duplication to the two end user apparatus.
  • program storage devices for example, digital data storage media, which are machine or computer readable and encode machine-executable or computer-executable programs of instructions, where said instructions perform some or all of the steps of the described method.
  • the program storage devices may be, for example, digital memories, magnetic storage media, such as magnetic disks and magnetic tapes, hard drives, or optically readable digital data storage media.
  • Another embodiment of the present subject matter is a computer-readable medium having embodied thereon a computer program configured to realize the method, presented figure 1.
  • This method allows the transmission of a data stream. This method comprises:
  • the method is configured to transmit the data stream between a server and at least two end user apparatus.
  • the data stream is a 360 AR/VR data stream.
  • the step 101 of receiving the two requests of the data stream is configured to receive these requests from at least two end user apparatus, furthermore within this embodiment each request being associated to one of the end user apparatus.
  • the step 103 of requesting the data stream is configured to request this data stream to a server and this step is further configured to receive, from the server, this data stream presented according the union of the requested field of view.
  • the step 105 of transmitting the relevant part of the received data stream is configured to transmit this relevant part of the received data stream to the end user apparatus associated to the requested field of view.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Databases & Information Systems (AREA)
  • Human Computer Interaction (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

Method, apparatus, system and computer readable medium to transmit data stream is described. This method comprises a step of receiving at least two requests of the data stream, the requests being associated to a requested field of view and a step of determining a union of at least two of the requested field of view and a step of requesting the data stream presented according to the union of the requested field of view and a step of extracting at least one relevant part of the received data stream corresponding to one of the requested field of view and a step of transmitting the relevant part of the received data stream.

Description

METHOD, APPARATUS, SYSTEM AND COMPUTER READABLE MEDIUM TO TRANSMIT DATA STREAM
FIELD OF INVENTION
[0001] The present subject matter relates to the transmission of 360 AR/VR data from a server to an end-user apparatus. By AR we mean augmented reality, who is a live direct or indirect view of a physical, real-world environment whose elements are "augmented" by computer generated or extracted real-world sensory input such as sound, video, graphics, haptics or GPS data. By VR we mean virtual reality who is a computer technology that uses virtual reality headsets or multi -projected environments, sometimes in combination with physical environments or props, to generate realistic images, sounds and other sensations that simulate a user's physical presence in a virtual or imaginary environment. By 360 video, also known as immersive videos or spherical videos we mean video recordings where a view in every direction is recorded at the same time, shot using an omnidirectional camera or a collection of cameras. During playback, the viewer has control of the viewing direction like a panorama. BACKGROUND
[0002] Currently the streaming of data generated by VR devices/360 video is sub optimal.
Either the complete data is streamed or a part of data gets streamed. Also, the infrastructure and operator network is not aware of this content and is not optimized for transmission of this kind of data. So, using the system of the state of the art there is an overhead of the data transmitted to the end-user apparatus.
[0003] The present subject matter has the advantages of optimizing and synchronizing the transmission of 360 AR/VR Data traffic from a server to end user apparatus. These optimization and synchronization allow the reduction of this overhead of data and reduce the throughput of the data transmitted between the server and the end user apparatus. SUMMARY
[0004] This summary is provided to introduce concepts related to the transmission 360
AR/VR data from a server to an end-user apparatus. [0005] In one implementation, a method to transmit data stream is described. This method comprises a step of receiving at least two requests of the data stream, the requests being associated to a requested field of view and a step of determining a union of at least two of the requested field of view and a step of requesting the data stream presented according to the union of the requested field of view and a step of extracting at least one relevant part of the received data stream corresponding to one of the requested field of view and a step of transmitting the relevant part of the received data stream.
[0006] In one implementation, an apparatus is described. This apparatus comprises a module for receiving, at least two requests of the data stream, the requests being associated to a requested field of view and a module for determining a union of at least two of the requested field of view and a module for requesting the data stream presented according to the union of the requested field of view and a module for extracting at least one relevant part of the received data stream corresponding to one of the requested field of view and a module for transmitting the relevant part of the received data stream. [0007] In one implementation, a system is described. The system comprises an end user apparatus, the apparatus described in the section above.
[0008] In one implementation, a computer-readable medium is described. This computer- readable medium has embodied thereon a computer program configured to realize a method to transmit a data stream. This method comprises a step of receiving at least two requests of the data stream, the requests being associated to a requested field of view and a step of determining a union of at least two of the requested field of view and a step of requesting the data stream presented according to the union of the requested field of view and a step of extracting at least one relevant part of the received data stream corresponding to one of the requested field of view and a step of transmitting the relevant part of the received data stream. BRIEF DESCRIPTION OF THE FIGURES
[0009] The detailed description is given with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The same numbers are used throughout the figures to reference like features and components. Some embodiments of a system and/or methods in accordance with embodiments of the present subject matter are now described, by way of example only, and with reference to the accompanying figures, in which:
[0010] Figure 1 presents a first embodiment of the method of the present subject matter.
[0011] Figures 2-a and 2-b present a 360/AR/VR field of view
[0012] Figure 3 presents an apparatus of the present subject matter.
[0013] Figure 4 presents a system of the present subject matter.
[0014] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.
[0015] It should be appreciated by those skilled in the art that any block diagrams herein represent conceptual views of illustrative systems embodying the principles of the present subject matter. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.
DESCRIPTION OF EMBODIMENTS
[0016] The figure 1 presents a first embodiment of the method of the present subject matter. This method allows the transmission of a data stream. This method comprises:
A step 101 of receiving at least two requests of the data stream, the requests being associated to a requested field of view and
A step 102 of determining a union of at least two of the requested field of view and A step 103 of requesting the data stream presented according to the union of the requested field of view and
A step 104 of extracting at least one relevant part of the received data stream corresponding to one of the requested field of view and
A step 105 of transmitting the relevant part of the received data stream. [0017] In an embodiment, the method is configured to transmit the data stream between a server and at least two end user apparatus.
[0018] In an embodiment, the data stream is a 360 AR/VR data stream.
[0019] In an embodiment, the step 101 of receiving the two requests of the data stream is configured to receive these requests from at least two end user apparatus, furthermore within this embodiment each request being associated to one of the end user apparatus.
[0020] In an embodiment, the step 103 of requesting the data stream, is configured to request this data stream to a server and this step is further configured to receive, from the server, this data stream presented according the union of the requested field of view.
[0021] In an embodiment, the step 105 of transmitting the relevant part of the received data stream, is configured to transmit this relevant part of the received data stream to the end user apparatus associated to the requested field of view.
[0022] The figure 2 presents a 360 Field of View (FOV). This Field of View can be divided in three parts: · Primary field of view (201): This covers the area of the primary focus and vision
• Secondary field of view (202-a & 202 -b): This forms the peripheral vision of the user and usually is not in focus.
• Blind spot (203): This area forms the area which is not visible to the user at a given time.
The user shall view the one direction of the data. Based on the above field of view definition it is not required to transmit the data for the entire spectrum. This can be optimized by differential grading across the field of view range. The Primary FoV can have the content with the highest quality as it is discernible to the user. The secondary FoV can be of lower resolution as this just forms the peripheral vision of the user. And the Blind spot information can be totally omitted.
The data stream and more precisely the 360/VR/AR data is represented according to a plurality of segments. Each segment presents a part of the view in every direction recorded at the same time within the data stream. Each segment is parameter by a center axial and an angle range. The Field of view of a user is constituted by a sub set of the segments of the complete set of segment presenting the view in all direction. More precisely this sub set of segments data which forms the area of interest of the user. [0023] In an embodiment of the present subject matter the step 102 of determining a union of at least two of the requested field of view, is configured to determine segments constituting the two requested fields of view and wherein the union of the fields of view is constituting of the previously determined segments.
[0024] In other words, within the above presented embodiment, the union of the requested field of views is constituted of the segments, that belong at least to one of the field of view requested by the user.
[0025] In an embodiment of the present subject matter the step 104 of extracting a relevant part of the received data stream is configured to determine segments constituting the requested field of view and wherein the relevant part of the received data stream according to the concatenation of these determined segments.
[0026] In other words, within the above presented embodiment the extraction of the relevant video segments is also done based on the segments requested from each user as part of the Field of view requests. The overlapping data is replicated towards each of the users and the union data is only buffered at the edge.
[0027] In another embodiment, the step 102 of determining the union of the requested field of view is configured to use meta data indicating the requested data streams and/or meta data indicating the requested fields of view.
[0028] In another embodiment, the step 103 of requesting and receiving the data stream is configured to use meta data to indicate the union of the requested field of view.
[0029] In an embodiment of the present subject matter a method of predictive data flow optimization is proposed. With increased understanding of the content of the data and machine learning aspects of user movement, it shall be possible to predict user movement. This information which can be provided as meta data with the video and/or leant from the user behavior is used to pre-sync data based on predicted user movement. Such optimization increases user experience by providing a seamless experience without lag.
[0030] Within this embodiment the data agents will perform predictive analysis at the device and the network based on the machine learning concepts of user behavior [head rotation etc.] and the meta information from the content itself. [0031] This information will to determine the most optimized segment of Field of view to be transmitted to the user equipment. Hence the next field of view of the user is already predicted and transmitted to the Edge and to the user equipment.
[0032] The figure 3 an embodiment of an apparatus of the present subject matter. This apparatus comprises:
A module 301 for receiving, at least two requests of the data stream, the requests being associated to a requested field of view and
A module 302 for determining a union of at least two of the requested field of view and A module 303 for requesting the data stream presented according to the union of the requested field of view and
A module 304 for extracting at least one relevant part of the received data stream corresponding to one of the requested field of view and
A module 305 for transmitting the relevant part of the received data stream.
[0033] In an embodiment, the apparatus is configured to transmit the data stream between a server and at least two end user apparatus.
[0034] In an embodiment, the data stream is a 360 AR/VR data stream.
[0035] In an embodiment, the module 301 for receiving the two requests ofthe data stream is configured to receive these requests from at least two end user apparatus, furthermore within this embodiment each request being associated to one of the end user apparatus.
[0036] In an embodiment, the module 303 for requesting the data stream, is configured to request this data stream to a server and this step is further configured to receive, from the server, this data stream presented according the union of the requested field of view.
[0037] In an embodiment, the module 305 for transmitting the relevant part of the received data stream, is configured to transmit this relevant part of the received data stream to the end user apparatus associated to the requested field of view.
[0038] In another embodiment, the module 302 for determining the union of the requested field of view is configured to use meta data indicating the requested data streams and/or meta data indicating the requested fields of view. [0039] In another embodiment, the module 303 for requesting and receiving the data stream is configured to use meta data to indicate the union of the requested field of view.
[0040] In another embodiment, the apparatus is an edge data agent.
[0041] In an embodiment of the present subject matter the module 302 for determining a union of at least two of the requested field of view, is configured to determine segments constituting the two requested fields of view and wherein the union of the fields of view is constituting of the previously determined segments.
[0042] In other words, within the above presented embodiment, the union of the requested field of views is constituted of the segments, that belong at least to one of the field of view requested by the user.
[0043] In an embodiment of the present subject matter the module 304 for extracting a relevant part of the received data stream is configured to determine segments constituting the requested field of view and wherein the relevant part of the received data stream according to the concatenation of these determined segments.
[0044] In other words, within the above presented embodiment the extraction of the relevant video segments is also done based on the segments requested from each user as part of the Field of view requests. The overlapping data is replicated towards each of the users and the union data is only buffered at the edge.
[0045] In another embodiment ofthe present subject matter is an apparatus. This apparatus comprises a processor or a processor circuitry or a microprocessor. This apparatus also comprises memory and a software integrated within this memory, this memory plus software configure to, with the processer or processor circuitry or microprocessor, cause the apparatus to receive, at least two requests of the data stream the requests being associated to a requested field of view and
determine a union of at least two of the requested field of view and
request the data stream presented according to the union of the requested field of view and extract at least one relevant part of the received data stream corresponding to one of the requested field of view and
transmit the relevant part of the received data stream. [0046] The modules will be realized by one or more processors and one or more memory.
The module may also be realized using dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the modules may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term“processor” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), read only memory (ROM) for storing software, random access memory (RAM), and non- volatile storage. Other hardware, conventional and/or custom, may also be included.
[0047] The memory may include any computer-readable medium known in the art including, for example, volatile memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM), and/or non-volatile memory, such as read only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes. The memory includes modules and data. The modules include routines, programs, objects, components, data structures, etc., which perform tasks or implement abstract data types. The data, amongst other things, serves as a repository for storing data processed, received, and generated by one or more of the modules.
[0048] In an embodiment of the present subject matter a cloud data server is proposed. This cloud data server comprises a module for receiving a request of data stream presented per the union of the requested field of view and module for transmitting this data stream presented per the union of the requested field of view.
[0049] In another embodiment of the cloud data server, this cloud data server comprises a processor or a processor circuitry or a microprocessor. The cloud data server also comprises memory and a software integrated within this memory, this memory plus software configure to, with the processer or processor circuitry or microprocessor, cause the cloud data server to:
• receive a request of data stream presented per the union of the requested field of view and
• transmit this data stream presented per the union of the requested field of view. [0050] In an embodiment of the present subject matter an end user apparatus is proposed.
This end user apparatus could be for example a virtual reality head set. This end user apparatus comprises a module for transmitting a request of a data stream and an associated indication of a requested field of view and a module for receiving a relevant part of a data stream.
[0051] In another embodiment of the end user apparatus, this end user apparatus comprises a processor or a processor circuitry or a microprocessor. The end user apparatus also comprises memory and a software integrated within this memory, this memory plus software configure to, with the processer or processor circuitry or microprocessor, cause the end user apparatus to
• transmit a request of a data stream and an associated indication of a requested field of view and
• receive a relevant part of a data stream.
[0052] The figure 4 presents an embodiment of a system of the present subject matter. This system comprises end user apparatus, for example virtual reality headset, edge data agents, configured to realize the method of the figure 2 and cloud data server, comprising a set of data stream.
[0053] In other words, and by using the terminology of the figure 4, the edge Data agents are configured to obtain form the could data server an aggregation of streamed content. To realize this if a video stream is being viewed by multiple end user apparatus under a common edge data agent the edge date agent uses field of view information from multiple sources and to realize an aggregate request at destination of the cloud data server.
[0054] This aggregation allows a reduction of the data which is being requested from the cloud data server. Indeed, the edge data agent downloads a single copy of the data which is restricted to a union of the field of view requests.
[0055] The edge data agent is also configured to split this video based on the request streams towards individual devices. A movement of the field of view of a user is handled locally from the edge data agent itself. When a user field of view and if parts of the new field of view is not covered in the union of field of view, the edge data agent shall request the cloud data server for this not covered part of the field of view, or vice versa. [0056] If for example two end user apparatus are requesting segment of 360 data based on their field of view.
The end user apparatus send this request for the field of view to the Edge Data Agent.
The Edge Data Agent uses the meta data to determine the commonality of the request and commonality in the field of view.
It performs a union of the field of view of both users.
This unified field of view which a subset of the 360 data is requested from the backend cloud which has the complete data.
The Backend Data Agent checks the meta data to determine the field of view to be returned. It extracts the relevant section and streams it to the Edge until it receives a change in the field of view.
The Edge Data agent splits the field of view with duplication to the two end user apparatus.
[0057] A person skilled in the art will readily recognize that steps ofthe methods, presented above, can be performed by programmed computers. Herein, some embodiments are also intended to cover program storage devices, for example, digital data storage media, which are machine or computer readable and encode machine-executable or computer-executable programs of instructions, where said instructions perform some or all of the steps of the described method. The program storage devices may be, for example, digital memories, magnetic storage media, such as magnetic disks and magnetic tapes, hard drives, or optically readable digital data storage media.
[0058] Another embodiment of the present subject matter is a computer-readable medium having embodied thereon a computer program configured to realize the method, presented figure 1. This method allows the transmission of a data stream. This method comprises:
A step 101 of receiving, at least two requests of the data stream, the requests being associated to a requested field of view and
A step 102 of determining a union of at least two of the requested field of view and A step 103 of requesting the data stream presented according to the union ofthe requested field of view and
A step 104 of extracting at least one relevant part of the received data stream corresponding to one ofthe requested field of view and • A step 105 of transmitting the relevant part of the received data stream.
[0059] In an embodiment, the method is configured to transmit the data stream between a server and at least two end user apparatus.
[0060] In an embodiment, the data stream is a 360 AR/VR data stream. [0061] In an embodiment, the step 101 of receiving the two requests of the data stream is configured to receive these requests from at least two end user apparatus, furthermore within this embodiment each request being associated to one of the end user apparatus.
[0062] In an embodiment, the step 103 of requesting the data stream, is configured to request this data stream to a server and this step is further configured to receive, from the server, this data stream presented according the union of the requested field of view.
[0063] In an embodiment, the step 105 of transmitting the relevant part of the received data stream, is configured to transmit this relevant part of the received data stream to the end user apparatus associated to the requested field of view.

Claims

1. Method to transmit a data stream comprising:
A step (101) of receiving at least two requests of the data stream, the requests being associated to a requested field of view and
A step (102) of determining a union of at least two of the requested field of view and A step (103) of requesting the data stream presented according to the union of the requested field of view and
A step (104) of extracting at least one relevant part of the received data stream corresponding to one of the requested field of view and
A step (105) of transmitting the relevant part of the received data stream.
2. Method according to the claim 1 wherein the step (101) of receiving the two requests of the data stream is configured to receive these requests from at least two end user apparatus, furthermore within this embodiment each request being associated to one of the end user apparatus.
3. Method according to the claim 1 or 2 wherein the step (103) of requesting the data stream, is configured to request this data stream to a server and this step is further configured to receive, from the server, this data stream presented according the union of the requested field of view.
4. Method according to the claim 1 or 2 or 3 wherein, the step (105) of transmitting the relevant part of the received data stream, is configured to transmit this relevant part of the received data stream to the end user apparatus associated to the requested field of view.
5. Method according to the claim 1 or 2 or 3 or 4 wherein, the step (102) of determining the union of the requested field of view is configured to use meta data indicating the data streams and/or meta data indicating the requested fields of view.
6. Method according to the claim 1 or 2 or 3 or 4 or 5 wherein, the step (103) of requesting and receiving the data stream is configured to use meta data indicating the data streams and/or meta data to indicate the union of the requested field of view.
7. Apparatus comprising
A module (301) for receiving, at least two requests of the data stream, the requests being associated to a requested field of view and
A module (302) for determining a union of at least two of the requested field of view and A module (303) for requesting the data stream presented according to the union of the requested field of view and
A module (304) for extracting at least one relevant part of the received data stream corresponding to one of the requested field of view and
A module (305) for transmitting the relevant part of the received data stream.
8. Apparatus according to the claim 7 wherein the module (301) for receiving the two requests of the data stream is configured to receive these requests from at least two end user apparatus, furthermore within this embodiment each request being associated to one of the end user apparatus.
9. Method according to the claim 7 or 8 wherein the module (303) for requesting the data stream, is configured to request this data stream to a server and this step is further configured to receive, from the server, this data stream presented according the union of the requested field of view.
10. Method according to the claim 7 or 8 or 9 wherein, the module (305) for transmitting the relevant part of the received data stream, is configured to transmit this relevant part of the received data stream to the end user apparatus associated to the requested field of view.
11. Apparatus according to the claim 7 or 8 or 9 or 10, wherein the module 302 for determining the union of the requested field of view is configured to use meta data indicating the requested data streams and/or meta data indicating the requested fields of view.
12. Apparatus according to the claim 7 or 8 or 9 or 10 or 1 1 , wherein the module 303 for requesting and receiving the data stream is configured to use meta data to indicate the union of the requested field of view.
13. Apparatus according to the claim 7 or 8 or 9 or 10 or 1 lor 12 wherein the apparatus is an edge data agent.
14. System comprising an end user apparatus, an apparatus according to any of the claims 7 to 13 and a cloud data server.
15. Computer-readable medium having embodied thereon a computer program configured to realize a method to transmit a data stream comprising:
A step (101) of receiving at least two requests of the data stream, the requests being associated to a requested field of view and
A step (102) of determining a union of at least two of the requested field of view and A step (103 of requesting the data stream presented according to the union of the requested field of view and
A step (104) of extracting at least one relevant part of the received data stream corresponding to one of the requested field of view and
A step (105) of transmitting the relevant part of the received data stream.
PCT/EP2018/081634 2017-12-12 2018-11-16 Method, apparatus, system and computer readable medium to transmit data stream WO2019115153A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
IN201741044590 2017-12-12
IN201741044590 2017-12-12

Publications (1)

Publication Number Publication Date
WO2019115153A1 true WO2019115153A1 (en) 2019-06-20

Family

ID=64362552

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2018/081634 WO2019115153A1 (en) 2017-12-12 2018-11-16 Method, apparatus, system and computer readable medium to transmit data stream

Country Status (1)

Country Link
WO (1) WO2019115153A1 (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016051127A1 (en) * 2014-09-30 2016-04-07 Sony Corporation Video data encoding and decoding
WO2018222996A1 (en) * 2017-06-02 2018-12-06 Vid Scale, Inc. 360-degree video delivery over next generation network

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016051127A1 (en) * 2014-09-30 2016-04-07 Sony Corporation Video data encoding and decoding
WO2018222996A1 (en) * 2017-06-02 2018-12-06 Vid Scale, Inc. 360-degree video delivery over next generation network

Similar Documents

Publication Publication Date Title
US20200236278A1 (en) Panoramic virtual reality framework providing a dynamic user experience
US20170078447A1 (en) Adaptive streaming of virtual reality data
JP6622308B2 (en) Interactive binocular video display
US11483629B2 (en) Providing virtual content based on user context
JP7085816B2 (en) Information processing equipment, information providing equipment, control methods, and programs
WO2018108104A1 (en) Method and device for transmitting panoramic videos, terminal, server and system
US10386634B2 (en) Collaborative scene sharing for overcoming visual obstructions
US20220256231A1 (en) Systems and methods for synchronizing data streams
CN112567759B (en) Digital media system supporting multiple features regarding virtual reality content
US11677925B2 (en) Information processing apparatus and control method therefor
TW201911240A (en) Image processing device and method, file generating device and method, and program
US11223662B2 (en) Method, system, and non-transitory computer readable record medium for enhancing video quality of video call
EP3465631B1 (en) Capturing and rendering information involving a virtual environment
KR20180052255A (en) Method and Apparatus for Providing and Storing Streaming Contents
US11138804B2 (en) Intelligent smoothing of 3D alternative reality applications for secondary 2D viewing
KR102095454B1 (en) Cloud server for connected-car and method for simulating situation
US11095695B2 (en) Teleconference transmission
WO2019115153A1 (en) Method, apparatus, system and computer readable medium to transmit data stream
US20230132420A1 (en) Asset reusability for lightfield/holographic media
Lee et al. Real‐time multi‐GPU‐based 8KVR stitching and streaming on 5G MEC/Cloud environments
US11212562B1 (en) Targeted video streaming post-production effects
US10917679B2 (en) Video recording of a display device
JP2021033354A (en) Communication device and control method therefor
KR20200080041A (en) Method and apparatus for generating multi channel images using mobile terminal
JP2019075740A (en) Image processing system, image processing apparatus, image transmission method, and program

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18804608

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18804608

Country of ref document: EP

Kind code of ref document: A1