US11282169B2 - Method and apparatus for processing and distributing live virtual reality content - Google Patents

Method and apparatus for processing and distributing live virtual reality content Download PDF

Info

Publication number
US11282169B2
US11282169B2 US16/958,698 US201916958698A US11282169B2 US 11282169 B2 US11282169 B2 US 11282169B2 US 201916958698 A US201916958698 A US 201916958698A US 11282169 B2 US11282169 B2 US 11282169B2
Authority
US
United States
Prior art keywords
images
panoramic
image
generate
rectified
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US16/958,698
Other languages
English (en)
Other versions
US20200349672A1 (en
Inventor
Wayne Cochran
Fai Yeung
Durga Raj Mathur
Gilson Goncalves de Lima
Patrick Youngung Shon
John A. Harrison
Ok Joon KIM
Harleen Gill
Kyle Siehl
Uma Jayaram
Sankar Jayaram
Archie Sharma
Gockcen Clingir
Stanley Baran
Mayuresh Varerkar
Barnan Das
Narayan Biswal
Nilesh Shah
Ritesh Kale
Greg WEINSTEIN
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Intel Corp
Original Assignee
Intel Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Intel Corp filed Critical Intel Corp
Priority to US16/958,698 priority Critical patent/US11282169B2/en
Publication of US20200349672A1 publication Critical patent/US20200349672A1/en
Assigned to INTEL CORPORATION reassignment INTEL CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SHAH, NILESH, WEINSTEIN, Greg, DAS, BARNAN, KALE, RITESH, KIM, OK JOON, SHON, Patrick Youngung, MATHUR, Durga Raj, BISWAL, NARAYAN, SHARMA, ARCHIE, JAYARAM, SANKAR, JAYARAM, UMA, CILINGIR, GOKCEN, GILL, HARLEEN, Varerkar, Mayuresh, YEUNG, FAI, BARAN, STANLEY, SIEHL, Kyle, COCHRAN, WAYNE, HARRISON, JOHN A., CONCALVES DE LIMA, GILSON
Application granted granted Critical
Publication of US11282169B2 publication Critical patent/US11282169B2/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/10—Geometric effects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/08—Volume rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G06T3/12—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4038—Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/97—Determining parameters from multiple pictures
    • G—PHYSICS
    • G11—INFORMATION STORAGE
    • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/243—Image signal generators using stereoscopic image cameras using three or more 2D image sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/21805—Source of audio or video content, e.g. local disk arrays enabling multiple viewpoints, e.g. using a plurality of cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/2187—Live feed
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
    • H04N21/23418—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234309—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 or from Quicktime to Realvideo
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
    • H04N21/2402—Monitoring of the downstream path of the transmission network, e.g. bandwidth available
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/242—Synchronization processes, e.g. processing of PCR [Program Clock References]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/8126—Monomedia components thereof involving additional data, e.g. news, sports, stocks, weather forecasts
    • H04N21/8133—Monomedia components thereof involving additional data, e.g. news, sports, stocks, weather forecasts specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/66—Remote control of cameras or camera parts, e.g. by remote control devices
    • H04N23/661—Transmitting camera control signals through networks, e.g. control via the Internet
    • H04N23/662—Transmitting camera control signals through networks, e.g. control via the Internet by using master/slave camera arrangements for affecting the control of camera image capture, e.g. placing the camera in a desirable condition to capture a desired image
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Definitions

  • This disclosure pertains to videography, image capture, and playback. More particularly, this disclosure relates to systems and methods for processing and distributing live virtual reality content.
  • Techniques are known for capturing and viewing panoramic images. Most techniques involve a capture system that captures and processes a series of images to form panoramic images. Sequences of these images form panoramic videos. Techniques for viewing panoramic images typically include a user input method for viewing a portion of the panoramic image on a screen, giving the user a feel of being able to look around in the panoramic image. In addition, techniques are known for transport controls using a buffer of a live video feed. These controls allow a viewer to buffer a live feed on a local device and forward, rewind, pause, and jog through the buffered video.
  • Another challenge is the amount of bandwidth consumed by the plurality of high definition virtual reality streams across the VR broadcast infrastructure (e.g., servers, network links).
  • the VR broadcast infrastructure e.g., servers, network links.
  • FIG. 1 is a schematic diagram showing unique embodiments of time code synchronization mechanisms that could be used to synchronize frames being captured by capture stations from a plurality of panoramic camera heads before being processed and distributed.
  • FIG. 2 is a schematic diagram showing how multiple receivers, or receiving modules on a viewer machine would receive time-stamped frames from the panoramic video feeds, and to show the user interface as the intermediate application for managing how the user input requests are handled and how the clients are manipulated to cater to the user request.
  • FIG. 3 is a schematic diagram showing how multiple panoramic video feeds can be received at a client by a receiver and user interface that also has controller functionality built in.
  • FIG. 4 is a flow chart showing the steps involved in a viewer machine to receive multiple panoramic video streams, to buffer the frames from each feed, and to determine the frame from the buffer to be displayed to the end user based on the camera in view and the time stamp sought by the user.
  • FIG. 5 is a flow chart showing the steps involved in handling a Camera Changed Event triggered by the user.
  • FIG. 6 is a flow chart showing the steps involved in handling a Video Playback State Changed Event triggered by the user.
  • FIG. 7 is a flow chart showing the steps involved in handling a Viewport Changed Event triggered by the user.
  • FIGS. 8 - a and 8 - b are two parts of a flowchart showing how the Transport Control Events are handled by the system and how the time stamp for the frame to be displayed to the user is determined based on the Video Playback State of the viewer application.
  • FIG. 9 shows how multiple panoramic cameras are strategically placed an event location and how they are connected to the capture stations, processing stations, and distribution channel.
  • FIG. 10 illustrates one embodiment of an architecture for capturing and streaming real time video of an event
  • FIG. 11 illustrates one embodiment which performs stitching using rectification followed by cylindrical projection
  • FIGS. 12A-D illustrate a top view of operations performed to generate a panoramic virtual reality video stream
  • FIG. 13 illustrates a front view of a subset of the operations performed to generate a panoramic virtual reality video stream
  • FIG. 14 illustrates a method in accordance with one embodiment of the invention
  • FIG. 15 illustrates one embodiment which performs stitching operations using Belief Propagation
  • FIG. 16 illustrates a stitching architecture which uses stitching parameters from one or more prior frames to stitch a current frame
  • FIG. 17 illustrates one embodiment which performs coordinate transformations to reduce bandwidth and/or storage
  • FIG. 18 illustrates a method in accordance with one embodiment of the invention
  • FIG. 19 illustrates an architecture for performing viewing transformations on virtual reality video streams to adjust
  • FIG. 20 illustrates one embodiment in which key and fill signals are used for inserting content into a captured video stream
  • FIG. 21 illustrates a comparison between a microservices architecture and other architectures
  • FIG. 22 illustrates microservices integrated within the architecture for capturing and streaming real time video
  • FIG. 23 illustrates a specific embodiment in which supervisory microservices query make monitoring requests to probes using RPC
  • FIG. 24 illustrates another specific embodiment in which a completion queue supports communication between an RPC server and an RPC client
  • FIG. 25 illustrates sequences of read/write transactions between an RPC server and an RPC client
  • FIG. 26 illustrates one embodiment in which a broadcaster's data feed is integrated by a data aggregator
  • FIGS. 27-29 illustrate examples of a scoreboard graphical user interface implemented on embodiments of the invention.
  • Embodiments of the present invention disclose an apparatus and method for receiving a video stream from a plurality of Panoramic Video Camera Heads or from a local storage disk, storing the video data in a local memory buffer, and viewing regions of interest within any one of the panoramic videos using user interface devices, while controlling the video time, playback speed, and playback direction globally across all panoramic video data in a synchronous manner.
  • multiple Panoramic Video Camera Heads and are synchronized through a time code generator that triggers the image capture across all camera heads synchronously.
  • multiple camera heads are synchronized by one “Master” camera head that sends trigger signals to all the camera heads.
  • each camera head is set to “free-run” with a pre-defined frame rate, and the processing computers all capture the latest frame from each of these cameras and timestamp them with a time code from a time code generator.
  • Transport Control is understood to mean a user interface that allows a viewer to control the video playback, such as choosing between play, pause, rewind and forward, and the speed of rewind or forward.
  • FIG. 1 shows construction of the time code synchronization mechanism 10 extending across a plurality of panoramic camera heads 12 , 14 and 18 and capture stations 22 , 24 and 25 .
  • a time code generator 20 is used to get a consistent time stamp based on the desired rate that frames 50 , 52 and 54 need to be captured from the panoramic cameras 12 , 14 and 18 .
  • the same time code from time code generator 20 is received by each of the Capture Stations 22 , 24 and 26 , and in one of the embodiments of this mechanism, the time code is used to trigger.sup.1 44 , 46 and 48 the panoramic cameras 12 , 14 and 18 . This is also referred to as a “software trigger” 44 , 46 and 48 of the panoramic cameras 12 , 14 and 18 .
  • the panoramic cameras 12 , 14 and 18 capture a frame 50 , 52 and 54 when triggered by trigger 44 , 46 and 48 , respectively, and return the frame 50 , 52 and 54 to the corresponding Capture Stations 22 , 24 and 26 that generated the trigger 44 , 46 and 48 .
  • the Capture Stations 22 , 24 and 26 attach the time-stamp information from the time code to the frames, forming “frames with time stamps” 56 , 58 and 60 . Because the time-code is shared between Capture Stations 22 , 24 and 26 , the frames 56 . 58 and 60 generated from each of the Capture Stations 22 , 24 and 26 for a given time-code are synchronized, as they have the same time-stamp.
  • the frames 56 , 58 and 60 are then transmitted to the Processing Station 28 , 30 and 32 , respectively, where they are compressed for transmission over the network and sent to some Distribution Channel 34 .
  • the time-stamp information on the frames 56 , 58 and 60 is maintained throughout this processing, compression, and distribution process.
  • the distribution device, or channel (switch) 34 is configured to distribute the processed images or compressed video stream to client processors in clients 36 , 38 and 40 .
  • Clients 36 , 38 and 40 also include memory.
  • FIG. 1 Another embodiment of the time code synchronization mechanism 10 of FIG. 1 involves triggering the panoramic camera heads 12 , 14 and 18 using a “hardware sync trigger.sup.2” 42 .
  • the hardware trigger 42 is generated at specific time intervals based on the desired frame rate. This rate of hardware triggering has to match the rate of time codes being generated by the time code generator 20 .
  • One of the panoramic camera heads 12 , 14 and 18 acts as a “Master” and all other panoramic camera heads 12 , 14 and 18 act as “Slaves”.
  • the “Master” panoramic camera triggers itself and all the “Slave” panoramic cameras synchronously. When a trigger is generated, a frame is captured at the panoramic camera 50 , 52 or 54 .
  • an event is invoked at the Capture Station 22 , 24 or 26 , and this is when the Capture Station 22 , 24 or 26 “grabs” the frame from the camera 12 , 14 or 18 , and associates the time stamp corresponding to the latest time-code received from the time-code generator 20 to the frame 50 , 52 or 54 .
  • a third embodiment of the time code synchronization mechanism 10 of FIG. 1 involves letting the panoramic cameras 12 , 14 and 18 capture frames in a “free run” mode, where each of the panoramic cameras 12 , 14 and 18 trigger as fast as possible.
  • the Capture Station 22 , 24 and 26 uses the time code signal to “grab” the latest frame 50 , 52 or 54 that was captured by the panoramic camera 12 , 14 or 18 , and associates the time stamp corresponding to the time-code with the frame.
  • FIG. 2 shows multiple receivers 64 , 66 and 68 on a client machine 36 receiving time-stamped slices 78 , 80 and 82 , respectively, from the panoramic video feeds via distribution channel 34 .
  • a user interface 70 on the client machine 36 determines which receiver is the active receiver 64 , 66 or 68 displayed to the user.
  • User interface 70 also manages the user interaction input from devices 62 like a joystick 75 , a keyboard 76 , and a touch or gesture based device(s) 77 .
  • User interface 70 uses this input to determine which client stream should be the active stream (switch between videos 74 ), and what section of the panoramic video should be displayed (zoom/tilt/pan 73 ) to the end-user.
  • Another input from the user-interaction devices is the input related to transport control 72 .
  • User interface 70 uses this input and passes it on to all the receivers. This enables all the receivers to perform the same transport control operations to their respective panoramic video streams, and ensures that all the panoramic video streams are synchronized.
  • FIG. 3 shows another embodiment of the client application on the viewer machine.
  • a single application serves as the receiver and user interface 84 .
  • the receiver receives time-stamped frames for all the panoramic video streams via distribution channel 34 and manages each of these streams in its own application memory.
  • the receiver also includes processing circuitry.
  • User interface functionality described in FIG. 2 is also integrated in this application. As described in FIG. 2 , the user interface manages the input from the user interaction devices 86 and performs the actions for switching between videos 89 , what section of the panoramic video should be displayed (zoom/pan/tilt 88 ) to the end-user, and how to apply the transport control 87 to all the streams in memory.
  • the following variables are stored with the controller module for receiver and user interface 84 that determine the state of the view that is displayed to the end-user: a. Current Camera to be displayed b. Current Time Stamp of the frame to be displayed c. Current Video Playback State—Possible values are Play, Pause, Fast Forward, Rewind, Live d. Current Viewport—The viewport is determined by the current zoom, pan, and tilt values
  • the user interaction devices 86 could generate the following types of events that are handled by the receiver and user interface 84 : a. Camera Changed Event b. Video Playback State Changed Event c. Viewport Changed Event d. Transport Control Event
  • FIG. 4 shows the steps involved in a viewer machine to receive multiple panoramic video streams and determine the frame to be displayed to the end user.
  • the frames from each panoramic video stream that is received by the viewer machine 102 are buffered in memory (Hard disk drive, application memory, or any other form of storage device) 104 .
  • Each frame received by the viewer machine has a time-stamp associated with it, which serves as the key to synchronize frames across multiple panoramic streams.
  • the viewer application enters a refresh cycle loop starting with a “wait for refresh cycle” 106 .
  • the refresh cycle is a periodic set of operations performed by the application at every refresh interval of the display.
  • the viewing application stores the information about the panoramic camera being displayed 108 and the information about the time stamp to be displayed based on the playback state of the application and user inputs related to transport controls. For each refresh cycle, the application checks the current panoramic camera that needs to be displayed, and then checks for the time stamp to be displayed 110 . Using these two pieces of information, the appropriate frame to be displayed is sought from the buffer in memory 112 . This frame is then passed on to the application for display 114 in that refresh cycle.
  • FIG. 5 shows the steps involved in handling the Camera Changed Event triggered by the user.
  • An initial camera is used, or defined 202 as the default after initiating a start 200 .
  • the application goes into a ‘listen’ mode 204 where it is waiting for Camera Changed Events 206 triggered by the user interaction devices.
  • the local variable in the application that stores current camera information is updated 208 , and the application goes back into the ‘listen’ mode, waiting for the next Camera Changed Event.
  • FIG. 6 shows the steps involved in handling the Video Playback State Changed Event triggered by the user from start 300 .
  • An initial video playback state 302 is used as the default to start with.
  • the application goes into a ‘listen’ mode 304 where it is waiting for Video Playback State Changed Events 306 triggered by the user interaction devices.
  • the local variable in the application that stores the current video playback state is updated 308 , and the application goes back in the ‘listen’ mode, waiting for the next Video Playback State Changed event.
  • FIG. 7 shows the steps involved in handling the Viewport Changed Event triggered by the user from start 400 .
  • the viewport could be changed by changing the zoom, tilt, or pan.
  • An initial zoom, tilt, and pan is used as a default 402 to start with.
  • the application goes into a ‘listen’ mode 404 where it is waiting for Viewport Changed Events triggered by the user interaction devices.
  • the application checks to see if the zoom 410 , pan 406 , or tilt 408 value has been changes, and updates the local variables 416 , 412 and 414 , respectively in the application that store the zoom, pan, and tilt.
  • the application then goes back in the ‘listen’ mode, waiting for the next Viewport Changed Event.
  • FIGS. 8 a and 8 b show how the Transport Control Events are handled by the viewing application initiated at start 500 .
  • the application is listening for Transport Control Changed Events 502 .
  • the application checks to see if the velocity of transport control was changed 504 . If the velocity was changed, the value of the velocity stored within the application is updated 518 and the application goes back to listening for Transport Control Changed Events. If velocity has not changed, then the application checks to see if the user has requested to “Transport to Start” 506 so that they view the start of the buffered video stream in memory.
  • the value of the current timestamp to display is changed to be the same as the timestamp of the frame at the start of the buffer in memory 520 , and the application goes back to listening for Transport Control Changed Events. If “Transport to Start” was not requested, then the application determines the current timestamp to be used for display based on playback state that the application is in. If the application is in “Play” state 508 , then the current timestamp is incremented to the next timestamp 522 . If the application is in the “Pause” state 520 , then the current timestamp is not changed 524 .
  • the current timestamp is incremented 526 or decremented 528 taking the frame rate and velocity of transport into account. If the application is in the “Live” state 516 , then the current timestamp is set to the timestamp of the frame at the end of buffered frames in memory 530 .
  • FIG. 9 shows a football field 90 as the event location where multiple panoramic cameras 12 , 14 , 16 and 18 are located at strategic locations such that they provide different angles to view a sporting event from and allow one or more end-users to choose the angle that is best suited (for them) for viewing the event at any given point in time.
  • Each of the panoramic video cameras 12 , 14 , 16 and 18 is connected to a capture station 22 , 24 , 25 and 26 , respectively.
  • Each capture station 22 , 24 , 25 and 26 receives a time-code from a time-code generator, and the time-stamp from the time-code is attached to the frames received from the panoramic video camera.
  • the frames are then transmitted to the processing stations 28 , 30 , 31 and 32 where they are processed and streamed out to the distribution channel 34 .
  • Distribution channel 34 receives the frames and communicates the frames over a network to multiple clients that are connected to the distribution channel.
  • a panoramic video capture device as used herein comprises multiple sensors placed in a circular array such that a portion of image captured by each sensor overlaps with a portion of image captured by adjacent sensors.
  • the overlapping images from the different sensors are captured synchronously based on a trigger mechanism, and these overlapping images form the basis for creation of a single, seamless panoramic image.
  • a processor is a high-performance server-grade machine housing multiple graphic processing units (GPUs).
  • a GPU is capable of performing large number of operations in parallel.
  • the use of multiple GPUs in the processor allows for highly parallelized computations on multiple image frames being communicated by the panoramic video capture device. Memory can also be resident.
  • a processor comprises the following modules.
  • a capture module is responsible for triggering the panoramic video capture device and retrieving the image frames once the exposure of the frame is complete.
  • the triggering of the sensors is not performed by this module.
  • There is a separate trigger mechanism for the sensors and the capture module is notified of the event every time a new image frame is available on the panoramic video capture device. When this notification is received by the capture module, it retrieves the image frame from the panoramic video capture device.
  • a processing module is operative to receive the raw frame from the capture module and applies the following filters to the raw frame: Demosaicing filter: In this filter, a full color image is reconstructed using the incomplete color samples from the raw image frames.
  • Coloring filter The full color image output from the demosaicing filter is then converted to appropriate color space (for example, RGB) for use in downstream modules.
  • Seam blending filter Colored images output from the coloring filter are used for blending the seam using stitching algorithms on the overlap between adjacent images.
  • a splicing module is responsible for using the images output from the processing module, and putting them together with the ends lined up against each other in such that the aggregate of these individual images creates one panoramic image.
  • a slicing module takes the seam blended panoramic image, and splits this image into multiple slices. This is done so that each slice of the panoramic image can be distributed over the network in an optimized fashion. This overcomes the existing limitations of certain network protocols that cannot communicate panoramic images above a certain size of the image.
  • a time stamp module listens for the time code from the time code generator. This time stamp is then attached to each slice of the image sections output from the slicing module.
  • a compression module takes the image frame output by the time stamp module and compresses it using certain image compression techniques (JPEG, H.264, etc.) for transmission of over the network.
  • JPEG image compression techniques
  • a distribution device is a kind of router or switch that is used for transmitting the compressed frames over the network. Multiple clients could connect to the distribution device and receive the image frames being transmitted. In addition to this, subsequent distribution devices themselves could be connected to a distribution device transmitting the images for relaying the images over a wide network.
  • a client process processes the combination of sub-processes and modules on a viewer's machine to receiving image frames from a distribution device, store them in buffer, manage the user input from the user interaction devices, and display the video images to the end-user.
  • the client process is broken down into the following modules:
  • a receiving module which connects to the source of the video images via the distribution device, receives the images over the network, and stores them in a buffer on the viewer's machine.
  • a user interface module is used for managing the user input from the user interaction devices.
  • the joystick controller is used for capturing the user input.
  • the user input could be provided using buttons on the joystick or using the multiple thumb pad controls on the joystick. Different buttons are used to track the video playback state change input for play, pause, fast forward, rewind, or live mode
  • a thumb pad control is used to track the viewport change inputs for zoom, pan, tilt of the view
  • Another thumb pad control is used to track the transport control input for jogging forward or back based on the velocity of jog determined by how far the thumb pad control has been pushed.
  • a display module is used for displaying portion of the panoramic video frames to the user.
  • the portion of the video frame to be displayed is determined based on the inputs from the user interface module.
  • Image frame from the buffer is fetched and based on the other user inputs, the portion of the panoramic image to be displayed is determined. This portion is then displayed to the end-user for viewing.
  • FIG. 10 illustrates one example of a panoramic broadcast virtual reality (VR) system.
  • a plurality of stereoscopic cameras 1001 capture video of an event from different perspectives (e.g., a sporting event, musical performance, theatrical performance, etc) and stereo audio capture unit 1002 simultaneously captures and encodes audio 1003 of the event.
  • the six pairs of stereoscopic cameras are integrated on a video capture device 1001 (referred to herein as a capture POD) and any number of such video capture devices 1001 are distributed at different event locations to capture video from different perspectives.
  • a stereoscopic camera is typically implemented as two cameras: one to reproduce a left eye perspective and one to reproduce a right eye perspective.
  • only the left (right) eye video may be captured and the right (left) stream may be reproduced by performing a transformation on the left (right) video stream (i.e., using the coordinate relationship between the left and right eyes of a user as well as the coordinates of the event).
  • any number of pairs of stereoscopic cameras may be used while still complying with the underlying principles of the invention (e.g., 10 pairs/POD, 12 pairs/POD, etc).
  • the video stream produced by each capture POD comprises an 8-bit Bayer mosaic at with 12 splits (i.e., 12 different image streams from the 6 pairs of cameras).
  • One or more graphics processing units (GPUs) 1005 then process the video stream in real time as described herein to produce a panoramic VR stream.
  • the GPU 1005 performs various image processing functions including, but not limited to, de-mosaic operations, cropping to remove redundant portions of adjacent video streams, lens distortion reduction, color adjustments, and image rotations.
  • the GPU 1005 performs stitch processing 1006 on adjacent image frames to form a stitched panoramic image.
  • One example of the stitch processing 1006 illustrated in FIG. 11 includes rectification operations 1102 , stitching operations 1104 , and cylindrical projection operations 1106 .
  • FIG. 11 illustrates a specific implementation of stitching using 5 image streams to generate the panoramic image stream. It is assumed that the 5 illustrated streams are processed for one eye (e.g., the left eye) and that the same set of operations are performed concurrently for the other eye (e.g., the right eye).
  • the highlighted regions 1101 A-B of two of the images in the top row of images 1101 indicates the overlapping portions of each image that will be used to identify the stitch.
  • the width of these regions is set to some fraction of the overall width of each image (e.g., 1 ⁇ 4, 1 ⁇ 3, 1 ⁇ 2).
  • the selected regions include overlapping video content from adjacent images.
  • the GPU aligns the left image with the right image by analyzing and matching this content. For example, one implementation performs a 2D comparison of the pixel content in each row of pixels.
  • One or more feature points from a first image region e.g., 1101 A
  • the second image region e.g., 1101 B
  • a more complex matching model may be used such as belief propagation.
  • Image rectification 1102 is performed, projecting the images 1103 onto a common image plane.
  • a stitcher 1104 implemented by the GPU uses the designated regions of adjacent rectified images 1103 to match pixels (in accordance with a specified matching algorithm) and identify the correct orientation and overlap between the rectified images 1103 .
  • the stitcher 1104 combines each adjacent image to form a plurality of stitched, rectified images 1105 . As illustrated, in this particular implementation there are two 1 ⁇ 2 image portions 1105 A-B remaining at each end of the panoramic video.
  • a cylindrical projector 1106 then projects the stitched images 1105 onto a virtual cylindrical surface to form a smooth, consistent view for the end user in the final panoramic video image 1107 .
  • the embodiments described above may be implemented in software executed on the GPU(s) 1005 , by fixed function circuitry, and/or a combination of software and fixed function circuitry (e.g., with some stages being implemented in hardware and others in software).
  • the data for each image may be stored in a system memory, a caching subsystem on the GPU(s) 1005 , a local GPU memory, and/or a GPU register file.
  • FIGS. 12A-E illustrate the effects of this sequence of operations on the video images from an elevated perspective (i.e., looking down in a direction parallel to the image planes).
  • correction for lens distortion is performed on the input images at this stage.
  • the two end portions A 0 and B 0 are also rotated but are not involved in stitching.
  • the following code specifies the operations of one embodiment:
  • a full cylindrical projection is performed for the five stitched images and “half” cylinder projections for the two end images. This is shown as image frames S 1 -S 5 being curved around the virtual cylinder to form C 1 -C 5 and end image frames A 0 and B 5 being similarly curved to form C 0 and C 6 , respectively.
  • the seven resulting images are concatenated together to form the final panoramic image, which is then processed by the remaining stages of the pipeline.
  • FIG. 13 illustrates another perspective using a simplified set of images 1301 - 1306 (i.e., captured with three cameras).
  • Image 1301 shows the arrangement of cameras used to capture the video frames shown in image 1302 (overlap not shown).
  • Each image is split vertically in image 1303 .
  • image 1304 each image is transformed using a homography transformation which is a perspective re-projection that effectively rotates neighboring image planes so that they are parallel (see, e.g., FIG. 12C ). This rectifies the images fed to the stitcher so that common features are aligned along the same image rows, which is an important operation for fast and accurate stitching.
  • image 1305 shows a cylindrical projection which is used to create the final panorama
  • a motion JPEG encoder 1007 encodes the image frames 1107 using motion JPEG (i.e., independently encoding each image frame without inter-frame data as used by other video compression algorithms such as MPEG-2).
  • the encoded/compressed video frames generated by the MJPEG encoder 1007 are packetized by Real-Time Transport Protocol (RTP) packetizer 1008 and stored in a buffer 1009 prior to being transmitted over a network/communication link to RTP depacketizer 1010 . While RTP is used to communicate the encoded/compressed video frames in this embodiment, the underlying principles of the invention are not limited to any particular communication protocol.
  • RTP Real-Time Transport Protocol
  • the depacketized video frames are individually decoded by MJPEG decoder 1011 and scaled 1012 based on desired scaling specifications (e.g., to a height of 800 pixels in one embodiment).
  • the scaled results are temporarily stored in a synchronization buffer 1013 .
  • An aggregator 1014 combines multiple video streams, potentially from different capture PODs 1001 and stores the combined streams in a temporary storage 1015 (e.g., such as the overlay buffer described herein).
  • an H.264 encoder 1016 encodes the video streams for transmission to end users and a muxer & file writer 1017 generates video files 1018 (e.g., in an MP4 file format) at different compression ratios and/or bitrates.
  • the muxer & file writer 1017 combines the H.264 encoded video with the audio, which is captured and processed in parallel as described directly below.
  • the stereo audio capture unit 1002 captures an audio stream 1003 simultaneously with the video capture techniques described herein.
  • the stereo audio capture unit 1002 comprises one or more microphones, analog-to-digital converters, and audio compression units to compress the raw audio to generate the audio stream 1003 (e.g., using AAC, MP3 or other audio compression techniques).
  • An audio decoder 1004 decodes the audio stream to a 16-bit PCM format 1021 , although various other formats may also be used.
  • An RTP packetizer generates RTP packets in an RTP buffer 1023 for transmission over a communication link/network.
  • an RTP depacketizer 1024 extracts the PCM audio data from the RTP packets and an AAC encoder 1024 encodes/compresses the PCM audio in accordance with the AAC audio protocol (although other encoding formats may be used).
  • a media segmenter 1019 temporally subdivides the different audio/video files into segments of a specified duration (e.g., 5 seconds, 10 seconds, 15 seconds, etc) and generates index values for each of the segments.
  • a separate set of media segments 1020 are generated for each audio/video file 1018 .
  • the index values may be used to access the media segments by clients. For example, a user may connect to the real time VR streaming service and be redirected to a particular URL pointing to a particular set of media segments 1020 .
  • the network characteristics of the client's network connection may initially be evaluated to determine an appropriate set of media segments encoded at an appropriate bitrate.
  • the metadata injectors 1030 , 1040 insert/inject various forms of metadata to the media segments 1020 .
  • the metadata may include the current scoring and other relevant data associated with the sporting event (e.g., player statistics, rankings, current score, time remaining, etc), information related to the musical performance (e.g., song titles, lyrics, authors, etc), and any other information related to the event.
  • the scoring data and other relevant data may be displayed within a graphical user interface of the VR client and/or integrated directly within the panoramic video stream (e.g., displayed over the actual scoreboard at the event).
  • various types of metadata may be injected including HTTP Live Streaming (HLS) metadata injected by an HLS metadata injector 1030 and ID3 metadata injected by the ID3 metadata injector 1040 .
  • HLS HTTP Live Streaming
  • a push unit 1025 dynamically pushes out the various media segments 1020 to one or more cloud services 1026 from which they may be streamed by the VR clients.
  • the cloud services 1026 may include the Amazon Web Services (AWS) CloudFront Web Distribution platform.
  • the pushing of media segments may be done in addition to or instead of providing the media segments 1020 directly to the VR clients via the VR service provider's network.
  • FIG. 14 A method for efficiently and accurately stitching video images in accordance with one embodiment of the invention is illustrated in FIG. 14 .
  • the method may be implemented within the context of the system architectures described above, but is not limited to any particular system architecture.
  • N raw camera streams are received (e.g., for each of the left and right eyes).
  • demosaicing is performed to reconstruct a full color image from potentially incomplete color samples received from the cameras.
  • Various other image enhancement techniques may also be employed such as distortion compensation and color compensation.
  • image rectification is performed on the N streams and, at 1404 , N ⁇ 1 overlapping regions of adjacent images are processed by the stitching algorithm to produce N ⁇ 1 stitched images and 2 edge images.
  • cylindrical projection and concatenation are performed on the N ⁇ 1 stitched images and the two edge images to form the panoramic image.
  • one embodiment of the invention employs belief propagation techniques to perform stitching of adjacent images.
  • Belief propagation (BP) (or “sum-product message passing”), is a technique in which inferences are made on graphical models including Bayesian networks and Markov random fields.
  • the belief propagation engine calculates a marginal distribution for each unobserved node, based on observed nodes.
  • belief propagation is used to identify a most likely matching pixel in a second frame for each pixel in a first frame.
  • Belief propagation has its own internal parameters which dictate how different variables are to be weighted to identify matching pixels.
  • the results using standard internal parameters are not ideal.
  • one embodiment of the invention performs modifications to the basic belief propagation parameters to generate significantly improved results.
  • One embodiment of the invention chooses a set of “ideal” parameters based on the requirements of the application.
  • FIG. 15 illustrates the sequence of operations 1501 - 1505 performed by one embodiment of the Belief Propagation engine. These operations include initially performing a data cost evaluation 1501 where, for each pixel in the w ⁇ H overlapping region between the left and right input image 1500 , a cost vector of length L is computed that estimates the initial cost of matching L different candidate pixels between the left and right images.
  • Each cost value is a real number (e.g., stored as a floating point number). There are many ways to compute this cost such as sum of absolute differences (SAD) or sub of squared differences (SSD). In one embodiment, the result of this computation is a w ⁇ H ⁇ L “cost volume” of real numbers.
  • the cost L i is converted into a probability 1/e Li and normalized.
  • the goal is to minimize the cost (energy minimization) or maximize the probability.
  • Different flavors of Belief Propagation One embodiment is described in terms of energy minimization, sometimes called the “negative log probability space.”
  • One implementation also normalizes the colors to adjust for different brightness and exposures between cameras.
  • the number of rows of the images being stitched are down-sampled by a factor (e.g., 2, 3, 4, etc) to speed up the process, thereby reducing the memory footprint and enhancing tolerance for misaligned frames. It is assumed that the images have been rectified so that common features are on the same scan lines (i.e., epipolar lines match and are parallel). Additional image processing may be done at this stage as well such as implementing a high-pass filter to reduce noise from cameras (e.g., charge coupled device (CCD) noise).
  • CCD charge coupled device
  • a data cost pyramid is constructed at 1502 .
  • a series of smaller volumes 1502 A are constructed of size ⁇ w/2 i ⁇ H/2 i ⁇ L
  • i 0 . . . ⁇ that make up the data-cost pyramid by averaging/down-sampling cost values. Note that the cost vectors are still of length L for all volumes in the pyramid.
  • Belief Propagation message passing 1503 A Starting with the smallest volume in the data-cost pyramid, several iterations of Belief Propagation message passing 1503 A are performed. The results are then up-sampled to the next largest volume at 1503 and Belief Propagation message passing 1503 A is performed again using the up-sampled values as a starting point. For each step four more volumes are created to hold the messages that are passed up, down, left, and right between neighboring cost vectors. Once the iterations are complete, the final costs are computed from the original cost volume and the message volumes. These are used to seed the iteration at the next higher level.
  • a stitch map is constructed at 1504 .
  • one embodiment short circuits the multi-scale Belief Propagation process by stopping the iterative process and forming the stitch map from a smaller volume. This results in a smaller stitch map that can be bi-linearly sampled from when stitching.
  • the stitch map is sorted in a hardware texture map managed by the GPU(s) 1005 .
  • the final image is then stitched by performing warping and blending in accordance with the stitch map 1504 to generate the final stitched image frame 1506 .
  • the stitch map is used to determine which two pixels to blend.
  • Laplacian Blending is used in one embodiment to reduce blurring artifacts.
  • a completely new stitch is performed for every frame. Given the significant processing resources used to identify the stitch, one embodiment of the invention feeds back the previous stitch parameters for one or a combination of previous frames to be used to stitch the current frame.
  • FIG. 16 illustrates one embodiment of an architecture which includes rectification circuitry/logic 1602 for performing rectification of images streams from the cameras 1601 (e.g., of one or more capture PODs) and stitcher circuitry/logic 1603 which stores stitching parameters from prior frames to be used as a starting point.
  • a lookahead buffer 1606 or other type of storage is used by the stitcher 1603 to store parameters from previous stitches and read those parameters when processing the current set of image frames.
  • the specific location of a set of prior feature points may be stored and used to identify the stitch for the current image frames (or at least as a starting point for the current image frames).
  • the parameters from previous stitches may simply be the parameters from the last stitch. In another embodiment a running average of these parameters is maintained (e.g., for the last N stitches).
  • the previously-determined depth map pyramids shown in FIG. 15 may be reused.
  • blending between adjacent images is used when a stitch fails.
  • a failed stitch may occur, for example, due to insufficient information, disparate lighting (which may be temporary), and any other circumstances where similarities between pixels cannot be determined.
  • one embodiment of the invention analyzes the previous and next scan lines and blends them together. Different types of blending may be selected based on characteristics of the two frames.
  • the blending may include (but is not limited to) linear blending, Laplacian blending, and Gaussian blending.
  • the stitch parameters from one or more prior stitches may be used (as described above).
  • the luminance (Y) plane is used to perform stitching operations, excluding the U and V planes, to reduce the amount of data required for stitching. Color does not provide significant value for stitching, unless certain types of operations such as background subtraction are used. Thus, the stitching pipeline is optimized with YUV requiring less memory and less time for conversions.
  • the U and the V values may then be evaluated to provide further differentiation between the pixels (e.g., to determine whether they have similar/same colors) thereby providing a more efficient culling mechanism (i.e., to cull candidates which are outside of the threshold).
  • One embodiment the invention quantifies stitch accuracy, potentially evaluating each seam down to a single number. As the stitch is changed, this embodiment searches for patterns, evaluates the associated numbers and identifies the one with the highest quantity as the stitch. This may be performed for each scan line where the belief propagation algorithm determines the extent to which this is a good stitch (i.e., quantifies the stitch accuracy).
  • Different types of variables may be evaluated to arrive at the number including data cost (how well left matches right pixel) and smoothness (how well two neighboring pixels agree).
  • one embodiment reproduces one stream (e.g., which is occluded) using video streams from one or more adjacent cameras.
  • one embodiment of the invention in response to detecting that a stream from camera N is detected (e.g., the left eye stream in a left/right stereoscopic pair of cameras) one embodiment of the invention performs an image transformation on the stream from adjacent cameras N+1 and/or N ⁇ 1 to reproduce the camera N stream.
  • FIG. 17 illustrates an example arrangement in which a plurality of left/right cameras 1701 - 1704 capture an event from different viewpoints.
  • An image of a stick figure is captured relative to a grey rectangle.
  • These two objects are used to illustrate the manner in which the perspective changes from camera N ⁇ 1 to camera N+1. For example, in the video stream from camera N ⁇ 1, there is a larger separation between the two objects while from camera N+1, there is no separation (i.e., the user is occluding a portion of the rectangle).
  • per-camera transformation matrices may be calculated prior to an event based on the orientation differences between a first camera (e.g., camera N) and one or more adjacent cameras (e.g., camera N+1). If the differences in orientation of the two cameras is known (e.g., X, Y, Z vector defining the 3D direction each camera is pointing, the distance to the event objects from the cameras, etc) then these differences may be used to generate a transformation matrix for camera N which can be used to reconstruct it's video stream.
  • two transformation matrices are generated for camera N: one for camera N+1 and one for camera N ⁇ 1. Using two cameras ensures that all of the necessary video data will be available to reconstruct camera N's video stream. However, in other embodiments, only one video stream from one adjacent camera is used. In this case, the camera selected for the reconstruction should be the corresponding left/right camera. For example, if camera N is a left eye camera, then camera N+1 (used for the transformation) should be the corresponding right eye camera. Choosing the alternate eye camera makes sense given the significant correlation in orientation between the left/right cameras. If there are portions of the image which cannot be reconstructed, these portions may be identified in the video stream from camera N ⁇ 1 (e.g., the right camera of the adjacent pair of cameras). The camera N matrix associated with camera N ⁇ 1 may be used to fill in any holes in the transformation performed on the video stream from camera N+1.
  • FIG. 18 A method in accordance with one embodiment of the invention is illustrated in FIG. 18 .
  • transformation matrices are calculated for each camera, based on spatial relationships and differences in orientation between cameras.
  • a degradation of a video stream of camera N is detected. For example, camera N may have failed or may there may be bandwidth issues with the network link.
  • the transformation matrices associated with adjacent cameras N+1 and N ⁇ 1 are retrieved and, at 1804 , a transformation is performed on one or both of the video streams from camera N+1 and camera N ⁇ 1.
  • the camera N matrix associated with camera N+1 may be used to transform camera N+1's video stream using the transformation matrix to reconstruct the video stream from the perspective of camera N.
  • the camera selected for the reconstruction is one of the left/right pair. For example, if camera N is a left eye camera, then camera N+1 (used for the transformation) is the corresponding right eye camera. Choosing the alternate eye camera makes sense given the significant correlation in orientation between the left/right cameras.
  • these portions may be identified in the video stream from camera N ⁇ 1 (e.g., the right camera of the adjacent pair of cameras).
  • the camera N matrix associated with camera N ⁇ 1 may be used to fill in any holes in the transformation performed on the video stream from camera N+1.
  • FIG. 19 illustrates an example architecture which includes a per-camera matrix calculation unit 1907 for calculating the various transformation matrices 1908 described herein based on the camera orientations and relative spatial relationships of the cameras 1906 (as described above).
  • the transformation matrices 1908 are stored for later use.
  • a video stream transformation unit 1904 reconstructs camera N's video stream based on the video streams of camera N+1 and camera N ⁇ 1.
  • the camera N matrix associated with camera N+1 may be used to transform camera N+1's video stream using the transformation matrix to reconstruct the video stream from the perspective of camera N. If there are portions of the image which cannot be reconstructed, these portions may be identified in the video stream from camera N ⁇ 1.
  • the camera N matrix associated with camera N ⁇ 1 may be used to fill in any holes in the transformation performed on the video stream from camera N+1.
  • the techniques described here may be used for a variety of circumstances including, but not limited to insufficient bandwidth, occlusion by objects, and/or equipment failures. While the embodiments described above focus on a camera failure, one embodiment performs the techniques described herein for the sole purpose of reducing bandwidth.
  • the techniques described above are used for efficiently storing video streams of an event for later playback (e.g., after the event has ended).
  • the amount of mass storage space consumed by 6-12 5k video streams is significant.
  • capture PODs capture video using motion JPEG (see, e.g., FIG. 10 , and MJPEG encoder 1007 ) which consumes significant bandwidth and storage space.
  • the transformation matrices are used to reconstruct those video streams which were not recorded. For example, only the left eye cameras may be recorded, and the transformation matrices may be used to reconstruct all of the right eye video streams.
  • a difference calculation unit may determine differences between the left and right streams. These differences can then be stored along with one of the two streams. For example, a disparity between adjacent streams (potentially from different pods) may be calculated and only one complete motion jpeg stream may be saved/transmitted. The other stream may be saved using differences between the motion jpeg stream and then reconstructed at the decoder, thereby removing a significant amount of redundancy.
  • Depth maps may also be generated and used by the algorithm to perform reconstruction of the original stream(s). For example, a monoscopic feed and a depth map may be used to reconstruct a stereo feed. The resolution of this depth map can be quite low. Disparity every inch, for example, is not required. At a low granularity, the depth map can be encoded using 8 bits total (e.g., granularity of 5-10 feet). Special types of processing may be performed for occluded objects (e.g., switching to data reduncancy).
  • one embodiment of the invention includes multiple transcoders 2004 , 2012 to composite video or graphics from another source as a key and fill operation to the synchronized multi-camera VR feeds described herein.
  • the key is implemented as an alpha channel and fill is implemented as the color channel.
  • a first video source 2000 receives key and fill input 2002 from one or more sources.
  • Video processing circuitry/software 2003 equipped with a serial digital interface (SDI) (potentially on an SDI card) performs interlaced-to-progressive conversion. In one embodiment, this is accomplished by one or more Teranex standards converters, although the underlying principles of the invention are not limited to any particular digital video formats or converters.
  • SDI serial digital interface
  • the progressive video streams are sent via one or more SDI outputs to a first transcoder 2004 which performs key and fill data aggregation on the inputs.
  • the resulting stream is packetized and transmitted to a second transcoder 2012 .
  • the Real-time Transport Protocol RTP
  • the second transcoder 2012 also receives a “background” video stream from a second video source 2010 which, in one implementation, is video captured by one or more capture PODs 1001 .
  • the second transcoder 2010 then overlays the key and fill stream onto the background video stream, effectively allowing different types of graphics and graphical effects to be displayed within the panoramic virtual reality image.
  • the overlay and background video are synchronized.
  • Parallax can be applied to the overlay so that the view can include depth effects within the panoramic virtual reality video.
  • the composited video or graphics can be used to show event-related, real-time data (such as a game clock, score, statistics, or other relevant data) or can be used as virtual jumbotron and/or a virtual advertisement board.
  • the background video is in received in a stereo format, with a left eye view and a right eye view.
  • the overlay video received from video source 2000 may have two channels, one for color and one for transparency.
  • the two videos are timestamped by a single synchronizer and transported over RTP.
  • the transcoder 2012 which may be a compositing video server, receives and aggregates (buffers) timestamped video frames from both sources 2000 , 2010 and finds matching frames based on the timestamps to composite the overlay video over the background video.
  • the transcoder 2012 applies parallax to the overlay (e.g., by locating the overlay in slightly different positions for the right and left eyes) to give the viewer a sense of depth in the virtual reality scene.
  • the embodiments described above provide the ability to composite video or graphics from another source as key and fill using the alpha channel and color channel, respectively, to the synchronized multi-camera virtual reality feeds (video source 2010 ).
  • Some embodiments described herein employ a distributed architecture in which service components are accessed remotely through a remote-access protocol, so these components can communicate across different processes, servers and networks. Similar to Object-Oriented Design (OOD) in software architecture, distributed architectures lend themselves to more loosely coupled, encapsulated and modular applications. This in turn promotes improved scalability, modularity and control over development, testing, and deployment of back-end service modules.
  • OOD Object-Oriented Design
  • portions of the overall architecture may be encapsulated into independent services.
  • a first Microservice is used for heart-beat injection
  • a second Microservice for capture controls a third Microservice for meta-data injection
  • a fourth Microservice for real-time operation monitoring All services may be developed and maintained independently but designed to work with the overall system.
  • microservices are service-oriented architecture (SOA)
  • SOA service-oriented architecture
  • the principle for microservices is to take the modularity of service-oriented architecture further into smaller and more manageable functional units.
  • the illustrated microservices application 2103 comprises a plurality of interconnected microservice components 2104 - 2105 which may be independently executed and updated.
  • microservices are used to implement components of the virtual reality architecture described herein.
  • FIG. 22 illustrates one such embodiment in which microservices 2211 - 2214 are executed to perform configuration, monitoring, and management of various portions of the panoramic virtual reality system.
  • the VR system is logically subdivided into components on which configuration, management, and monitoring may be performed.
  • the particular example illustrated in FIG. 22 includes an image processing pipeline 2201 , a transport pipeline 2202 , a video encoding pipeline 2203 , and a streaming/decoding/rendering pipeline 2204 .
  • One or more probes 2251 - 2254 are configured to monitor operation of each respective processing component 2201 - 2204 .
  • each probe 2251 - 2254 is executed on one or more servers within the processing component 2201 - 2204 for which it performs monitoring.
  • the server may be the same server used to perform the pipeline processing functions or may be implemented as a dedicated monitoring/management/configuration server.
  • one or more servers may be deployed to support the various microservices 2211 - 2213 .
  • the probes 2251 - 2254 run continuously, monitoring various relevant quality metrics for each respective processing component 2201 - 2204 including, but not limited to, latency metrics, load/resource allocation metrics, and/or bandwidth metrics.
  • the probes 2251 - 2254 report these quality metrics back to each respective microservice 2211 - 2213 which may implement a designated response (e.g., reallocating processing resources, offloading work from overloaded servers or GPUs, etc).
  • the microservices 2211 - 2213 may generate notifications to a supervisory microservice 2230 which is capable of rendering system-wide allocation decisions (e.g., moving processing resources from one processing component to another).
  • Communication between the microservices 2211 - 2213 and the supervisory microservice may be direct or via a microservice API gateway 2220 .
  • One or more clients 2231 such as management consoles and/or GUI-based clients may provide user access the data compiled by the microservices 2211 - 2214 , 2230 and to the user to configure the microservices to implement corrective responses.
  • the corrective responses are indicated by the dotted arrows originating from the microservices 2211 - 2213 to each respective processing component 2201 - 2204 . Only one microservice 2211 - 2214 is illustrated for each processing component 2201 - 2204 for simplicity. However, in one embodiment, a first set of microservices may collect and analyze the data provided by the probes 2251 - 2254 , while a second set of microservices, in communication with the first set, may implement corrective actions.
  • a load balancing microservice may allocate additional processing resources to the video encoding pipeline 2203 (e.g., allocating a new server or set of video encoding resources to handle the load with acceptable quality).
  • the microservices 2211 - 2213 , probes 2251 - 2254 , and other configuration/monitoring/management components may implement remote procedure calls (RPCs) to perform the various operations described herein.
  • RPCs remote procedure calls
  • the underlying principles of the invention are not limited to any particular techniques for distributed data collection and control.
  • Each microservice 2211 - 2213 , 2230 may be deployed with a Docker container, making it easy for other scripts or executables to launch the microservices.
  • Each physical server can host as few as one microservice container or as many as N containers. Orchestration tools such as Swarm and Kubernetes may be useful in managing the configuration and deployment of these microservice containers.
  • FIG. 23 illustrates one specific embodiment in which supervisory microservices 2301 - 2303 executed within Docker containers 1-3 on a supervisory server 2300 perform monitoring of different sets of processing resources 2311 - 2313 , respectively.
  • a monitoring request is generated by an RPC client on the supervisory microservices 2301 - 2303 and executed by an RPC server on the processing resources 2311 - 2313 (e.g., the probe is implemented as an RPC server).
  • CQ Completion Queue
  • a finite wait time between each read/write cycle RPC is implemented in one embodiment.
  • Each RPC call is not instantaneous because there is an overhead communicating across the boundaries of physical computing machines. In fact, two successive read/write operations with no wait time between the calls would likely result in an RPC failure. In one implementation, 500 milliseconds is used as the default wait time.
  • FIG. 16 illustrates a specific implementation which operates according to the following sequence of operations:
  • the sporting Broadcast Data Feed (BDF) 2601 pushes data by doing an HTTP POST to a registered endpoint.
  • the Scorecard Receiver receives Request from BDF and stores the data in a database 2603 in an S3 instance.
  • Data Parser downloads the corresponding file from S3, processes the file and stores the required information in the database 2603 .
  • Scorecard API Servers (EC2s part of an Auto-Scaling group) serve APIs called by mobile clients 2604 (through the content distribution network (CDN)).
  • CDN content distribution network
  • Scoreboard client 2604 fetches the box score and injects it into the video stream as a part of the metadata.
  • the response packet contains 23 fields separated by backslash characters. Here are the expected values of each field:
  • FIG. 27 An example of a scoreboard user interface is illustrated in FIG. 27 .
  • the response packet contains 29 fields separated by backslash characters. Here are the expected values of each field:
  • FIG. 28 An example user interface is illustrated in FIG. 28 .
  • the response packet contains variant numbers of fields according to the number of players. Each field is separated by backslash characters and the maximum number of players should be 4.
  • FIG. 29 An example user interface is illustrated in FIG. 29 .
  • RTMP Real Time Streaming Protocol
  • a modified version of RTMP is used.
  • one embodiment of the invention takes a 180° ⁇ 60° or 180° ⁇ 90° video stream and formats it for a 360° view. In one embodiment, this is done by filling in the space around the video frame with either static or dynamic content. This may be accomplished, for example, using a static overlay or dynamic overlay, respectively.
  • Traditional video sources for RTMP endpoints such as those found in social media platforms capture video with a spherical lens camera, allowing scenes to be rendered in a 360° view. Such video data can be delivered to 360° viewers without adjusting, altering or adding to the video data.
  • video data can be delivered to 360° viewers without adjusting, altering or adding to the video data.
  • FOV field of view
  • One embodiment of the cameras used for example, capture video streams with cameras that have a 180° horizontal FOV.
  • one embodiment of the invention not only transcodes video data to the appropriate resolutions and bitrates, but also ensures proper rendering in viewers by embedding static or dynamic content to complete a 360° view.
  • Traditional VR apps have the 360° environment view statically built into the application and do not necessarily require the source stream to add this in the stream.
  • the video capture engine (e.g., TrueVR) captures the video content, embeds static or dynamic content to complete a 360° view, transcodes the data to the correct resolution, rate, format and leverages a distribution server (e.g., a live streaming server such as Wowza) to forward and distribute the RTMP stream to the media endpoints.
  • a distribution server e.g., a live streaming server such as Wowza
  • the video capture engine captures the video content, embeds static or dynamic content to complete a 360° view, transcodes the data to the correct resolution, rate, format and delivers the RTMP streams directly to the media endpoints.
  • the one embodiment of the system To embed static content—such as 360° background overlays—into a live-media streaming system with minimal intrusion on the rest of the system, the one embodiment of the system employs a “constant-function” definition of static media, in which the static media is defined as a type of pseudo-video that produces the same frame repeatedly. This allows the system to use static media in any context in which standard video would have been allowed—most notably, as a background on which to overlay videos, be it for RTMP, HLS, or any other streaming protocol.
  • One embodiment of the system also employs a caching mechanism to minimize the performance impact of this pseudo-video, so that it may re-use the same memory for each frame of the pseudo-video.
  • CMD1 COM.SM.FD_GET_DESTINATION retrieve the S3 bucket upload location from the current Panoptes process.
  • CMD2 COM.SM.FD_START Instruct the MP4Stream Fault Detector (running on a worker thread in an active Panoptes process) to start collecting frame information.
  • CMD3 COM.SM.FD_STOP Unless the backup stream is writing to another S3 location than the main stream, we need to tell the main stream to stop writing to prevent overwriting files. This is needed when the main stream has non-total failures.
  • CMD4 COM.SM.FD_RESTART
  • a stop/start combination command CMD5 COM.SM.FD_GET_FRAME_INFO
  • supervisor will keep reading from the completion queue until no more READ tags remains. The frequency for polling may affect the overall performance. Supervisor should be configured to poll for frame data chunks in an time interval optimized for performance and latency tradeoffs.
  • Response REPLY1 COM.SM.FD — Respond with a sequence of framed data.
  • REPLY2 COM.SM.FD Respond with the S3 URI location DESTINATION: Format: URI delimited by “:” xxx.xxx.xxxx.xxx: xxxxx REPLY3 COM.SM.FD_OK Request is executed successfully.
  • REPLY4 COM.SM.FD_INVALID Request is not executed because of invalid parameter or request.
  • REPLY5 COM.SM.FD_ERROR Request is valid but there is an error.
  • a waterTank algorithm is implemented to detect the delayed or missed frame information.
  • the water tank keeps draining at a constant drain rate while newly arrived frame info samples from MP4 fault detector fills up the water tank.
  • SupervisorMicroservice::runFailureFallback( ) will be triggered to start the backup server.
  • waterLevel The initial water level to begin with. No unit is specified. The algorithm works as long as the same unit is used consistently across.
  • drainRate The amount of water to be drained away periodically. Water drainage is triggered each time when DF_GET_FRAME_INFO is called.
  • Configuration Condition Description SupervisorConfigure.h static, hard coded default values for compile configurable parameters; intended for time serving as fallback when parameters have not been set properly.
  • SMS.config static run assigned values for configurable time parameters, loaded once during object instantiation; suitable for scripting and one time initialization.
  • RPC Commands variable commands for assigning new run time configurable parameters via RPC; intending for modifying a configurable object's behavior.
  • a SM instance is run independently of Panoteps process. There could be one or many instances of SM in a physical server. Alternatively, each SM instance could be running on a Lambda instance in the cloud such as Amazon Web Services (AWS). If we want to adopt AWS only back-end architecture, Lambda based deployment will work seamlessly with AWS gateway, S3 storage and EC2 instances.
  • AWS Amazon Web Services
  • a recommended way, however, is to deploy a SM instance in a Docker container.
  • a virtualized environment there could be many containers in a physical server while the logics and I/O of each SM instance are independently contained.
  • the physical machine can simply be an additional server running next to the existing transcoder servers. This approach based on Docker container gives us the maximum flexibility when it comes to design our back-end server-based or server-less architecture.
  • One embodiment of the invention uses a unicast IP protocol such as RTMP which utilizes the connection-based TCP protocol for transmitting medium-volumes of media with low latency and guarantees that packets will arrive in order, which means error handling on the client-side is minimal.
  • RTMP unicast IP protocol
  • Embodiments of the invention may include various steps, which have been described above.
  • the steps may be embodied in machine-executable instructions which may be used to cause a general-purpose or special-purpose processor to perform the steps.
  • these steps may be performed by specific hardware components that contain hardwired logic for performing the steps, or by any combination of programmed computer components and custom hardware components.
  • instructions may refer to specific configurations of hardware such as application specific integrated circuits (ASICs) configured to perform certain operations or having a predetermined functionality or software instructions stored in memory embodied in a non-transitory computer readable medium.
  • ASICs application specific integrated circuits
  • the techniques shown in the figures can be implemented using code and data stored and executed on one or more electronic devices (e.g., an end station, a network element, etc.).
  • Such electronic devices store and communicate (internally and/or with other electronic devices over a network) code and data using computer machine-readable media, such as non-transitory computer machine-readable storage media (e.g., magnetic disks; optical disks; random access memory; read only memory; flash memory devices; phase-change memory) and transitory computer machine-readable communication media (e.g., electrical, optical, acoustical or other form of propagated signals—such as carrier waves, infrared signals, digital signals, etc.).
  • non-transitory computer machine-readable storage media e.g., magnetic disks; optical disks; random access memory; read only memory; flash memory devices; phase-change memory
  • transitory computer machine-readable communication media e.g., electrical, optical, acoustical or other form of propagated signals—such as carrier waves, infrared signals, digital signals, etc.
  • such electronic devices typically include a set of one or more processors coupled to one or more other components, such as one or more storage devices (non-transitory machine-readable storage media), user input/output devices (e.g., a keyboard, a touchscreen, and/or a display), and network connections.
  • the coupling of the set of processors and other components is typically through one or more busses and bridges (also termed as bus controllers).
  • the storage device and signals carrying the network traffic respectively represent one or more machine-readable storage media and machine-readable communication media.
  • the storage device of a given electronic device typically stores code and/or data for execution on the set of one or more processors of that electronic device.
US16/958,698 2018-02-07 2019-01-24 Method and apparatus for processing and distributing live virtual reality content Active US11282169B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/958,698 US11282169B2 (en) 2018-02-07 2019-01-24 Method and apparatus for processing and distributing live virtual reality content

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201862627747P 2018-02-07 2018-02-07
PCT/US2019/014862 WO2019156819A1 (en) 2018-02-07 2019-01-24 Method and apparatus for processing and distributing live virtual reality content
US16/958,698 US11282169B2 (en) 2018-02-07 2019-01-24 Method and apparatus for processing and distributing live virtual reality content

Publications (2)

Publication Number Publication Date
US20200349672A1 US20200349672A1 (en) 2020-11-05
US11282169B2 true US11282169B2 (en) 2022-03-22

Family

ID=67548556

Family Applications (1)

Application Number Title Priority Date Filing Date
US16/958,698 Active US11282169B2 (en) 2018-02-07 2019-01-24 Method and apparatus for processing and distributing live virtual reality content

Country Status (4)

Country Link
US (1) US11282169B2 (de)
CN (1) CN111542862A (de)
DE (1) DE112019000271T5 (de)
WO (1) WO2019156819A1 (de)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11336750B1 (en) * 2019-06-10 2022-05-17 EMC IP Holding Company LLC Remote procedure calls that offload search pattern matching from clients to servers
TR201910049A2 (tr) * 2019-07-05 2021-01-21 Karinca Teknoloji Ve Ilet San Tic Ltd Sti Üç boyutlu ortam aktarma, yayinlama si̇stemi̇ ve yöntemi̇
US20210329100A1 (en) * 2020-04-10 2021-10-21 Oracle International Corporation System and method for use of remote procedure call with a microservices environment
IL289178A (en) * 2021-12-20 2023-07-01 HAREL Shai An advanced multimedia system for accurate analysis and simulation of live events
CN114827633B (zh) * 2022-06-17 2022-09-13 浙江华创视讯科技有限公司 一种媒体流容灾方法、装置和相关设备

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6359617B1 (en) * 1998-09-25 2002-03-19 Apple Computer, Inc. Blending arbitrary overlaying images into panoramas
US20070110335A1 (en) * 2004-11-12 2007-05-17 Microsoft Corporation Image Processing System for Digital Collage
US20080024594A1 (en) 2004-05-19 2008-01-31 Ritchey Kurtis J Panoramic image-based virtual reality/telepresence audio-visual system and method
US20080056612A1 (en) * 2006-09-04 2008-03-06 Samsung Electronics Co., Ltd Method for taking panorama mosaic photograph with a portable terminal
US20080240612A1 (en) * 2007-03-30 2008-10-02 Intel Corporation Non-overlap region based automatic global alignment for ring camera image mosaic
US20150304634A1 (en) 2011-08-04 2015-10-22 John George Karvounis Mapping and tracking system
US20160353090A1 (en) * 2015-05-27 2016-12-01 Google Inc. Omnistereo capture and render of panoramic virtual reality content
US20170061686A1 (en) * 2015-08-28 2017-03-02 Hai Yu Stage view presentation method and system
US20170154445A1 (en) 2015-11-26 2017-06-01 Fujitsu Limited Method for processing graphs and information processing apparatus
KR101764063B1 (ko) 2016-07-27 2017-08-03 네이버 주식회사 Vr 컨텐츠의 분석 및 프리-렌더링을 위한 방법 및 시스템
US20180343372A1 (en) * 2017-05-26 2018-11-29 Hon Hai Precision Industry Co., Ltd. Printed circuit board and method of controlling camera

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017091019A1 (en) * 2015-11-27 2017-06-01 Samsung Electronics Co., Ltd. Electronic device and method for displaying and generating panoramic image

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6359617B1 (en) * 1998-09-25 2002-03-19 Apple Computer, Inc. Blending arbitrary overlaying images into panoramas
US20080024594A1 (en) 2004-05-19 2008-01-31 Ritchey Kurtis J Panoramic image-based virtual reality/telepresence audio-visual system and method
US20070110335A1 (en) * 2004-11-12 2007-05-17 Microsoft Corporation Image Processing System for Digital Collage
US20080056612A1 (en) * 2006-09-04 2008-03-06 Samsung Electronics Co., Ltd Method for taking panorama mosaic photograph with a portable terminal
US20080240612A1 (en) * 2007-03-30 2008-10-02 Intel Corporation Non-overlap region based automatic global alignment for ring camera image mosaic
US20150304634A1 (en) 2011-08-04 2015-10-22 John George Karvounis Mapping and tracking system
US20160353090A1 (en) * 2015-05-27 2016-12-01 Google Inc. Omnistereo capture and render of panoramic virtual reality content
US20170061686A1 (en) * 2015-08-28 2017-03-02 Hai Yu Stage view presentation method and system
US20170154445A1 (en) 2015-11-26 2017-06-01 Fujitsu Limited Method for processing graphs and information processing apparatus
KR101764063B1 (ko) 2016-07-27 2017-08-03 네이버 주식회사 Vr 컨텐츠의 분석 및 프리-렌더링을 위한 방법 및 시스템
US20180343372A1 (en) * 2017-05-26 2018-11-29 Hon Hai Precision Industry Co., Ltd. Printed circuit board and method of controlling camera

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
International Preliminary Report on Patentability, PCT App. No. PCT/US2019/014862, dated Aug. 20, 2020, 8 pages.
PCT Notification of Transmittal of the International Search Report and the Written Opinion of the International Searching Authority, or the Declaration from PCT/US2019/014862, dated May 14, 2019, 11 pages.

Also Published As

Publication number Publication date
US20200349672A1 (en) 2020-11-05
DE112019000271T5 (de) 2020-09-10
WO2019156819A1 (en) 2019-08-15
CN111542862A (zh) 2020-08-14

Similar Documents

Publication Publication Date Title
US11381739B2 (en) Panoramic virtual reality framework providing a dynamic user experience
US11282169B2 (en) Method and apparatus for processing and distributing live virtual reality content
Fan et al. A survey on 360 video streaming: Acquisition, transmission, and display
CN112738010B (zh) 数据交互方法及系统、交互终端、可读存储介质
JP6979035B2 (ja) 仮想現実メディアコンテンツのストリーミングを向上させる方法、デバイス及びコンピュータプログラム
US20220210512A1 (en) Content based stream splitting of video data
Lou et al. A real-time interactive multi-view video system
US20200388068A1 (en) System and apparatus for user controlled virtual camera for volumetric video
CN112738534B (zh) 数据处理方法及系统、服务器和存储介质
Shi et al. Freedom: Fast recovery enhanced vr delivery over mobile networks
EP2795919A1 (de) Ausrichtung von verschiedene blickpunkte darstellenden videos
CN112738495B (zh) 虚拟视点图像生成方法、系统、电子设备及存储介质
JP2022536182A (ja) データストリームを同期させるシステム及び方法
US10609440B1 (en) Timing data anomaly detection and correction
KR20210065151A (ko) 비디오 스트림을 생성하고 렌더링하기 위한 장치 및 방법
US10607396B2 (en) Dynamic rendering of low frequency objects in a virtual reality system
US11706375B2 (en) Apparatus and system for virtual camera configuration and selection
JP6355634B2 (ja) データ処理装置、データ処理方法、プログラム、記録媒体及びデータ処理システム
Gül et al. Low latency volumetric video edge cloud streaming
Zeng et al. A new architecture of 8k vr fov video end-to-end technology
CN112738009B (zh) 数据同步方法、设备、同步系统、介质和服务器
CN107835433B (zh) 一种赛事宽视角直播系统、相关联的设备和直播方法
CN112738646B (zh) 数据处理方法、设备、系统、可读存储介质及服务器
CN112734821B (zh) 深度图生成方法、计算节点及计算节点集群、存储介质
Budd et al. Web delivery of free-viewpoint video of sport events

Legal Events

Date Code Title Description
FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

AS Assignment

Owner name: INTEL CORPORATION, CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:COCHRAN, WAYNE;YEUNG, FAI;MATHUR, DURGA RAJ;AND OTHERS;SIGNING DATES FROM 20200721 TO 20201220;REEL/FRAME:054858/0966

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STCF Information on status: patent grant

Free format text: PATENTED CASE