HK1237168B - Video transmission based on independently encoded background updates - Google Patents

Description

Video transmission based on background updates based on independent coding

Technical Field

The present application relates generally to video transmission. Specifically, the present application relates to apparatus and methods for alleviating bandwidth limitations of video transmission and improving the quality of video at a receiver. More specifically, an improved video transmission system and method are provided for generating high-resolution video at a receiver based on independently encoded background and background updates.

Background Art

Real-time video communication systems and the emerging field of telepresence face inherent challenges as they attempt to simulate the experience of being in another physical space for remote users. This is because the human eye, with its ability to focus its high-resolution fovea on an object of interest, maintains a superior field of view compared to commercially available single-lens cameras with state-of-the - art resolution. See http://www.clarkvision.com/imagedetail/eye-resolution.html (the human eye's resolution is estimated at 576 megapixels over a 120-degree range). Furthermore, telepresence systems are practically limited by the network bandwidth available to most users. Therefore, it's not surprising that, beyond simple person-to-person video chats using the narrow-field-of-view cameras found in most tablets, phones, and laptops, telepresence has had limited uptake.

Automatic and manual pan-tilt-zoom (PTZ) cameras in commercial telepresence systems attempt to overcome the resolution limitations of single-lens cameras by optically and mechanically fixing the field of view to a selected portion of interest in the scene. This partially mitigates the resolution limitation, but it still has some drawbacks. For example, only one mechanical fixation can be performed at a given time; therefore, multiple remote users with different points of interest may not be able to be satisfactorily served. Furthermore, the zoom lens and mechanical pan-tilt mechanism increase the cost of the camera system and pose new challenges to the reliability of the entire system. Specifically, compared to manual systems, which typically maintain fewer move cycles over their service life, automatic PTZ systems place higher demands on the mechanics. Compared to fixed cameras, the bandwidth requirements for high-quality video encoding are also significantly increased. Similarly, some digital PTZ systems in existing systems suffer from many of the drawbacks described above, including, for example, the inability to be controlled by multiple remote users and the high bit rate requirements for video encoding.

Panoramic and ultra-wide-angle video cameras can meet the resolution requirements of telepresence systems to provide an ideal user experience. These cameras have the potential to increase sensor resolution and pixel rate far beyond existing standards. This can be achieved, for example, through curved sensor surfaces and single-center lens designs. See http://www.jacobsschool.ucsd.edu/news/news_releases/release.sfe?id=1418 ( discussing a 120-degree FOV imager with a resolution of at least 85 megapixels); http://image-sensors-world.blogspot.co.il/2014/04/vlsi-symposia-sony-presents-curved.html (sensor manufacturer unveils prototype curved image sensor). However, such designs would place significant strain on current network capacity and video coding efficiency, making them impractical for widespread, real-world deployment. For example, an 85-megapixel camera operating at 30 frames per second would require compression as low as 0.0002 bits per pixel to accommodate a 10 megabit per second (Mbit/s) link. This is generally unattainable today considering current video compression standards such as H.264 which run at 0.05 bits per pixel under good conditions.

Therefore, there is a need for improved methods and systems to alleviate the bandwidth limitations of video transmission and generate high-resolution video based on traditional camera hardware. It is also necessary to utilize these improvements to enable modern real-time communication systems and ideal telepresence experiences.

Summary of the Invention

Therefore, the purpose of this application is to provide methods and systems for alleviating bandwidth limitations on video transmission, thereby generating wide-angle, high-resolution videos using traditional hardware devices.

In particular, according to the present application, in one embodiment, a method for transmitting a video is provided, the method comprising: 1) initializing a background model by determining a static background of the scene from the video; and 2) transmitting the background of the scene as the background model by encoding the background model independently from the video. The background model is incrementally updated, and the updates are further encoded and transmitted independently from the video.

In another embodiment, the method further comprises generating an enhanced video at a receiver by merging the background with the video. In yet another embodiment, the background model is updated and transmitted at a lower bit rate than the bit rate of the video. In a further embodiment, the method further comprises transmitting a geometric mapping between the background and the video for each frame.

In another embodiment, the method further comprises determining the field of view of the video by scene analysis.In yet another embodiment, the background model is used to suppress noise variations in the background of the video.

According to one embodiment, the method of the present application further comprises compressing the video using a standard video codec. In another embodiment, the video codec is one of H.264, H.265, VP8, and VP9. In yet another embodiment, the background is sent in an auxiliary data channel defined by one of H.264, H.265, VP8, and VP9.

According to another embodiment, the background model is a parametric model.In a further embodiment, the parametric model is a Mixture of Gaussians (MOG).

According to yet another embodiment, the background model is a non-parametric model.In a further embodiment, the non-parametric model is a visual background extractor (ViB).

According to another embodiment of the present application, a method for simulating pan-tilt zoom operations on a video of a scene is provided, the method comprising: 1) initializing a background model by determining a static background of the scene from the video; 2) sending the background of the scene as the background model by encoding the background model independently from the video, wherein the background model is incrementally updated, wherein the update is further encoded and sent independently from the video, and wherein a geometric mapping between the background and the video is sent for each frame; and 3) selecting one or more fields of view of the video through scene analysis; and generating an enhanced video at a receiver by merging the background with the video.

In another embodiment, the method further comprises controlling the simulated pan-tilt-zoom operation at the receiver. In yet another embodiment, the method further comprises controlling the simulated pan-tilt-zoom operation at the transmitter of the video.

According to another embodiment of the present application, a system for transmitting a video of a scene is provided, the system comprising: 1) a transmitter comprising an external encoder and a core encoder, wherein the external encoder is adapted to receive the video and output a significant video and background and geometry bitstreams to the core encoder, respectively, wherein the core encoder is adapted to output an encoded bitstream; and 2) a receiver comprising a core decoder, wherein the core decoder is adapted to receive the encoded bitstream and output the significant video.

According to a further embodiment of the present application, a system for transmitting a video of a scene is provided, the system comprising: 1) a transmitter comprising an external encoder and a core encoder, wherein the external encoder is adapted to receive the video and output the salient video and background and geometry bitstreams separately to the core encoder, wherein the core encoder is adapted to output the encoded bitstream; and 2) a receiver comprising a core decoder and an external decoder, wherein the core decoder is adapted to receive the encoded bitstream and output the salient video and the background and geometry bitstreams separately to the external decoder, wherein the external decoder is adapted to merge the salient video and the background and geometry bitstreams to output an enhanced video of the scene.

In another embodiment, the outer encoder further comprises a background estimation unit adapted to initialize a background model by determining a static background of the scene from the video and to incrementally update the background model at a bit rate lower than the bit rate of the video. In yet another embodiment, the outer encoder further comprises a background encoder coupled to the background estimation unit. The background encoder is adapted to encode the background model and the updates independently of the video. In a further embodiment, the background encoder comprises an entropy encoder, an entropy decoder, an update prediction unit, and an update storage unit.

According to another embodiment, the background encoder is connected to a bitstream multiplexer in a downstream direction. In yet another embodiment, the outer encoder further comprises a saliency framing unit adapted to output the geometry bitstream to the bitstream multiplexer. The bitstream multiplexer is adapted to merge the geometry bitstream with the background bitstream, thereby outputting the background and geometry bitstreams.

In a further embodiment, the outer encoder further comprises a downscaling unit capable of scaling and cropping the video. The downscaling unit is connected to a noise suppression unit in a downstream direction. The noise suppression unit is adapted to suppress noise in the salient video based on the background model.

According to another embodiment, the outer decoder further comprises: i) a bitstream demultiplexer adapted to receive the background and geometry bitstreams from the core encoder and output the geometry bitstream and the background bitstream, respectively; ii) a background decoder connected to the bitstream demultiplexer and adapted to receive the background bitstream; and iii) a background merging unit connected to the bitstream demultiplexer and the background decoder in a downstream direction. The background merging unit is adapted to receive the salient video from the core decoder and merge the geometry bitstream and the background bitstream with the salient video to produce an enhanced video of the scene.

In yet another embodiment, the background decoder includes an entropy decoder, an update prediction unit, and an update storage unit.

In a further embodiment, the external decoder further comprises a virtual pan-tilt-zoom unit capable of receiving control input to generate enhanced video.

According to another embodiment, the core encoder in the system of the present application is an H.264/H.265 video encoder, and the background and geometry bitstreams are carried through the network abstraction layer of the H.264/H.265 video encoder. In another embodiment, the core decoder in the system of the present application is an H.264/H.265 video decoder, and the background and geometry bitstreams are carried through the network abstraction layer of the H.264/H.265 video decoder.

In a further embodiment, the core encoder is in a multimedia container format, and the background and geometry bitstreams are carried via an auxiliary data channel of the core encoder. In another embodiment, the core decoder is in a multimedia container format, and the background and geometry bitstreams are carried via an auxiliary data channel of the core decoder.

According to yet another embodiment, the core encoder in the system of the present application is a standard video encoder, and the background and geometry bitstreams are carried via an auxiliary data channel of the core encoder. In a further embodiment, the core decoder is a standard video decoder, and the background and geometry bitstreams are carried via an auxiliary data channel of the core decoder.

According to another embodiment of the present application, a method for transmitting and presenting a video of a scene from multiple fields of view is provided, the method comprising: (1) initializing a three-dimensional background model by determining a static background of the scene from the video; (2) sending the background of the scene as the background model by encoding the background model independently from the video, wherein the background model is incrementally updated, and wherein the update is further encoded and sent independently from the video; and (3) presenting an enhanced video at a receiver by merging the background with the video.

In yet another embodiment, the receiver is a VR/AR device. In a further embodiment, the method further comprises: self-learning a region of interest in a direction of sight from the VR/AR receiver; and transmitting a high-resolution video of the region of interest, wherein the enhanced video is created by merging the high-resolution video of the region of interest with the background.

According to another embodiment of the present application, a system for transmitting and presenting a video of a scene from multiple fields of view is provided, the system comprising: (1) a transmitter comprising an external encoder and a core encoder, wherein the external encoder is adapted to receive the video and output a salient video and a 3D background and geometry bitstream to the core encoder, respectively, wherein the core encoder is adapted to output an encoded bitstream; and (2) a VR/AR receiver comprising a core decoder and an external decoder, wherein the core decoder is adapted to receive the encoded bitstream and output the salient video and the background and geometry bitstream to the external decoder, respectively, wherein the external decoder is adapted to merge the salient video and the background and geometry bitstream to present an enhanced video of the scene. In another embodiment, the 3D background model is updated incrementally.

In a further embodiment, the outer encoder comprises a background estimation unit adapted to initialize a three-dimensional background model by determining a static background of the scene from the video and to incrementally update the background model at a lower bitrate than the bitrate of the video.

In a further embodiment, the system further comprises a video source for capturing the scene. In another embodiment, the video source comprises one or more cameras with partially overlapping fields of view. In yet another embodiment, the cameras are mobile cameras. In a further embodiment, the system is adapted to estimate moving and stationary portions of the scene. In another embodiment, the outer encoder comprises a background estimation unit adapted to generate a three-dimensional background model based on the stationary portion of the scene and to incrementally update the background model at a bit rate lower than the bit rate of the video.

In a further embodiment, the mobile camera is a PTZ camera.In another embodiment, the VR/AR receiver is adapted to self-learn an area of interest from its line of sight, and wherein the one or more PTZ cameras are adapted to capture high-resolution video of the area of interest.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 shows a video transmission system according to an embodiment of the present application.

FIG2 shows an outer encoder of a video transmission system according to another embodiment.

FIG3 shows an outer decoder of a video transmission system according to another embodiment.

FIG4 shows an H.264/H.265 core encoder of a video transmission system according to another embodiment.

FIG5 shows an H.264/H.265 core decoder of a video transmission system according to another embodiment.

FIG6 shows a multimedia container format core encoder of a video transmission system according to another embodiment.

FIG7 shows a multimedia container format core decoder of a video transmission system according to another embodiment.

FIG8 shows a standard video encoder with an auxiliary data channel as a core encoder of a video transmission system according to another embodiment.

FIG9 shows a standard video decoder with an auxiliary data channel as a core decoder of a video transmission system according to another embodiment.

FIG10 shows a background encoder in a video transmission system according to another embodiment.

FIG. 11 shows a background decoder in a video transmission system according to another embodiment.

DETAILED DESCRIPTION

The methods and systems according to various embodiments of the present invention employ a background model based on which the background of a scene in a video is encoded and incrementally updated. The encoded background and updates are transmitted independently of the video. At the receiver, the background can then be merged with the video to produce an enhanced, high-resolution video.

Method Overview

In one embodiment, for example, a video of a scene including both foreground and background is transmitted. It is compressed using a standard video codec such as H.264. The static background of the scene is transmitted as a background model, which is incrementally updated at a lower bit rate than the video. The background model is generated and initialized from the static background of the video based on established surveillance system technology.

In an alternative embodiment, multiple cameras with partially overlapping fields of view are deployed as video sources, which generate one or more synchronized and coordinated video streams for transmission and presentation. In some embodiments, such video sources include mobile cameras. The moving and stationary portions of the scene are estimated based on the video streams, and thus, a three-dimensional background model is generated based on the stationary portion of the image.

In another embodiment, the field of view of the transmitted video is automatically limited by scene analysis, for example, limiting the field of view to human subjects, to better utilize the resolution of the video format. According to this embodiment, the exact spatial relationship between the video and the background is transmitted for each frame.

In yet another embodiment, a background model is used to suppress spurious noise in the background of the video. Background model data and other related information are sent in an auxiliary data channel defined by a video standard such as H.264. This background and related data can be ignored and bypassed by a decoder that is configured not to interpret the data carried by the auxiliary data channel. Thus, the system according to this embodiment provides the flexibility to integrate with older and existing pre-existing systems.

In some embodiments, at the receiver, the output from the background model is merged with the video to produce an enhanced video. In a specific embodiment, PTZ operations are simulated on the enhanced video at the receiver. According to one embodiment, the simulated PTZ operations are controlled at the transmitter or receiver. According to alternative embodiments, the control is performed by a user or through automated processing at the transmitter or receiver.

Background Processing

Some existing video encoders apply foreground-background segmentation, in which the background is subtracted from the video before encoding and the background is sent separately. According to one embodiment of the present application, a standard video encoder such as H.264 or H.265 is used to encode both the foreground and background videos. In this embodiment, stray noise in the background is suppressed by comparing the input video pixels with the predicted pixel states of the background model. Therefore, in this embodiment, a nearly still image in the background area is provided to the video encoder. The background model is sent and incrementally updated in an auxiliary channel of the standard codec. Therefore, the background sending method according to this embodiment relaxes the bandwidth requirements for video transmission and enables high-resolution video to be presented at the receiver by merging the background update with the video.

According to one embodiment, the video is decoded by a standard decoder that is unaware of the background model data. The standard decoder ignores the unknown auxiliary fields and bypasses the background model data. The system of this embodiment utilizes the existing core video codec, which provides a lower implementation cost. Therefore, the system of this embodiment provides backward compatibility with older and existing systems.

In another embodiment, the systems and methods of the present application send the background at an enhanced level of representation relative to the foreground. In a specific embodiment, the background data is sent at a higher resolution and a higher dynamic range. This is advantageous for a number of reasons. For example, while a conventional hybrid video codec could be modified to send high resolution intraframes and predicted frames at a lower resolution, intraframes may require many bits to encode and therefore may not be possible to transfer in a low latency implementation without interrupting the video stream. With background sending in the outer layer according to this embodiment, the core video transmission proceeds normally without interruption because the background sending is completing.

Compared to high-resolution intra frames, according to this embodiment, the core encoder can be made simpler by using background transmission in the outer layer. This provides cost savings and wide system compatibility.

Simulated pan-tilt zoom

According to another embodiment, as described above, the system of the present application simulates PTZ operation. In this embodiment, the view is determined by the simulated PTZ processing on the receiving side, rather than being fixed on the transmitting side. Therefore, all receiving users can access different views on the other side. Because this simulated PTZ is not limited by mechanical structure, it is open to many additional transitions and transformations in other embodiments. In particular, in one embodiment, instantaneous switching between views and scrolling of views are provided.

These non-mechanical, analog PTZ systems according to the present application also provide cost savings compared to existing PTZ telepresence solutions and further enhance the reliability of telepresence.

Devices and components

1 , in one embodiment, the system of the present application includes a video source, a transmitter, and a receiver. In a specific embodiment, each of the video source, the transmitter, and the receiver is panoramic.

According to one embodiment, a panoramic video source is a device that provides a wide-angle or panoramic digital video stream. In this embodiment, it provides high-bitrate uncompressed video suitable for further processing. In one embodiment, the video source is a single lens and image sensor assembly; in another embodiment, it includes multiple lenses and sensors and suitable image stitching software or hardware that can simulate the operation of a single lens and sensor. In yet another embodiment, the video source includes a graphics rendering device that simulates the geometric projection of a three-dimensional (3D) scene onto a surface. Therefore, the system of this embodiment can be advantageously deployed for computer video games.

In one embodiment, the geometric projection of the panoramic video source may be different from the desired presentation projection. Therefore, it can be calibrated in a form suitable for embedding into the video transmitter during the design, manufacture or setup of the video source device, or forwarded to the video transmitter as side information. The transmitter in turn provides this information to the receiver, which can then be used to present the video using another projection. Therefore, the system of this embodiment provides considerable flexibility in presenting the video at the receiver based on the desired control, which can be built-in by design or input from the user. In alternative embodiments, such control can be implemented from the transmitter or receiver.

The transmitter of the system according to one embodiment includes an external encoder. Referring to Figure 2, in one embodiment, the external encoder receives a panoramic digital video stream and outputs a salient video stream, a coded background model update sequence, and geometric projection data. According to one embodiment, this data from the external encoder is then transmitted to the core encoder of the system. In one embodiment, the video stream is in uncompressed form and is suitable for compression by a standard video encoder. According to another embodiment, the coded background model data and geometric projection data are multiplexed and framed into a format suitable for transmission in an auxiliary data frame of a standard video encoder. The core encoder of the system in this embodiment outputs an encoded bitstream.

As shown in Figure 4, in one embodiment, the core encoder is an H.264/H.265 encoder. The H.264/H.265 core encoder uses the standard's network abstraction layer to send auxiliary data in an SEI header marked as user data. In one embodiment, this data is ignored by receivers not configured to receive such SEI headers. As described above, this system provides backward compatibility and facilitates integration into existing telepresence systems.

According to one embodiment, the background model used in the system of the present application is a parametric model. In such a parametric background model, multiple statistics are determined for each pixel based on samples from past video frames. According to another embodiment, the background model is a non-parametric model. In such a non-parametric background model, multiple samples from past video frames are stored or aggregated for each pixel - no statistics or parameters are determined in a finite-dimensional space. According to one embodiment, the non-parametric background model is a visual background extractor (ViBe). In another embodiment, the parametric background model is a Gaussian mixture (MOG). In certain embodiments of the present application, the background model of the system is a three-dimensional model and supports VR/AR applications. For the purposes of various embodiments of the present application, the term "three-dimensional" covers the following scenario, in which the model is an image from a single viewpoint, and the image from the single viewpoint has a depth for each point in the image, which is sometimes referred to as "2.5 dimensions".

According to one embodiment, the system's background model is initialized based on pixels in a video frame known to be background, either by controlling the scene or by bootstrapping using a simpler background model. In an alternative embodiment, the system assumes that all pixels are part of the background when the background model is initialized.

After initialization, in one embodiment, the background model is updated based on changes in the background in new samples that are determined to be or likely to be background according to the model.

According to one embodiment, the updates are encoded by predicting each update based on previously reconstructed updates and sending only the difference between the predicted update and the actual update, the residual. In another embodiment, the bit rate of the residual is further reduced by quantization and entropy coding.

10 and 11 , according to certain embodiments of the present application, updates are reconstructed by the same process in both the background encoder and the background decoder. The residual is first decoded by reversing entropy coding and quantization, and then each update or group of updates is predicted based on the previous update, and the actual update is reconstructed by adding the residual and the predicted update.

According to one embodiment, the system's transmitter includes an external encoder and a core encoder as shown in Figure 1. In this embodiment, the transmitter and its components are implemented in the same physical device. For example, the transmitter in one embodiment is a mobile system-on-chip (SoC). In some embodiments, the external encoder is implemented in software for a GPU or CPU core, and the core encoder is implemented using a hardware accelerator for video encoding equipped in such an SoC. The implementation of this SoC transmitter is beneficial for telepresence systems in which a mobile phone or tablet device provides the transmitter functionality.

In another embodiment, the transmitter is implemented in a custom SoC for the camera. In addition to the accelerator for video encoding, other functions are implemented as software running on the DSP core. This particular embodiment of the transmitter is beneficial for telepresence systems that use stand-alone cameras.

As described above, the video receiver of the present application includes a core decoder. Referring to Figures 5, 7, and 9, in some embodiments, the core decoder receives a coded bitstream and outputs uncompressed video in addition to auxiliary data. According to these embodiments, the auxiliary data includes background model data and geometric mapping data. As shown in Figure 3, this data is transmitted to an external decoder, which, according to one embodiment, merges the significant video and background model outputs to produce an enhanced panoramic video stream. In another embodiment, the external decoder changes the geometric mapping of the video to simulate the effect of an optical PTZ camera.

In the event that the auxiliary data channel between the sender and receiver experiences packet loss or other reliability issues, the system in another embodiment of the present invention provides functionality to send a request to the sender to resend the lost packets. These may include portions of other sent metadata and background model data.

According to one embodiment, the video receiver of the system is implemented in a cloud service that runs on a general-purpose data center or media processor. In another embodiment, the receiver is implemented in a web browser of an end-user device such as a smart phone, tablet computer or personal computer. In the web browser, the receiver function is implemented in a specific embodiment by a browser extension or using standardized network components such as WebRTC (for core decoders) and WebGL (for external decoders). In another embodiment, the receiver is implemented as a native application (native application) in the operating system of an end-user device such as a smart phone, tablet computer or personal computer. In another embodiment, the receiver is implemented in an appliance dedicated to video communication.

In another embodiment, the receiver is implemented as part of a virtual reality (VR) or augmented reality (AR) system, along with an immersive eyewear display, head-mounted tracking, or alternative technologies that project selected images onto the user's retina. According to this embodiment, the apparatus and method of the present invention can alleviate bandwidth limitations of VR/AR-enabled video conferencing systems, where a remote live image is projected onto a near-end field of view.

In yet another embodiment, information about the VR/AR receiver's eye gaze and view direction is relayed back to the camera system of the present invention. High-resolution video from that particular view direction is sent accordingly, allowing for some additional margin around that particular view direction. In yet another embodiment, the system of the present invention adapts itself to learn to map areas of interest. Specifically, the VR/AR receiver analyzes eye gaze direction over time, and the areas that receive the most views or "hits" are encoded at a higher resolution for transmission and presentation.

According to one embodiment, the system of the present application includes a video source. In some embodiments, the video source includes one or more mobile PTZ cameras. These mobile PTZ cameras capture high-resolution video of a specific region of interest ("ROI"), and according to one embodiment, the high-resolution video is merged with a background. In this embodiment, the background is a still image and is presented at a higher resolution than the resolution of the ROI video, thereby enhancing the VR/AR experience.

According to one embodiment, mobile cameras are synchronized in time and coordinated in position, allowing efficient blending between ROI videos collected from multiple cameras.

In another embodiment using a spatially moving camera system as a video source, a three-dimensional model of the background is pre-generated using multiple fixed high-resolution cameras with partially overlapping fields of view (FOVs). In one embodiment, these cameras also include background and foreground segmentation filters to distinguish the moving parts of the scene from the non-moving parts. Only the background (stationary) portion of the scene is used to generate the 3D model of the scene. In an alternative embodiment, super-resolution imaging techniques are used to increase the resolution of the 3D model before generating the 3D model.

In yet another embodiment, a combination of gyroscopes and accelerometers for spatial and angular positioning, along with visual information for fine-tuning, is applied to a moving camera video feed. Using simultaneous localization and mapping (SLAM) technology, the system of the present application estimates which parts of a scene are moving and which are not, thereby generating a 3D model of the scene.

As an example, when a camera video source is moving, the system in one embodiment determines the moving parts of the scene according to the following steps. First, for each consecutive video frame, Harris corner feature points (or other types of feature points) are estimated; for each pair of video frames (the two are adjacent in time, and some pairs have a large time interval between them), the rotation and translation of the camera between the frames (with six axes of freedom) are estimated; and outliers are removed. Some outliers are due to noise, while other outliers reflect objects that have moved between frames. Second, for the Harris corners of the outliers, 3D motion vectors are introduced for the part of the scene containing the outliers; the movement of these points is estimated; and, for the feature points that have been moving together, 3D motion vectors are estimated. Thus, a 3D model based on the stationary part of the scene is generated, taking into account the pointing direction of the camera.

According to some embodiments, the receiver and transmitter in the system of the present application are implemented in the same device for two-way video communication.

Application Areas

According to various embodiments, the system of the present application can be advantageously deployed in real-time video communications (video conferencing and telepresence), live streaming (sports, concerts, event sharing, and computer game competitions), traffic monitoring (dashboard cameras, road monitoring, parking lot monitoring and billing), virtual reality, surveillance, home monitoring, storytelling, movies, news, social and traditional media, and art installations, among other applications and industries.

In live video and two-way communication VR/AR applications where bandwidth is insufficient to transmit high-resolution video of the entire scene, according to one embodiment, high-resolution still images (stills) of the entire field of view are periodically transmitted, while high-resolution video of selected areas of interest is transmitted at a regular frequency. In another embodiment, the video and still images (stills) are mixed locally at the VR/AR receiver, enabling fast AR/VR rendering and low latency. In this context, typical latency is 20ms or less.

The description of the various embodiments provided in the present application, including the various figures and examples, is illustrative of the present application and its various embodiments and is not intended to be limiting.

Claims (45)

1. A method for transmitting and presenting video of a scene from multiple fields of view, comprising: initializing a three-dimensional background model by determining a static background of the scene from the video; transmitting the background of the scene as the background model by encoding the background model independently of the video, wherein the background model is incrementally updated, and wherein the updates are further encoded and transmitted independently of the video, and wherein the incremental updates of the background model are transmitted via an auxiliary data channel; and presenting an enhanced video at a receiver by merging the background with the video. 2. The method according to claim 1, wherein the receiver is a VR/AR device. 3. The method of claim 2, further comprising: self-learning a region of interest from the gaze direction of the VR/AR receiver; and transmitting a high-resolution video of the region of interest, wherein the enhanced video is created by merging the high-resolution video of the region of interest with the background. 4. A system for transmitting and presenting video of a scene from multiple fields of view, comprising: i) a transmitter including an external encoder and a core encoder, wherein the external encoder is adapted to receive the video and output salient video and incrementally updated 3D background and geometry bitstreams to the core encoder, wherein the core encoder is adapted to output an encoded bitstream, and wherein the incremental updates of the background are transmitted via an auxiliary data channel; and ii) a VR/AR receiver including a core decoder and an external decoder, wherein the core decoder is adapted to receive the encoded bitstream and output the salient video and the incrementally updated background and geometry bitstreams to the external decoder, wherein the incremental updates of the background are received via an auxiliary data channel, and wherein the external decoder is adapted to merge the salient video and the incrementally updated background and geometry bitstreams to present an enhanced video of the scene. 5. The system of claim 4, wherein the external encoder includes a background estimation unit adapted to initialize a three-dimensional background model by determining a static background of the scene from the video, and to incrementally update the background model at a bit rate lower than that of the video. 6. The system according to claim 4 further includes: a video source for capturing the scene. 7. The system of claim 6, wherein the video source comprises one or more cameras having partially overlapping fields of view. 8. The system according to claim 7, wherein the camera is a mobile camera. 9. The system according to claim 8 is further adapted to estimate the moving and stationary portions of the scene. 10. The system of claim 9, wherein the external encoder includes a background estimation unit adapted to generate a three-dimensional background model based on the static portion of the scene, and to update the background model incrementally at a bit rate lower than that of the video. 11. The system according to claim 8, wherein the mobile camera is a pan-tilt-zoom (PTZ) camera. 12. The system of claim 11, wherein the VR/AR receiver is adapted to self-learn an area of interest from its line of sight, and wherein the one or more PTZ cameras are adapted to capture high-resolution video of the area of interest. 13. A method for transmitting video of a scene, comprising: initializing a background model by determining a static background of the scene from the video; and transmitting the background of the scene as the background model by encoding the background model independently of the video, wherein the background model is incrementally updated, and wherein the update is further encoded and transmitted independently of the video, and wherein the incremental update of the background model is transmitted via an auxiliary data channel. 14. The method of claim 13, further comprising: generating an enhanced video at the receiver by merging the background with the video. 15. The method of claim 14, wherein the background model is updated and transmitted at a lower bit rate than that of the video. 16. The method of claim 13, further comprising: sending a geometric mapping between the background and the video for each frame. 17. The method of claim 16, further comprising: determining the field of view of the video through scene analysis. 18. The method of claim 13, wherein the background model suppresses noise variations in the background of the video. 19. The method of claim 13, further comprising: compressing the video using a standard video codec. 20. The method of claim 19, wherein the video codec is one of H.264, H.265, VP8, and VP9. 21. The method of claim 20, wherein the background is transmitted in an auxiliary data channel defined by one of H.264, H.265, VP8, and VP9. 22. The method according to claim 13, wherein the background model is a parametric model. 23. The method of claim 22, wherein the parameter model is a Gaussian mixture (MOG). 24. The method of claim 13, wherein the background model is a nonparametric model. 25. The method of claim 24, wherein the nonparametric model is a visual background extractor (ViB). 26. A method for simulating gimbal zoom operation on a video of a scene, comprising: initializing a background model by determining a static background of the scene from the video; transmitting the background of the scene as the background model by encoding the background model independently of the video, wherein the background model is incrementally updated, wherein the updates are further encoded and transmitted independently of the video, wherein the incremental updates of the background model are transmitted via an auxiliary data channel, and wherein a geometric mapping between the background and the video is transmitted for each frame; selecting one or more fields of view of the video by scene analysis; and generating an enhanced video at a receiver by merging the background with the video. 27. The method of claim 26, wherein the simulated gimbal zoom operation is controlled at the receiver. 28. The method of claim 26, wherein the simulated gimbal zoom operation is controlled at the transmitter of the video. 29. A system for transmitting video of a scene, comprising: i) a transmitter including an external encoder and a core encoder, wherein the external encoder is adapted to receive the video and output salient video and incrementally updated background and geometry bitstreams to the core encoder, wherein the incremental updates of the background are transmitted via an auxiliary data channel, and wherein the core encoder is adapted to output an encoded bitstream; and ii) a receiver including a core decoder, wherein the incremental updates of the background are received via the auxiliary data channel, and wherein the core decoder is adapted to receive the encoded bitstream and output the salient video. 30. A system for transmitting video of a scene, comprising: i) a transmitter including an external encoder and a core encoder, wherein the external encoder is adapted to receive the video and output salient video and incrementally updated background and geometry bitstreams respectively to the core encoder, wherein the incremental updates of the background are transmitted via an auxiliary data channel, and wherein the core encoder is adapted to output an encoded bitstream; and ii) a receiver including a core decoder and an external decoder, wherein the core decoder is adapted to receive the encoded bitstream and output the salient video and the incrementally updated background and geometry bitstreams respectively to the external decoder, wherein the incremental updates of the background are received via an auxiliary data channel, and wherein the external decoder is adapted to merge the salient video and the incrementally updated background and geometry bitstreams to output an enhanced video of the scene. 31. The system of claim 30, wherein the external encoder includes a background estimation unit adapted to initialize a background model by determining a static background of the scene from the video, and to incrementally update the background model at a bit rate lower than that of the video. 32. The system of claim 31, wherein the external encoder further comprises a background encoder connected to the background estimation unit, the background encoder being adapted to encode the background model and the update independently of the video. 33. The system of claim 32, wherein the background encoder comprises an entropy encoder, an entropy decoder, an update prediction unit, and an update storage unit. 34. The system of claim 33, wherein the background encoder is connected to the bitstream multiplexer in the downstream direction. 35. The system of claim 34, wherein the external encoder further comprises a saliency framing unit adapted to output a geometric bitstream to the bitstream multiplexer, wherein the bitstream multiplexer is adapted to merge the geometric bitstream and the background bitstream to output a background and geometric bitstream. 36. The system of claim 35, wherein the external encoder further includes a reduction unit capable of scaling and cropping the video, the reduction unit being connected downstream to a noise suppression unit adapted to suppress noise in the significant video based on the background model. 37. The system of claim 36, wherein the external decoder further comprises: i) a bitstream demultiplexer adapted to receive the background and geometry bitstreams from the core encoder and output the geometry bitstream and the background bitstream respectively; ii) a background decoder connected to the bitstream demultiplexer and adapted to receive the background bitstream; and iii) a background merging unit connected downstream to the bitstream demultiplexer and the background decoder, wherein the background merging unit is adapted to receive the salient video from the core decoder and merge the geometry bitstream and the background bitstream with the salient video to produce an enhanced video of the scene. 38. The system of claim 37, wherein the background decoder comprises an entropy decoder, an update prediction unit, and an update storage unit. 39. The system of claim 37, wherein the external decoder further includes a virtual gimbal zoom unit capable of receiving control input to generate enhanced video. 40. The system of claim 37, wherein the core encoder is an H.264/H.265 video encoder, and wherein the background and geometric bitstreams are carried through the network abstraction layer of the core encoder. 41. The system of claim 37, wherein the core decoder is an H.264/H.265 video decoder, wherein the background and geometry bitstreams are carried through the network abstraction layer of the core decoder. 42. The system of claim 37, wherein the core encoder is in a multimedia container format, and wherein the background and geometric bitstreams are carried through an auxiliary data channel of the core encoder. 43. The system of claim 37, wherein the core decoder is in a multimedia container format, and wherein the background and geometry bitstreams are carried through an auxiliary data channel of the core decoder. 44. The system of claim 37, wherein the core encoder is a standard video encoder, and wherein the background and geometry bitstreams are carried through an auxiliary data channel of the core encoder. 45. The system of claim 37, wherein the core decoder is a standard video decoder, and wherein the background and geometry bitstreams are carried through an auxiliary data channel of the core decoder.