WO2020048485A1 - Vr全景摄影系统 - Google Patents

Vr全景摄影系统 Download PDF

Info

Publication number
WO2020048485A1
WO2020048485A1 PCT/CN2019/104389 CN2019104389W WO2020048485A1 WO 2020048485 A1 WO2020048485 A1 WO 2020048485A1 CN 2019104389 W CN2019104389 W CN 2019104389W WO 2020048485 A1 WO2020048485 A1 WO 2020048485A1
Authority
WO
WIPO (PCT)
Prior art keywords
view
field
image
wide
module
Prior art date
Application number
PCT/CN2019/104389
Other languages
English (en)
French (fr)
Inventor
方璐
戴琼海
朱天奕
Original Assignee
清华-伯克利深圳学院筹备办公室
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 清华-伯克利深圳学院筹备办公室 filed Critical 清华-伯克利深圳学院筹备办公室
Publication of WO2020048485A1 publication Critical patent/WO2020048485A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B37/00—Panoramic or wide-screen photography; Photographing extended surfaces, e.g. for surveying; Photographing internal surfaces, e.g. of pipe
    • G03B37/04—Panoramic or wide-screen photography; Photographing extended surfaces, e.g. for surveying; Photographing internal surfaces, e.g. of pipe with cameras or projectors providing touching or overlapping fields of view
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Definitions

  • the embodiments of the present application relate to the field of image technology, for example, to a VR panoramic photography system.
  • VR Virtual Reality
  • video acquisition and image capture have made tremendous progress and development in both scale and performance.
  • On the spatial scale, its scalability is still greatly limited by the video imaging quality of cameras in related technologies.
  • On the time scale its imaging delay and corresponding imaging frame rate are still Limited by the frame rate and real-time performance of the imaging system in the related art. Therefore, in the field of high-definition surveillance or in systems such as circular panoramic imaging photography such as VR, how to effectively improve the corresponding resolution of real-time video and the corresponding display refresh frame rate has become a major problem that must be solved now.
  • the embodiment of the present application provides a multi-scale unstructured billion-pixel VR panoramic photography system based on a hybrid camera array, so as to realize the collection and display of a billion-pixel VR panoramic video.
  • the embodiment of the present application provides a multi-scale unstructured multi-scale unstructured gigapixel VR panoramic photography system based on a hybrid camera array, including: a wide field of view ring camera array, a narrow field of view ring camera array, an image processing device, and a multi-camera image Stitching equipment, VR display equipment; the output of the wide field of view camera array and the output of the narrow field of view camera array are connected to the input of the image processing device, and the output of the image processing device is connected to the input of the multi-camera image stitching device The end of the multi-camera image stitching device is connected to the input of the VR display device; the feedback of the VR display device is connected to the input of the image processing device;
  • the wide field of view ring camera array collects multiple wide field of view images
  • the narrow field of view ring camera array collects multiple narrow field of view images
  • the image processing device performs compression preprocessing on multiple wide-view images and multiple narrow-view images based on the preview perspective feedback information fed back by the VR display device, and decodes and decompresses the compressed pre-processed images in parallel after acquisition
  • Multi-camera image stitching equipment uses zero-mean normalized cross-correlation (ZNCC) algorithm to decode decoded multiple wide-view images and multiple narrow-view images according to the calibration parameter matrix.
  • ZNCC zero-mean normalized cross-correlation
  • VR display devices use displacement mapping to render and display billion-pixel-level ring images, and generate preview perspective feedback information based on the user's current preview perspective.
  • FIG. 1 is a schematic structural diagram of a multi-scale unstructured billion-pixel VR panoramic photography system based on a hybrid camera array according to an embodiment of the present application;
  • FIG. 2 is a schematic structural diagram of a multilayer ring array camera according to an embodiment of the present application.
  • FIG. 3 is a schematic structural diagram of another multi-scale unstructured gigapixel VR panoramic photography system based on a hybrid camera array according to an embodiment of the present application;
  • 4a is a left-eye mapping diagram of a VR perspective of a user in an embodiment of the present application
  • FIG. 4b is a VR perspective right-eye mapping diagram of a user in an embodiment of the present application.
  • FIG. 4c is a partial high-resolution video screenshot at A in FIG. 4a.
  • FIG. 1 is a schematic structural diagram of a multi-scale unstructured billion-pixel VR panoramic photography system based on a hybrid camera array according to an embodiment of the present application. This embodiment is applicable to the case of VR panoramic photography, as shown in FIG. 1
  • the system includes a wide-field-of-view annular camera array 110, a narrow-field-of-view annular camera array 120, an image processing device 130, a multi-camera image stitching device 140, and a VR display device 150.
  • the output of the wide field of view camera array 110 and the output of the narrow field of view camera array 120 are both connected to the input of the image processing device 130, and the output of the image processing device 130 is connected to the input of the multi-camera image stitching device 140.
  • the output terminal of the multi-camera image stitching device 140 is connected to the input terminal of the VR display device 150, and the feedback terminal of the VR display device 150 is connected to the input terminal of the image processing device 130.
  • the wide-field-of-view annular camera array 110 collects a plurality of wide-field-of-view images
  • the narrow-field-of-view annular camera array 120 collects a plurality of narrow-field-of-view images.
  • the image processing device 120 performs compression preprocessing on the multiple wide-view images and multiple narrow-view images based on the preview perspective feedback information fed back by the VR display device 150, and performs parallel decoding and decompression processing on the compressed and preprocessed images.
  • the multi-camera image stitching device 140 uses the ZNCC algorithm to perform multi-scale fusion of the decoded and decompressed multiple wide-view images and multiple narrow-view images according to the calibration parameter matrix to generate a billion-pixel ring image.
  • the VR display device 150 renders and displays a billion-pixel ring image by using a displacement map, and generates preview perspective feedback information according to the current preview perspective of the user.
  • the wide-field-of-view ring camera array 110 may be a 360-degree ring-shaped camera array composed of a plurality of wide-field-of-view cameras, and the wide-field of view camera may be a wide-field of view and low-resolution short-focus camera.
  • the narrow-field-of-view annular camera array 120 may be a 360-degree annular camera array composed of a plurality of narrow-field-of-view cameras, and the narrow-field of view camera may be a narrow-field of view high-resolution telephoto.
  • FIG. 2 is a schematic structural diagram of a multilayer ring array camera provided in an embodiment of the present application. As shown in FIG.
  • the circular array disk there are three layers of circular array disks, and the three layers of circular array disks are mounted on a central support cylinder.
  • a plurality of wide-field cameras are installed on the lowest layer circular array disk, and a plurality of narrow-field cameras are respectively installed on the uppermost layer and the middle layer circular array disk.
  • the circular array disk should be as small as possible to reduce the difference in perspective between the cameras.
  • the multi-scale unstructured billion-pixel VR panoramic photography system based on the hybrid camera array provided by the embodiment of the present application can realize the collection and display of the billion-pixel VR panoramic video.
  • FIG. 3 is a schematic structural diagram of another multi-scale unstructured billion-pixel VR panoramic photography system based on a hybrid camera array according to an embodiment of the present application.
  • an image The processing device 130 includes a compression pre-processing module 131 based on a preview perspective, a high-speed parallel acquisition module 132 and an image decoding and decompression module 133.
  • the multi-camera image stitching device 140 includes a camera modeling module 141 and an image stitching module 142.
  • the VR display device 150 includes a VR rendering module 151, a VR display 152, an interaction module 153, and a VR preview perspective feedback module 154.
  • the output of the wide field of view camera array 110 and the output of the narrow field of view camera array 120 are both connected to the input of the compression preprocessing module 131 based on the preview angle, and the output of the compression preprocessing module 130 based on the preview angle.
  • the terminal is connected to the input of the high-speed parallel acquisition module 132, and the output of the high-speed parallel acquisition module 132 is connected to the image decoding and decompression module 133.
  • the preview perspective-based compression pre-processing module 131 receives preview perspective feedback information sent by the VR display device 150, and determines the compression ratios of multiple wide-view images and multiple narrow-view images according to the preview perspective feedback information. Multiple wide-view images and multiple narrow-view images are compressed.
  • the high-speed parallel acquisition module 132 performs multi-channel parallel acquisition on the compressed image.
  • the image decoding and decompression module 133 performs adaptive decoding and decompression on the compressed multiple wide-view images and multiple narrow-view images.
  • the preview pre-processing-based compression pre-processing module 131 obtains a wide field of view image and / or a narrow field of view image in the current field of view according to the feedback information of the preview field of view, and performs low compression on the wide field of view image and / or the narrow field of view image in the current field of view.
  • Rate compression which compresses a wide field of view image and a narrow field of view image outside the current field of view with a high compression rate.
  • the preview perspective feedback information may include the user ’s preview orientation.
  • the compression pre-processing module 131 based on the preview perspective estimates the different compression rates of the acquired wide-view image and narrow-view image based on the preview perspective feedback information.
  • the compression ratios of the FOV image and the narrow FOV image are correspondingly compressed.
  • the compression pre-processing module 131 based on the preview viewing angle performs high compression on the image in the compressed area. Compression, to save network bandwidth as much as possible under the condition of ensuring the sharpness of the image in the field of view, thereby increasing the transmission frame rate and reducing the transmission delay; while when the preview viewing angle in the VR display device is narrow, only a few numbers are concentrated.
  • the compression pre-processing module 131 based on the preview viewing angle compresses the image in the field of view with a low compression rate, and compresses the image outside the field of view with a high compression rate to save network bandwidth.
  • the high-speed parallel acquisition module 132 relies on a precursor or a 10-Gigabit network group to summarize and distribute image data.
  • a star-shaped network structure of “multi-channel acquisition and one-channel display” is adopted, and compressed images are collected and processed by multiple nodes and then concentrated to a central node for further processing and corresponding display.
  • the image decoding and decompressing module 133 may be a dynamic and adaptive image decoding and decompressing module, which decodes and decompresses the images compressed by the compression preprocessing module 131 based on the preview view at different compression rates.
  • the input end of the camera modeling module 141 is connected to the output end of the wide field of view camera array 110 and the output end of the narrow field of view camera array 120 respectively, and the input end of the image stitching module 142 is decompressed and decoded to the image.
  • the output of the module 133 is connected.
  • the camera modeling module 141 obtains in advance a frame of image acquired by each wide-view camera in the wide-view field annular camera array as an advance calibration frame, and uses a set calibration algorithm to calibrate and model the wide-view image according to the advance calibration frame.
  • a calibration parameter matrix is obtained for each wide-field camera.
  • the image stitching module 142 uses the ZNCC algorithm to perform multi-scale fusion of the decoded and decompressed multiple wide-view images and multiple narrow-view images according to the calibration parameter matrix of each wide-view camera to generate a billion-pixel ring image.
  • the camera modeling module 141 records the calibration parameter matrix of each wide-view camera for subsequent multi-scale image fusion.
  • multiple narrow-field-of-view images need to be re-projected onto a circular low-resolution image stitched by multiple wide-field-of-view images.
  • a frame of image is collected from each camera in advance as a calibration frame in advance, and the corresponding wide field image is calibrated and modeled accordingly to form the corresponding calibration parameter matrix.
  • the ZNCC algorithm is used to multi-scale fusion of multiple wide-view images and multiple narrow-view images after decoding and decompression according to the calibration parameter matrix of each wide-view camera to generate a billion-pixel ring image. And record the calculated calibration parameter matrix, which can be used for the subsequent links of real-time stitching and display.
  • the input of the VR rendering module 151 is connected to the output of the image stitching module 142, the output of the VR rendering module 151 is connected to the input of the VR display 152, and the output of the VR preview perspective feedback module 154 is based on The input end of the compression pre-processing module 131 of the preview angle is connected.
  • the VR rendering module 151 is based on the NVIDIA graphics card ’s Unified Computing Device Architecture (CUDA) computing library and Open Graphics Library (OpenGL) image rendering library. It uses displacement mapping to perform billion-pixel ring images. Render.
  • the VR display 152 displays the rendered billion-pixel ring image.
  • the interaction module 153 receives a user's operation on the VR display device to operate a billion-pixel-level ring image.
  • the VR preview perspective feedback module 154 obtains the line of sight defense of the user, and obtains the preview perspective information according to the user's line of sight.
  • the interaction module 153 may be provided in the controller.
  • the user can perform corresponding operations such as zooming in and out of the perspective of the panorama through the controller.
  • the direction observed by the user can be obtained, so that the user's area of interest for the currently rendered ball is calculated.
  • This area contains the user's zoom level, related camera images, and other related information. parameter.
  • FIG. 4a is a left-view mapping diagram of a user's VR perspective provided by an embodiment of the present application
  • FIG. 4a is a right-view mapping diagram of a user's VR perspective provided by an embodiment of the present application
  • FIG. 4c is a local high-resolution video at A in FIG. 4a Screenshot.
  • the user can zoom in on the area and view a high-resolution real-time preview of the area through the VR display device.

Abstract

本申请实施例公开了一种VR全景摄影系统，包括：宽视场环形相机阵列、窄视场环形相机阵列、图像处理设备、多相机图像拼接设备、VR显示设备；宽视场环形相机阵列采集多个宽视场图像，窄视场环形相机阵列采集多个窄视场图像；图像处理设备基于VR显示设备反馈的预览视角反馈信息对多个宽视场图像和多个窄视场图像进行压缩预处理，对压缩预处理后的图像并行采集后进行解码解压处理；多相机图像拼接设备采用ZNCC算法按照标定参数矩阵将解码减压后的多个宽视场图像和多个窄视场图像进行多尺度融合，生成十亿像素级环形图像。

Description

VR全景摄影系统
本申请要求在2018年09月05日提交中国专利局、申请号为201811032604.9的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及图像技术领域，例如涉及一种VR全景摄影系统。
背景技术
随着虚拟现实(Virtual Reality，VR)技术的发展和相关相机产业的发展，在计算机和计算机摄像学领域，视频采集与图像捕获无论在规模上还是在性能上都取得了巨大的进步和发展。如今，VR视频仍旧受限于两大方面，在空间尺度上，其可扩展性仍旧大大受限于相关技术中相机的视频成像质量，在时间尺度上，其成像延迟与相应的成像帧率依旧受限于相关技术中的成像系统的帧率与系统的实时性。因此，在高清监控领域或是在VR等环形全景成像摄影等系统中，如何有效的提高实时视频的相应分辨率，提高相应的显示刷新帧率，则成为了当下势必需要解决的主要问题。
在相关技术中的实时视频采集系统中，由于受数据流量、计算复杂度等相关的限制，常常无法实现十亿像素级的并行实时输出与处理，另一方面，超大的视场与重视细节的应用之间也常常有非常大的矛盾，因此当下的实时的视频采集常常难以兼顾宽视场的总体视频流信息与窄视场的视频细节信息。另外，对于此类超宽视场在现场显示方面，传统的显示屏显示方式也有着非常大的局限，其不仅对于用户的体验非常不友好，也常常无法以合理的方式显示360度的全局视频信息。
发明内容
以下是对本文详细描述的主题的概述。本概述并非是为了限制权利要求的保护范围。
本申请实施例提供一种基于混合相机阵列的多尺度非结构化的十亿像素VR全景摄影系统，以实现十亿像素VR全景视频的采集与显示。
本申请实施例提供了一种基于混合相机阵列的多尺度非结构化的十亿像素VR全景摄影系统，包括：宽视场环形相机阵列、窄视场环形相机阵列、图像处理设备、多相机图像拼接设备、VR显示设备；宽视场环形相机阵列的输出端与窄视场环形相机阵列的输出端均与图像处理设备的输入端相连，图像处理设备的输出端与多相机图像拼接设备的输入端相连，多相机图像拼接设备的输出端与VR显示设备的输入端相连；VR显示设备的反馈端与图像处理设备的输入端相连；
宽视场环形相机阵列采集多个宽视场图像，窄视场环形相机阵列采集多个窄视场图像；
图像处理设备基于VR显示设备反馈的预览视角反馈信息对多个宽视场图像和多个窄视场图像进行压缩预处理，对压缩预处理后的图像并行采集后进行解码解压处理；
多相机图像拼接设备采用零均值归一化互相关(Zero-mean Normalized Cross-Correlation，ZNCC)算法按照标定参数矩阵将解码减压后的多个宽视场图像和多个窄视场图像进行多尺度融合，生成十亿像素级环形图像；
VR显示设备采用置换贴图的方式对十亿像素级环形图像进行渲染并显示，并根据用户当前的预览视角生成预览视角反馈信息。
在阅读并理解了附图和详细描述后，可以明白其他方面。
附图说明
图1是本申请实施例提供的一种基于混合相机阵列的多尺度非结构化的十亿像素VR全景摄影系统的结构示意图；
图2是本申请实施例提供的一种多层环形阵列相机的结构示意图；
图3是本申请实施例提供的另一种基于混合相机阵列的多尺度非结构化的十亿像素VR全景摄影系统的结构示意图；
图4a是本申请实施例中用户的VR视角左眼映射图；
图4b是本申请实施例中用户的VR视角右眼映射图；
图4c是图4a中A处的局部高分辨的视频截图。
具体实施方式
下面结合附图和实施例对本申请作进一步的详细说明。可以理解的是，此 处所描述的具体实施例仅仅用于解释本申请，而非对本申请的限定。另外还需要说明的是，为了便于描述，附图中仅示出了与本申请相关的部分而非全部结构。
图1为本申请实施例提供的一种基于混合相机阵列的多尺度非结构化的十亿像素VR全景摄影系统的结构示意图，本实施例可适用于VR全景摄影的情况，如图1所示，该系统包括：宽视场环形相机阵列110、窄视场环形相机阵列120、图像处理设备130、多相机图像拼接设备140、VR显示设备150。宽视场环形相机阵列110的输出端与窄视场环形相机阵列120的输出端均与图像处理设备130的输入端相连，图像处理设备130的输出端与多相机图像拼接设备140的输入端相连，多相机图像拼接设备140的输出端与VR显示设备150的输入端相连，VR显示设备150的反馈端与图像处理设备130的输入端相连。
宽视场环形相机阵列110采集多个宽视场图像，窄视场环形相机阵列120采集多个窄视场图像。图像处理设备120基于VR显示设备150反馈的预览视角反馈信息对多个宽视场图像和多个窄视场图像进行压缩预处理，对压缩预处理后的图像并行采集后进行解码解压处理。多相机图像拼接设备140采用ZNCC算法按照标定参数矩阵将解码减压后的多个宽视场图像和多个窄视场图像进行多尺度融合，生成十亿像素级环形图像。VR显示设备150采用置换贴图的方式对十亿像素级环形图像进行渲染并显示，并根据用户当前的预览视角生成预览视角反馈信息。
其中，宽视场环形相机阵列110可以是由多个宽视场相机组成的360度环形相机阵列，宽视场相机可以是宽视场低分辨短焦相机。窄视场环形相机阵列120可以是由多个窄视场相机组成的360度环形相机阵列，窄视场相机可以是窄视场高分辨长焦。本实施例中，多个宽视场相机和多个窄视场相机分别设置于多层环形阵列盘上形成宽视场环形相机阵列110、窄视场环形相机阵列120，多个宽视场相机以均匀排布的方式设置于多层环形阵列盘上，多个窄视场相机以均匀排布或者不均匀排布的方式设置于多层环形阵列盘上。示例性的，图2为本申请实施例提供的一种多层环形阵列相机的结构示意图，如图2所示，有3层环形阵列盘，3层环形阵列盘安装于中心支撑圆柱上，在最下一层环形阵列盘上安装多个宽视场相机，最上一层和中间一层环形阵列盘上分别安装多个窄视场相机。本应用场景下，环形阵列盘应该尽可能的小，以减少相机之间的视角差异。
本申请实施例提供的基于混合相机阵列的多尺度非结构化的十亿像素VR全景摄影系统可以实现十亿像素VR全景视频的采集与显示。
图3为本申请实施例提供的另一种基于混合相机阵列的多尺度非结构化的十亿像素VR全景摄影系统的结构示意图，作为对上述实施例的进一步解释，如图2所示，图像处理设备130包括基于预览视角的压缩预处理模块131、高速并行采集模块132和图像解码解压模块133。多相机图像拼接设备140包括相机建模模块141和图像拼接模块142。VR显示设备150包括VR渲染模块151、VR显示器152、交互模块153和VR预览视角反馈模块154。
其中，宽视场环形相机阵列110的输出端与窄视场环形相机阵列120的输出端均与基于预览视角的压缩预处理模块131的输入端相连，基于预览视角的压缩预处理模块130的输出端与高速并行采集模块132的输入端相连，高速并行采集模块132的输出端与图像解码解压模块133相连。
基于预览视角的压缩预处理模块131接收VR显示设备150发送的预览视角反馈信息，并根据预览视角反馈信息确定多个宽视场图像和多个窄视场图像的压缩率，并根据压缩率对多个宽视场图像和多个窄视场图像进行压缩。高速并行采集模块132对压缩后图像进行多路并行采集。图像解码解压模块133对压缩后的多个宽视场图像和多个窄视场图像进行自适应的解码解压。
基于预览视角的压缩预处理模块131根据预览视角反馈信息获取当前视野内的宽视场图像和/或窄视场图像，对当前视野内的宽视场图像和/或窄视场图像进行低压缩率的压缩，对当前视野外的宽视场图像和窄视场图像进行高压缩率的压缩。具体的，预览视角反馈信息可以包括用户的预览方位，基于预览视角的压缩预处理模块131根据预览视角反馈信息推算出采集的宽视场图像和窄视场图像的不同的压缩率，根据各个宽视场图像和窄视场图像的压缩率进行相应的压缩。示例性的，当VR显示设备150中的预览视角较宽时，视野中的图像以较低的分辨率显示显示，基于预览视角的压缩预处理模块131对压缩区域内的图像进行高压缩率的压缩，在保证视野内图像清晰度的条件下尽可能的节约网络带宽，从而提高传输帧率，减少传输延迟；而当VR显示设备中的预览视角较窄时，仅仅集中在某几个数量较少的图像上时，基于预览视角的压缩预处理模块131对视野内的图像进行低压缩率的压缩，而对于视野外的图像则进行进行高压缩率的压缩，以节约网络带宽。
高速并行采集模块132依赖于前兆或者万兆网络组进行图像数据的汇总与 分发。本实施例中，采取“多路采集，一路显示”的星型网络结构，由多个节点采集处理压缩后的图像然后集中到中央节点进行下一步处理和相应显示等流程。
图像解码解压模块133可以是动态自适应的图像解码解压模块，对由基于预览视角的压缩预处理模块131以不同的压缩率压缩后的图像进行解码解压。
在一实施例中，相机建模模块141的输入端分别与宽视场环形相机阵列110的输出端与窄视场环形相机阵列120的输出端相连，图像拼接模块142的输入端与图像解码解压模块133的输出端相连。
相机建模模块141预先获取宽视场环形相机阵列中每个宽视场相机采集的一帧图像作为提前标定帧，根据提前标定帧采用设定标定算法对宽视场图像进行标定及建模，获得每个宽视场相机的标定参数矩阵。图像拼接模块142采用ZNCC算法按照每个宽视场相机的标定参数矩阵将解码减压后的多个宽视场图像和多个窄视场图像进行多尺度融合，生成十亿像素级环形图像。相机建模模块141将每个宽视场相机的标定参数矩阵进行记录，用于后续的图像多尺度融合。
本实施例中，为了最终合成多尺度的十亿像素级的视频流，需要将多个窄视场图像，重新投影至由多个宽视场图像拼接的环形低分辨图像上。采用设定标定，预先从每个相机中采集一帧图像作为提前标定帧，对宽视场图像进行相应的标定及建模，形成相应的标定参数矩阵。然后再通过ZNCC算法按照每个宽视场相机的标定参数矩阵将解码减压后的多个宽视场图像和多个窄视场图像进行多尺度融合，生成十亿像素级环形图像。并记录下计算得到的标定参数矩阵，可用于后续的实时拼接与显示的相应环节。
在一实施例中，VR渲染模块151的输入端与图像拼接模块142的输出端相连，VR渲染模块151的输出端与VR显示器152的输入端相连，VR预览视角反馈模块154的输出端与基于预览视角的压缩预处理模块131输入端相连。
VR渲染模块151基于NVIDIA显卡的统一计算设备架构(Compute Unified Device Architecture，CUDA)计算库与开放图形库(Open Graphics Library，OpenGL)图像渲染库，采用置换贴图的方式对十亿像素级环形图像进行渲染。VR显示器152将渲染后的十亿像素级环形图像进行显示。交互模块153接收用户对VR显示设备的操作，以对十亿像素级环形图像进行操作。VR预览视角反馈模块154获取用户的视线防线，根据用户的视线获得预览视角信息。
本实施例中，交互模块153可以设置于控制器中，为了方便用户与VR显示设备进行交互等环节，用户可通过控制器对全景图进行视角的拉近拉远，轴移动等相应操作。根据用户在VR显示设备中的移动与查看，可以获取到用户观察到的方向，从而计算出用户对于当前渲染球的兴趣区域，该区域包含了用户的缩放程度、涉及到的相机图像等相关的参数。
图4a为本申请实施例提供的用户的VR视角左眼映射图，图4a为本申请实施例提供的用户的VR视角右眼映射图，图4c是图4a中A处的局部高分辨的视频截图。用户可通过VR显示设备放大该区域并查看该区域的高清实时预览图。

Claims (7)

  1. 一种虚拟现实VR全景摄影系统，包括：宽视场环形相机阵列、窄视场环形相机阵列、图像处理设备、多相机图像拼接设备、VR显示设备；所述宽视场环形相机阵列的输出端与所述窄视场环形相机阵列的输出端均与所述图像处理设备的输入端相连，所述图像处理设备的输出端与所述多相机图像拼接设备的输入端相连，所述多相机图像拼接设备的输出端与所述VR显示设备的输入端相连；所述VR显示设备的反馈端与所述图像处理设备的输入端相连；
    所述宽视场环形相机阵列采集多个宽视场图像，所述窄视场环形相机阵列采集多个窄视场图像；
    所述图像处理设备基于所述VR显示设备反馈的预览视角反馈信息对所述多个宽视场图像和所述多个窄视场图像进行压缩预处理，对压缩预处理后的图像并行采集后进行解码解压处理；
    所述多相机图像拼接设备采用ZNCC算法按照标定参数矩阵将解码减压后的多个宽视场图像和多个窄视场图像进行多尺度融合，生成十亿像素级环形图像；
    所述VR显示设备采用置换贴图的方式对所述十亿像素级环形图像进行渲染并显示，并根据用户当前的预览视角生成预览视角反馈信息。
  2. 根据权利要求1所述的系统，其中，多个宽视场相机和多个窄视场相机分别设置于多层环形阵列盘上，形成宽视场环形相机阵列、窄视场环形相机阵列；
    所述多个宽视场相机以均匀排布的方式设置于多层环形阵列盘上；所述多个窄视场相机以均匀排布或者不均匀排布的方式设置于多层环形阵列盘上。
  3. 根据权利要求1所述的系统，其中，所述图像处理设备包括基于预览视角的压缩预处理模块、高速并行采集模块和图像解码解压模块；所述宽视场环形相机阵列的输出端与所述窄视场环形相机阵列的输出端均与所述基于预览视角的压缩预处理模块的输入端相连，所述基于预览视角的压缩预处理模块的输出端与所述高速并行采集模块的输入端相连，所述高速并行采集模块的输出端与所述图像解码解压模块的输入端相连；
    所述基于预览视角的压缩预处理模块接收所述VR显示设备发送的预览视角反馈信息，并根据所述预览视角反馈信息确定所述多个宽视场图像和所述多个窄视场图像的压缩率，并根据所述压缩率对所述多个宽视场图像和所述多个窄视场图像进行压缩；
    所述高速并行采集模块对压缩后的图像进行多路并行采集；
    所述图像解码解压模块对压缩后的多个宽视场图像和所述多个窄视场图像进行自适应的解码解压。
  4. 根据权利要求3所述的系统，其中，所述基于预览视角的压缩预处理模块根据预览视角反馈信息获取当前视野内的宽视场图像和窄视场图像中的至少一种，对当前视野内的宽视场图像和窄视场图像中的至少一种进行低压缩率的压缩，对当前视野外的宽视场图像和窄视场图像进行高压缩率的压缩。
  5. 根据权利要求3所述的系统，其中，所述多相机图像拼接设备包括相机建模模块和图像拼接模块；
    所述相机建模模块的输入端分别与所述宽视场环形相机阵列的输出端与所述窄视场环形相机阵列的输出端相连，所述图像拼接模块的输入端与所述图像解码解压模块的输出端相连；
    所述相机建模模块预先获取宽视场环形相机阵列中每个宽视场相机采集的一帧图像作为提前标定帧，根据所述提前标定帧采用设定标定算法对宽视场图像进行标定及建模，获得每个宽视场相机的标定参数矩阵；
    所述图像拼接模块采用ZNCC算法按照每个宽视场相机的标定参数矩阵将解码减压后的多个宽视场图像和多个窄视场图像进行多尺度融合，生成十亿像素级环形图像。
  6. 根据权利要求5所述的系统，其中，所述相机建模模块将每个宽视场相机的标定参数矩阵进行记录，用于后续的图像多尺度融合。
  7. 根据权利要求5所述的系统，其中，所述VR显示设备包括VR渲染模块、VR显示器、交互模块和VR预览视角反馈模块；所述VR渲染模块的输入端与所述图像拼接模块的输出端相连，所述VR渲染模块的输出端与所述VR显示器的输入端相连，所述VR预览视角反馈模块的输出端与所述基于预览视角的压缩预处理模块输入端相连；
    所述VR渲染模块基于NVIDIA显卡的CUDA计算库与OpenGL图像渲染库，采用置换贴图的方式对所述十亿像素级环形图像进行渲染；
    所述VR显示器将渲染后的十亿像素级环形图像进行显示；
    所述交互模块接收用户对VR显示设备的操作，以对十亿像素级环形图像进行操作；
    所述VR预览视角反馈模块获取用户的视线防线，根据用户的视线获得预览视角信息。
PCT/CN2019/104389 2018-09-05 2019-09-04 Vr全景摄影系统 WO2020048485A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811032604.9A CN109040601B (zh) 2018-09-05 2018-09-05 一种多尺度非结构化的十亿像素vr全景摄影系统
CN201811032604.9 2018-09-05

Publications (1)

Publication Number Publication Date
WO2020048485A1 true WO2020048485A1 (zh) 2020-03-12

Family

ID=64623599

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/104389 WO2020048485A1 (zh) 2018-09-05 2019-09-04 Vr全景摄影系统

Country Status (2)

Country Link
CN (1) CN109040601B (zh)
WO (1) WO2020048485A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112311985A (zh) * 2020-10-12 2021-02-02 珠海格力电器股份有限公司 一种多摄拍照处理方法、装置及存储介质

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109040601B (zh) * 2018-09-05 2020-06-26 清华-伯克利深圳学院筹备办公室 一种多尺度非结构化的十亿像素vr全景摄影系统
CN110634190A (zh) * 2019-09-24 2019-12-31 赵岩 一种远程摄像vr体验系统
CN111343367B (zh) * 2020-02-17 2021-06-08 清华大学深圳国际研究生院 一种十亿像素虚拟现实视频采集装置、系统与方法
CN112468832A (zh) * 2020-10-22 2021-03-09 北京拙河科技有限公司 一种十亿级像素全景视频直播方法、装置、介质及设备
CN114866747B (zh) * 2022-07-07 2022-09-30 国网瑞嘉(天津)智能机器人有限公司 高空作业装置及作业图像的处理方法
CN116471490B (zh) * 2023-06-19 2023-08-29 清华大学 变光照十亿像素光场智能成像系统、方法与装置

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080143821A1 (en) * 2006-12-16 2008-06-19 Hung Yi-Ping Image Processing System For Integrating Multi-Resolution Images
CN101651841A (zh) * 2008-08-13 2010-02-17 华为技术有限公司 一种立体视频通讯的实现方法、系统和设备
CN103152518A (zh) * 2011-12-01 2013-06-12 索尼公司 图像处理系统和方法
CN106165415A (zh) * 2014-04-07 2016-11-23 诺基亚技术有限公司 立体观看
CN107659774A (zh) * 2017-09-30 2018-02-02 深圳市未来媒体技术研究院 一种基于多尺度相机阵列的视频成像系统及视频处理方法
CN108139799A (zh) * 2016-04-22 2018-06-08 深圳市大疆创新科技有限公司 基于用户的兴趣区(roi)处理图像数据的系统和方法
CN108399600A (zh) * 2018-02-23 2018-08-14 清华-伯克利深圳学院筹备办公室 一种全景成像系统及方法
CN109040601A (zh) * 2018-09-05 2018-12-18 清华-伯克利深圳学院筹备办公室 一种多尺度非结构化的十亿像素vr全景摄影系统

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102821238B (zh) * 2012-03-19 2015-07-22 北京泰邦天地科技有限公司 宽视场超高分辨率成像系统
CN103281484A (zh) * 2013-06-17 2013-09-04 清华大学 十亿像素视频采集装置
US9811946B1 (en) * 2016-05-30 2017-11-07 Hong Kong Applied Science and Technology Research Institute Company, Limited High resolution (HR) panorama generation without ghosting artifacts using multiple HR images mapped to a low resolution 360-degree image
US10482574B2 (en) * 2016-07-06 2019-11-19 Gopro, Inc. Systems and methods for multi-resolution image stitching
CN106303497A (zh) * 2016-08-12 2017-01-04 南方科技大学 一种虚拟现实内容生成方法和装置
CN107197166A (zh) * 2017-08-01 2017-09-22 哈尔滨市舍科技有限公司 一种双分辨率一体式全景摄像装置及方法
CN107396077B (zh) * 2017-08-23 2022-04-08 深圳看到科技有限公司 虚拟现实全景视频流投影方法和设备
CN107529064A (zh) * 2017-09-04 2017-12-29 北京理工大学 一种基于vr终端反馈的自适应编码方法
CN108322727A (zh) * 2018-02-28 2018-07-24 北京搜狐新媒体信息技术有限公司 一种全景视频传输方法及装置

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080143821A1 (en) * 2006-12-16 2008-06-19 Hung Yi-Ping Image Processing System For Integrating Multi-Resolution Images
CN101651841A (zh) * 2008-08-13 2010-02-17 华为技术有限公司 一种立体视频通讯的实现方法、系统和设备
CN103152518A (zh) * 2011-12-01 2013-06-12 索尼公司 图像处理系统和方法
CN106165415A (zh) * 2014-04-07 2016-11-23 诺基亚技术有限公司 立体观看
CN108139799A (zh) * 2016-04-22 2018-06-08 深圳市大疆创新科技有限公司 基于用户的兴趣区(roi)处理图像数据的系统和方法
CN107659774A (zh) * 2017-09-30 2018-02-02 深圳市未来媒体技术研究院 一种基于多尺度相机阵列的视频成像系统及视频处理方法
CN108399600A (zh) * 2018-02-23 2018-08-14 清华-伯克利深圳学院筹备办公室 一种全景成像系统及方法
CN109040601A (zh) * 2018-09-05 2018-12-18 清华-伯克利深圳学院筹备办公室 一种多尺度非结构化的十亿像素vr全景摄影系统

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112311985A (zh) * 2020-10-12 2021-02-02 珠海格力电器股份有限公司 一种多摄拍照处理方法、装置及存储介质

Also Published As

Publication number Publication date
CN109040601B (zh) 2020-06-26
CN109040601A (zh) 2018-12-18

Similar Documents

Publication Publication Date Title
WO2020048485A1 (zh) Vr全景摄影系统
US11528468B2 (en) System and method for creating a navigable, three-dimensional virtual reality environment having ultra-wide field of view
US10349068B1 (en) Apparatus and method for panoramic video hosting with reduced bandwidth streaming
JP5172863B2 (ja) バナナ・コーデック
Yang et al. A real-time distributed light field camera.
JP6044328B2 (ja) 画像処理システム、画像処理方法およびプログラム
US8723951B2 (en) Interactive wide-angle video server
EP3804349B1 (en) Adaptive panoramic video streaming using composite pictures
US20040027451A1 (en) Immersive imaging system
WO2019161814A2 (zh) 一种全景成像系统及方法
JP5743016B2 (ja) 画像を生成する装置および方法
TW201143430A (en) System and method for capturing and displaying cinema quality panoramic images
US10572764B1 (en) Adaptive stereo rendering to reduce motion sickness
CN102801963A (zh) 基于高清数字摄像头监控的电子ptz方法及装置
US11270413B2 (en) Playback apparatus and method, and generation apparatus and method
WO2007060497A2 (en) Interactive wide-angle video server
EP3190566A1 (en) Spherical virtual reality camera
CN113205599B (zh) 一种视频三维融合时gpu加速的视频纹理更新方法
EP3695593A1 (en) Creating multi-camera panoramic projections
US20100302403A1 (en) Generating Images With Different Fields Of View
JP6394682B2 (ja) 方法および画像処理装置
CN202818497U (zh) 基于高清数字摄像头监控的电子ptz装置
CN110533577A (zh) 鱼眼图像校正方法及装置
Chi et al. RA360SR: A real-time acceleration-adaptive 360-degree video super-resolution system
US20230222754A1 (en) Interactive video playback techniques to enable high fidelity magnification

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19856609

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19856609

Country of ref document: EP

Kind code of ref document: A1