WO2018068481A1

WO2018068481A1 - Binocular 720-degree panoramic acquisition system

Info

Publication number: WO2018068481A1
Application number: PCT/CN2017/079091
Authority: WO
Inventors: 王超; 沈靖程; 王士博; 刘亚辉; 张睿妍
Original assignee: 深圳市圆周率软件科技有限责任公司
Priority date: 2016-10-12
Filing date: 2017-03-31
Publication date: 2018-04-19
Also published as: CN106993177A

Abstract

Disclosed is a binocular 720-degree panoramic acquisition system, comprising: an acquisition module (1) used for acquiring N paths of audio data and twin paths of video data; a core processing module (2) used for matching the N paths of audio data and the two paths of video data and stitching same into a panoramic video; a cloud streaming media server module (3) and a local server module (5), which are used for establishing a live broadcast, generating a stream pushing address and a broadcast address, and performing format conversion on the received audio and video stream format; and a wide area network terminal experience module (4) and a local area network terminal experience module (5), which are used for completing the audio and video on-site live broadcast. By means of two ISP with high processing capability and at least two MIPI interfaces, hardware synchronous acquisition of a twin paths of video, real-time processing of a data volume of two sensors of more than ten million pixels and real-time stitching of a 720-degree panoramic video can be realized, thereby guaranteeing that high-definition image information can be quickly transferred under a relatively low bit-rate network.

Description

A binocular 720 degree panoramic gather system

Technical field

[0001] The present invention belongs to the field of image acquisition technology, and in particular relates to a binocular 720 degree panoramic acquisition system.

Background technique

[0002] With the rapid development of computer technology, multimedia has more and more types, and more and more performances can be expressed. Some of the more traditional expressions are increasingly unable to satisfy most customers. The requirements of the presentation method. In the traditional way of expression, the means of display are nothing more than static flat pictures and dynamic video, but also through three-dimensional panoramic display. Static pictures can only provide a certain angle image of the scene. Even a wide-angle lens cannot be effective and comprehensive. The performance of the scene; while the dynamic video allows the user to have a comprehensive understanding of the scene, but the image viewing angle is still limited, the way of viewing depends on the photographer's shooting method, not free, so we need to be true, comprehensive and intuitive Showing a certain scene, the 360 panorama is undoubtedly the best choice.

[0003] With the continuous improvement of people's needs, people hope to build a virtual panoramic environment with continuous roaming, information richness and interactivity. How to quickly and effectively construct the image information needed by virtual reality and deliver it to customers. And display, to enhance the realism and comfort of the user experience, will become an urgent problem to be solved.

technical problem

[0004] In summary, in order to solve the above technical problem, the present invention provides a binocular 720-degree panoramic acquisition system to quickly and efficiently construct image information required for virtual reality, thereby improving the realism and comfort of the user experience. .

Problem solution

Technical solution

[0005] A binocular 720-degree panoramic acquisition system provided by the present invention includes:

[0006] an acquisition module, configured to collect N channels of audio data and two channels of video data, where N is a positive integer greater than or equal to 2;

[0007] a core processing module, configured to perform dual-channel video data collected by the collection module for panoramic video Simultaneously, the N channels of audio data are fused by the surround sound algorithm to the N channels of audio data, and the position information of the algorithm is matched with the panoramic video, so that the surround sound audio can be matched according to different perspective positions of the panoramic video. The occurrence of the sound source position felt by the human ear in the real scene, and the hardware acceleration encoding of the video data and the hardware acceleration encoding of the audio data by the matched panoramic video and the surround sound audio, and then performing a time stamp synchronization, and finally The synchronized audio data and video data are stored locally or directly in the RTMP format for audio and video streaming, which is connected to the collection module;

[0008] a cloud streaming server module, configured to receive audio and video streams that are sent by the core processing module through an Ethernet network, create a live broadcast on the cloud platform, generate a push stream address and a play address, and receive the received audio and video. The format of the stream is formatted to distribute the processed video and audio data, which is connected to the core processing module;

[0009] a WAN terminal experience module, configured to receive and decode audio and video data distributed by the cloud streaming server module, and perform a live live experience of a remote immersive panoramic audio and video scene, which is associated with the cloud streaming server module Connection

[0010] a local server module, configured to receive audio and video streams that are sent by the core processing module through the Ethernet, and create a live broadcast, generate a push stream address, and a play address on the local server, and receive the received audio and video streams. The format is formatted to distribute the processed video and audio data, and is connected to the core processing module;

[0011] The local area network terminal experience module is configured to receive and decode audio and video data distributed by the local server module, and complete a live live experience of the local immersive panoramic audio and video scene, which is connected to the local server module.

[0012] Further, the acquisition module includes a first module of the camera module and a second unit of the camera module electrically connected to each other, and the first unit of the camera module includes an image first sensor and the first sensor of the image The first fisheye lens is electrically connected, the second unit of the camera module includes an image second sensor and a second fisheye lens electrically connected to the second sensor of the image, and the collecting module further comprises a first unit for collecting sound And pick up the second unit.

[0013] Further, an optical axis of the first unit of the camera module and an optical axis of the second unit of the camera module are coincident with each other or parallel to each other. [0014] Further, the core processing module includes a CPU management unit, an ISP first unit respectively connected to the CPU management unit, an ISP second unit, a multi-channel mixing unit, a memory unit, a GPU unit, and a panorama. a splicing unit and an audio encoding unit, the core processing module further including MIPI

a first interface, a MIPI second interface, a video encoding unit, a multiplexing unit, an RTMP push stream unit, and a local storage unit, where the video encoding unit, the multiplexing unit, and the RTMP push stream unit are sequentially connected and the video encoding unit and the The panoramic tiling unit is connected, the local storage unit is connected to the multiplexing unit, the multiplexing unit is further connected to the audio coding unit, and the multi-channel mixing unit and the collection module are The first unit of the sound pickup is connected to the second unit of sound pickup, the first unit of the ISP is connected to the first interface of the MIPI, and the second unit of the ISP is connected to the second interface of the MIPI. The MIPI first interface is connected to the image first sensor in the acquisition module, and the MIPI second interface is connected to the image second sensor in the acquisition module.

[0015] Further, the cloud streaming server module is connected to the RTMP push unit in the core processing module.

[0016] Further, the WAN terminal experience module includes a first demultiplexing unit, a first video decoding unit, a first display unit connected to the first video decoding unit, a first audio decoding unit, and a a first playback unit connected to the first audio decoding unit, the first demultiplexing unit is connected to the first video decoding unit and the first audio decoding unit, and the cloud streaming server module and the first A demultiplexing unit is connected.

[0017] Further, the WAN terminal experience module may be a VR

All-in-one, mobile, tablet, MAC, laptop or desktop computer.

[0018] Further, the local server module is connected to an RTMP push flow unit in the core processing module.

[0019] Further, the local area network terminal experience module includes a second demultiplexing unit, a second video decoding unit, a second display unit connected to the second video decoding unit, a second audio decoding unit, and a a second playback unit connected to the second audio decoding unit, the second demultiplexing unit is connected to the second video decoding unit and the second audio decoding unit, the local server module and the second solution The multiplexing units are connected.

[0020] Further, the local area network terminal experience module may be a VR All-in-one, mobile, tablet, MAC, laptop or desktop computer. Advantageous effects of the invention

Beneficial effect

[0021] Compared with the prior art, the binocular 720-degree panoramic acquisition system provided by the present invention can realize dual-channel video hardware synchronous acquisition, two-way million by two high-processing ISPs and at least two MIPI interfaces. The actual processing of the data volume of the sensor above the pixel and the actual splicing of the 720-degree panoramic video ensure that the high-speed image information can be transmitted quickly at a lower bit rate of the network, greatly improving the realism of the user experience. And comfort, peers, the system's small size, low power consumption and low cost make it more in line with consumer demand.

Brief description of the drawing

DRAWINGS

1 is a schematic structural diagram of a binocular 720-degree panoramic acquisition system according to an embodiment of the present invention.

Embodiments of the invention

The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0024] The implementation of the present invention is described in detail below in conjunction with the specific embodiments.

Referring to FIG. 1, FIG. 1 is a schematic structural diagram of a binocular 720-degree panoramic acquisition system according to an embodiment of the present invention. As shown in FIG. 1, a binocular 720-degree panoramic acquisition system includes:

[0026] The acquisition module 1 is configured to collect N channels of audio data and two channels of video data, where N is a positive integer greater than or equal to 2;

[0027] The core processing module 2 is configured to perform the panoramic video real-time splicing of the two-way video data collected by the acquisition module 1, and simultaneously combine the N-channel audio sampling data with the N-channel audio data by using a surround sound algorithm, and the panorama The video performs the position information matching of the algorithm, so that the surround sound audio can simulate the occurrence of the sound source position perceived by the human ear according to the different perspective position matching of the panoramic video, further enhancing the shocking feeling of the immersive experience of the experiencer. Matching panoramic video and surround sound The audio performs hardware accelerated encoding of video data and hardware accelerated encoding of audio data, and then performs a time-stamp synchronization, and finally stores the synchronized audio data and video data locally or directly performs audio and video streaming in RTMP format through Ethernet. , which is connected to the acquisition module 1;

[0028] The cloud streaming server module 3 is configured to receive audio and video streams that the core processing module 2 pushes through the Ethernet, create a live broadcast on the cloud platform, generate a push stream address and a play address, and receive the received audio and video. The format of the stream is formatted to distribute the processed video and audio data, which is connected to the core processing module 2;

[0029] The WAN terminal experience module 4 is configured to receive and decode the audio and video data distributed by the cloud streaming media server module 3, and perform a live live experience of the remote immersive panoramic audio and video scene, and the cloud streaming media server module 3 Connected

[0030] The local server module 5 is configured to receive audio and video streams that the core processing module 2 pushes through the Ethernet and create a live broadcast, generate a push stream address, and a play address on the local server, and receive the received audio and video streams. After the format is formatted, the processed video and audio data are distributed, and the core processing module 2 is connected;

[0031] The local area network terminal experience module 6 is configured to receive and decode the audio and video data distributed by the local server module 5, and complete the live live experience of the local immersive panoramic audio and video scene, which is connected to the local server module 5.

[0032] In the embodiment of the present invention, the acquisition module 1 includes a camera module first unit 10 and a camera module second unit 11 that are electrically connected to each other, and the camera module first unit 10 includes an image first sensor 101 and an image The first fisheye lens 102 electrically connected to the sensor 101, the camera module second unit 11 includes an image second sensor 111 and a second fisheye lens 112 electrically connected to the second image sensor 111. The acquisition module 1 further includes a pickup. The first unit 113 and the second unit 114 are picked up. The image first sensor 101 and the image second sensor 111 reach a pixel level of at least 13M, and the angles of the first fisheye lens 102 and the second fisheye lens 112 are at least 190 degrees and above, and the light of the first unit 10 of the camera module The optical axes of the shaft and camera module second unit 11 are coincident with each other or parallel to each other.

[0033] The core processing module 2 includes a CPU management unit 201, an ISP first unit 202, an ISP second unit 203, a multi-channel mixing unit 204, and a memory unit respectively connected to the CPU management unit 201.

205. The GPU unit 206, the panoramic splicing unit 207, and the audio encoding unit 208, the core processing module 2 The MIPI first interface 209, the MIPI second interface 210, the video encoding unit 211, the multiplexing unit 212, the RTMP push stream unit 213, and the local storage unit 214, the video encoding unit 211, the multiplexing unit 212, and the RTMP push unit 213 are further included. Connected in sequence and the video encoding unit 211 is connected to the panoramic splicing unit 207, the local storage unit 214 is connected to the multiplexing unit 212, and the multiplexing unit 212 is also connected to the audio encoding unit 208, the multi-channel mixing unit 204 and the acquisition module. The first sound pickup unit 113 and the second sound pickup unit 114 are connected to each other, the ISP first unit 202 is connected to the MIPI first interface 209, and the ISP second unit 203 is connected to the MIPI second interface 210, MIPI first The interface 209 is connected to the image first sensor 101 in the acquisition module 1, and the MIPI second interface 210 is connected to the image second sensor 111 in the acquisition module 1.

[0034] The cloud streaming server module 3 is connected to the RTMP push unit 213 in the core processing module 2.

[0035] The WAN terminal experience module 4 includes a first demultiplexing unit 401, a first video decoding unit 402, a first display unit 403 connected to the first video decoding unit 402, a first audio decoding unit 404, and the first The first playback unit 405 is connected to the audio decoding unit 404, the first demultiplexing unit 401 is connected to the first video decoding unit 402 and the first audio decoding unit 404, and the cloud streaming media server module 3 and the first demultiplexing unit are connected. 401 is connected.

[0036] The local server module 5 is connected to the RTMP push flow unit in the core processing module 2.

[0037] The local area network terminal experience module 6 includes a second demultiplexing unit 601, a second video decoding unit 602, a second display unit 603 connected to the second video decoding unit 602, a second audio decoding unit 604, and The second audio decoding unit 604 is connected to the second playback unit 605, and the second demultiplexing unit 601 is connected to the second video decoding unit 602 and the second audio decoding unit 604, the local server module 5 is connected to the second demultiplexing unit 601.

[0038] In the embodiment of the present invention, the camera unit first unit 10 and the camera module second unit 11 can collect image/video data of 720 degrees in the entire space without dead angles and pass the MIPI of the core processing module 2 respectively. An interface 209 and an MIPI second interface 210 transmit image/video data to the core processing module 2, and the core processing module 2 receives the first interface 209 and MIPI from the MIPI through the ISP first unit 202 and the ISP second unit 203, respectively. The image/video data transmitted by the second interface 210 is subjected to noise reduction processing, and the GPU unit 206 is scheduled by the CPU management unit 201 in the core processing module 2 to perform splicing of the panoramic video and hardware acceleration processing of the video encoding unit 211. . Acquisition module 1 The first unit 113 of the pickup and the second single of the pickup

[0039] The audio data collected by the element 114 is fused by the multi-channel mixing unit 204, and the audio encoding unit 208 is scheduled by the CPU management unit 201 to perform AAC encoding of the audio, and finally with the panorama processed by the video encoding unit 211. After the video is synchronized, it is stored in the local storage unit 214 that is included in the panoramic camera or directly pushes the RTK stream of 4K/30fps through the Ethernet to the cloud streaming server module 3/local server module 5.

[0040] The cloud streaming server module 3/local server module 5 can perform protocol conversion, and convert the received video stream format into various video formats such as HTTP, HLS, RTP, RTSP, RTCP, RTMP, PNM, MMS, Onvif, and the like. The same video format is distributed to the WAN terminal experience module 4/LAN terminal experience module 6 capable of accepting the live broadcast of the corresponding audio and video format, and the CDN acceleration is also performed during the live broadcast of the panoramic audio and video.

[0041] The WAN terminal experience module 4/LAN terminal experience module 6 can be a VR-body machine, a mobile phone, a tablet computer, a laptop computer or a desktop computer, and the user can decode the cloud stream through the corresponding player on different experience devices. The video and audio data distributed by the media server module 3/local server module 5 achieves the effect of remote immersive panoramic audio and video live broadcast, and the WAN terminal experience module 4/LAN terminal experience module 6 can support N personal peers to watch online.

[0042] The binocular 720-degree panoramic acquisition system of the present invention passes two high-processing ISPs and at least two channels

The MIPI interface can realize hardware synchronous acquisition of dual-channel video, real-time processing of data of two-way multi-pixel sensors, and real-time splicing of 720-degree panoramic video, ensuring that the network can also be at a lower bit rate. The rapid transmission of high-definition image information greatly enhances the realism and comfort of the user experience. At the same time, the system's small size, low power consumption and low cost make it more in line with consumer needs.

The above is only the preferred embodiment of the present invention, and is not intended to limit the present invention. Any modifications, equivalents, and improvements made within the spirit and scope of the present invention should be included in the present invention. Within the scope of protection of the invention.

Claims

Claim

[Claim 1] A binocular 720-degree panoramic image acquisition system, comprising:

An acquisition module, configured to collect N channels of audio data and two channels of video data, where N is a positive integer greater than or equal to 2;

The core processing module is configured to perform the panoramic video splicing of the two-way video data collected by the collection module, and simultaneously combine the N-channel audio data with the N-channel audio data by using a surround sound algorithm, and The video performs the position information matching of the algorithm, so that the surround sound audio can simulate the occurrence of the sound source position perceived by the human ear according to the different viewing angle positions of the panoramic video, and then the matched panoramic video and the surround sound audio are used for video. The hardware accelerated encoding of the data and the hardware accelerated encoding of the audio data, and then a synchronization of the inter-turn stamp, and finally the synchronized audio data and video data are stored locally or directly in the RTMP format for audio and video streaming, and The collection modules are connected;

The cloud streaming server module is configured to receive audio and video streams that are sent by the core processing module through the Ethernet, and create a live broadcast, generate a push stream address, and a play address on the cloud platform, and format the received audio and video streams. Distributing the processed video and audio data after format conversion, which is connected to the core processing module;

The WAN terminal experience module is configured to receive and decode audio and video data distributed by the cloud streaming server module, and perform live live experience of the remote immersive panoramic audio and video scene, which is connected to the cloud streaming server module;

a local server module, configured to receive audio and video streams that are sent by the core processing module through the Ethernet, create a live broadcast on the local server, generate a push stream address and a play address, and format the received audio and video streams. Transmitting and processing the completed video and audio data, which is connected to the core processing module;

The LAN terminal experience module is configured to receive and decode audio and video data distributed by the local server module, and complete a live live experience of the local immersive panoramic audio and video scene, which is connected to the local server module.

[Claim 2] The binocular 720-degree panoramic acquisition system according to claim 1, wherein The set module includes a first module of the camera module electrically connected to each other and a second unit of the camera module, the first unit of the camera module includes an image first sensor and a first fisheye lens electrically connected to the image first sensor The camera module second unit includes an image second sensor and a second fisheye lens electrically connected to the image second sensor, and the collection module further includes a sound pickup first unit and a sound pickup second unit.

[Claim 3] The binocular 720-degree panoramic acquisition system according to claim 2, wherein an optical axis of the first unit of the camera module and an optical axis of the second unit of the camera module overlap each other or Parallel to each other.

[Claim 4] The binocular 720-degree panoramic acquisition system according to claim 1, wherein the core processing module includes a CPU management unit, an ISP first unit respectively connected to the CPU management unit, and an ISP. a second unit, a multi-channel mixing unit, a memory unit, a GPU unit, a panoramic splicing unit, and an audio encoding unit, the core processing module further includes an MIPI first interface, a MIPI second interface, a video encoding unit, a multiplexing unit, An RTMP push unit and a local storage unit, the video encoding unit, the multiplexing unit, and the RTMP push unit are sequentially connected, and the video encoding unit is connected to the panoramic splicing unit, the local storage unit and the multiplexing The unit is connected, the multiplexing unit is further connected to the audio encoding unit, and the multi-channel mixing unit is connected to the first unit of the sound pickup and the second unit of the sound pickup in the collecting module. The first unit of the ISP is connected to the first interface of the MIPI, and the second unit of the ISP is connected to the second interface of the MIPI, the MIPI Interface and the image acquisition module is connected to a first sensor, said second interface MIPI connected with a second image sensor module of the acquisition.

[Claim 5] The binocular 720-degree panoramic acquisition system of claim 1, wherein the cloud streaming server module is connected to an RTMP push unit in the core processing module.

[Claim 6] The binocular 720-degree panoramic acquisition system of claim 1, wherein the WAN terminal experience module comprises a first demultiplexing unit, a first video decoding unit, and the first video a first display unit connected to the decoding unit, a first audio decoding unit, a first playback unit connected to the first audio decoding unit, the first demultiplexing unit and The first video decoding unit is connected to the first audio decoding unit, and the cloud streaming server module is connected to the first demultiplexing unit.

[Claim 7] The binocular 720-degree panoramic acquisition system according to claim 1 or 6, wherein the WAN terminal experience module can be a VR-body machine, a mobile phone, a tablet computer, a MAC computer, a laptop computer or Desktop computer.

[Claim 8] The binocular 720-degree panoramic acquisition system according to claim 1, wherein the local server module is connected to an RTMP push flow unit in the core processing module.

[Claim 9] The binocular 720-degree panoramic acquisition system of claim 1, wherein the local area network terminal experience module comprises a second demultiplexing unit, a second video decoding unit, and the second video a second display unit, a second audio decoding unit, and a second playback unit connected to the second audio decoding unit, the second demultiplexing unit and the second video decoding unit The two audio decoding units are connected, and the local server module is connected to the second demultiplexing unit.

[Claim 10] The binocular 720-degree panoramic acquisition system according to claim 1 or 9, wherein the local area network terminal experience module can be a VR-computer, a mobile phone, a tablet computer, a MAC computer, a laptop computer or Desktop computer.