WO2019113935A1

WO2019113935A1 - Closed wearable panoramic image capturing and processing system and operating method therefor

Info

Publication number: WO2019113935A1
Application number: PCT/CN2017/116448
Authority: WO
Inventors: 李昂
Original assignee: 李昂
Priority date: 2017-12-15
Filing date: 2017-12-15
Publication date: 2019-06-20

Abstract

The present invention relates to a closed wearable panoramic image capturing and processing system and an operating method therefor. The system comprises: a camera component; a built-in processor, the built-in processor comprising a video image processor, a computing processor, a coding processor, built-in and plug-in storage media; a built-in gravity sensing, acceleration sensing, compass and gyroscope chip, used for detecting movements when photographing and when worn, thus performing image stabilization processing with respect to a photographed content, determining the direction of photographing and/or used for other human-machine interaction purposes; a virtual reality display, a near-eye display or a regular liquid crystal display for receiving and displaying a processed video signal transmitted by the built-in processor; and a built-in power supply for supplying electricity to the components of the wearable panoramic image capturing and processing system. The wearable panoramic image capturing and processing system of the present invention has a compact form factor and powerful functions, thus being convenient to be worn by a user.

Description

Closed wearable panoramic camera and processing system and operating method thereof

Technical field

The present invention relates to the technical field of virtual reality technology, and more particularly to a closed wearable panoramic camera and processing system and method of operating the same.

Background technique

Virtual reality technology is a technology that allows users to get an immersive experience by completely or almost completely covering the video information of the eyes of the person, plus the virtual perspective that changes as the wearer's head moves. In order to capture the real environment that users feel, many companies use multiple cameras with different angles to capture images and then stitch together several image streams into a stereo 360° image, allowing virtual reality viewers to turn around to see the same Images from different angles are achieved to achieve an immersive effect.

The Infineon invention patent application with the publication number of CN 105072393 A discloses a multi-lens panoramic network camera and a splicing method, but the invention uses the invention in the field of monitoring, the device is not wearable, and the output streaming media is not for the virtual reality binocular. Video, but a composite panoramic image (for display on the monitor screen).

Currently, other virtual reality cameras LucidCam and motion camera GoPro have only two lenses and are fisheye/angle 150-180 degree ultra wide-angle lenses. The thickness of this lens is much thicker than the camera lens of the mobile phone. It will be very cumbersome.

The images taken by these existing products will have some unnatural distortions, without considering the portability of the lens and the wearability of the device. The multi-lens surveillance camera is fixed in one position, lacking built-in battery and storage function.

Summary of the invention

1. Technical problem to be solved by the present invention

It is an object of the present invention to provide a closed wearable panoramic imaging and processing system and method of operating the same, and a primary object of the present invention is to reduce the size of a virtual reality camera/panoramic video capture and processing device to make it more convenient. Carry/wear. Furthermore, by using a plurality of undistorted high-definition small cameras arranged at different angles, the overall video capturing angle is at or above the field of view of the human eyes, eliminating the unnatural distortion caused by the shooting of the existing products. At the same time, using the built-in processor and motion sensing, combined with low-distortion overlapping images captured by multi-lens, graphics recognition, depth of field analysis, visual scene modeling, video image digital image stabilization, video image orientation and others Analysis and processing of machine interaction. The built-in or modular additional display device allows the user to view the recorded video image while viewing and using the product and visually enjoy its built-in processing capabilities.

2. The complete technical solution provided by the invention

According to an aspect of the present invention, a closed wearable panoramic imaging and processing system is provided, comprising:

a camera assembly comprising two or more undistorted cameras, each undistorted camera simultaneously capturing video images at a consistent frame rate;

A built-in processor includes a video processor, a computing processor, an encoding processor, a built-in and/or an external storage medium, and the video processor uses a video stitching algorithm to capture the same time point from each undistorted camera. The video frames of each video stream are merged and spliced into a larger field of view of the undistorted video, and the computing processor or the encoding processor encodes the spliced video, wherein the computing processor can analyze the video signal that has not been processed by the video processor. Processing to meet the needs of graphic recognition, depth of field analysis, and visual scene modeling;

Built-in gravity sensing, acceleration sensing, compass and gyroscope chip for detecting the action during shooting and wearing, to anti-shake the shooting content, determine the shooting direction (such as landscape, vertical) and / or for others Machine interaction use;

a virtual reality display, a near-eye display, or a general liquid crystal display, receiving and processing a processed video signal transmitted from a built-in processor;

The built-in power supply supplies power to the components of the enclosed wearable panoramic camera system to ensure proper system operation.

In the above solution of the present invention, two or more undistorted cameras are staggered and arranged by the built-in battery of the device; the camera is assembled to form an image, and two or more video streams are used to pass through the built-in processor of the device, and are merged by a video stitching algorithm. Undistorted video into a larger field of view; built-in code processor encodes the processed video in H264 or other formats; the finished encoded video is saved to internal storage and/or plug-in storage media, attached or modularized by device The display device is played either in real time or delayed by wire (eg HDMI/usb type-c/DP) or wireless (eg wifi/cellular mobile network adapter/Bluetooth) to the outside.

According to another aspect of the present invention, a method for operating a closed wearable panoramic camera is provided, the method comprising: a method for operating a closed wearable panoramic camera system, comprising:

After the closed wearable panoramic camera system is activated, each camera simultaneously captures images at a consistent frame rate;

Two or more video streams from each camera pass through a built-in video processor, and use a dynamic video stitching algorithm to splicing video frames from several video streams taken at the same time point into an undistorted video frame with a larger field of view;

A built-in encoding processor or computing processor encodes the stitched video;

The built-in encoding processor or computing processor performs secondary processing on the output format to adjust a fixed or active field of view, virtual lay length, and virtual orientation viewed by the virtual reality device;

The finished encoded video is saved to internal storage or plug-in storage media, and/or exported to the Internet, playback devices, external storage devices, and other computer/storage devices in real time via wire or wireless.

3. The beneficial effects brought by the technical solution of the present invention

The wide-angle video stitched by multiple distortion-free lens camera components has no unnatural distortion with respect to the video taken by the fisheye lens (GoPro, LucidCam) with FOV of 150°-180°, which makes the virtual reality user feel more Naturally, closer to the human eye, and easier to post-process;

An undistorted lens with multiple FOVs below 130° (usually at 90° to 120°, but smaller) is lighter and thinner than the fisheye lens described above (a typical fixed-focus, undistorted lens has a thickness of less than 4 mm) The fisheye lens of a general sports camera is as high as 20 to 30 mm, so that the wearable camera can be made smaller and more convenient for the user to wear.

The video is captured using two or more undistorted lenses arranged at different angles; the video is processed by a built-in processor to fit the virtual reality display, and the entire system is a mobile, wearable device.

Undistorted video and images are not only easier to process and analyze, but capturing overlapping images from multiple angles can also aid in pattern recognition, depth of field analysis, and visual scene modeling.

Compared with the previous products, only the video image is recorded, and the product can perform the above analysis and processing on the captured content by using the built-in processor.

Compared to previous products, this product can view the captured content using its own or modular additional virtual reality display, near-eye display or general LCD display. The real-time view of the captured image (commonly known as Live View) can also realize the superposition of digital information on the image content (commonly known as Augmented Reality).

Reference 1: Application No.: 201510466737.7, Patent Name: A multi-lens panoramic network camera and splicing method

DRAWINGS

The present invention includes the accompanying drawings to facilitate a further understanding of the technical solutions of the present invention.

1a-1c are front, top and side views, respectively, of an exemplary arrangement of two cameras of a wearable panoramic camera system in accordance with a preferred embodiment of the present invention;

2a-2c are front, top and side views, respectively, of an exemplary arrangement of three cameras of a wearable panoramic camera system in accordance with a preferred embodiment of the present invention;

3a-3c are front, top, and side views, respectively, of an exemplary arrangement of four cameras of a wearable panoramic camera system in accordance with a preferred embodiment of the present invention;

4a-4c are front, top, and side views, respectively, of an exemplary arrangement of five cameras of a wearable panoramic camera system in accordance with a preferred embodiment of the present invention;

5a-5c are front, top, and side views, respectively, of an exemplary arrangement of six cameras of a wearable panoramic camera system in accordance with a preferred embodiment of the present invention;

6a-6c are front, top, and side views, respectively, of an exemplary arrangement of seven cameras of a wearable panoramic camera system in accordance with a preferred embodiment of the present invention;

7a-7c are front, top, and side views, respectively, of an exemplary arrangement of eight cameras of a wearable panoramic camera system in accordance with a preferred embodiment of the present invention;

8 is a flow chart of a method of operation of a wearable panoramic camera system in accordance with a preferred embodiment of the present invention.

Reference numerals in the drawings

1 camera

Detailed ways

The preferred embodiments of the present invention will be described in more detail below, but it should be noted that the preferred embodiments are only intended to assist those skilled in the art to understand the present invention. The scope of the present invention is not limited to the preferred embodiments below, but rather The claims are limited. In Figures 1a-7c, the camera 1 is labeled in Figures 1a-2c, similar to the other figures, and will not be repeated, but those skilled in the art will appreciate that the convex cylindrical structure of the other figures is For the camera.

First, a closed wearable panoramic camera system according to an embodiment of the present invention is described. The closed wearable panoramic camera system includes:

a camera assembly comprising two or more undistorted cameras 1 , preferably the two or more undistorted cameras are arranged at different angles to capture video, which may be arranged in a staggered manner, and the undistorted camera may be a conventional camera assembly, an array camera or a photon counting type light field camera, each camera of the camera assembly preferably simultaneously captures video images at a uniform frame rate;

Built-in processor, the built-in processor may include a video processor, a computing processor, an encoding processor, a built-in and/or an external storage medium, etc., and correspondingly process 2 or more video streams captured by each camera To adapt to a virtual reality display, a near-eye display, or a general liquid crystal display and to enable video signal transmission, preferably, the video processor uses two or more undistorted cameras from the same time through a video stitching algorithm such as a dynamic video stitching algorithm. The video frames of each video stream of the point shot are merged and merged into an undistorted video of a larger field of view, thereby achieving the effect of capturing a larger field of view video, and the computing processor or the encoding processor encodes the stitched video in H.264 or other formats. The other format includes a binocular stereo format or other possible format required by the virtual reality playing device, and the computing processor or the encoding processor may perform secondary processing on the format of the output video to adjust the viewing on the virtual reality display. Fixed or active field of view, virtual lay length, virtual orientation, etc., and later, will be compiled The coded video is saved to the built-in and/or external storage medium, or played by the device itself or by a modular additional display device, wherein the computing processor can also analyze and process the video signal that has not been processed by the video processor. To meet various purposes such as graphic recognition, depth of field analysis, visual scene modeling, etc.

Built-in gravity sensing, acceleration sensing, compass and gyroscope chip, used to detect the action during shooting and wearing, to anti-shake the shooting content, determine the shooting direction (such as horizontal, vertical, etc.) or for other human-machines Interactive use;

The built-in power supply supplies power to the components of the wearable panoramic camera system to ensure proper system operation.

Among them, the undistorted camera is an undistorted lens with a field of view (FOV) of 130° or less, which is lighter and thinner than a fisheye lens.

Preferably, the built-in processor of the present invention further comprises a data transmission interface, which can realize real-time or delay through wired (such as HDMI/usb type-c/DP) or wireless (wifi/cellular mobile network adapter/Bluetooth, etc.). The video signal processed by the built-in processor is then exported to an external device such as a playback device, an external storage device, other computer/storage device, or exported to the Internet in real time or delayed.

The built-in power supply further includes a charging interface, and the built-in power source may include a rechargeable battery through which the built-in power source can be charged.

Of course, the closed wearable panoramic camera system of the present invention can also output information captured by two or more undistorted cameras 1 to the Internet, playback devices, external storage devices, and the like in an unprocessed format. Computer/storage device for processing, analysis, and viewing.

The arrangement of the two or more undistorted cameras of the camera assembly can be set according to actual conditions to cover a wider field of view than a single lens.

In a preferred embodiment, the enclosed wearable panoramic camera system of the present invention may further comprise an activation device, which may be a fuselage switch. Of course, the closed wearable panoramic camera system of the present invention can also be activated by remote control.

Optionally, the camera assembly of the closed wearable panoramic camera system of the present invention comprises two or more undistorted cameras, wherein one or two cameras are located directly in front of the closed wearable panoramic camera system. One or two cameras use a fisheye camera. The other cameras in the camera unit use a distortion-free camera to supplement the field of view or assist the shooting. Here, the supplementary field of view or auxiliary shooting refers to the shooting of overlapping images by two or more cameras to calculate Analyze graphic/spatial information such as depth of field.

In a preferred manner, the lens arrangement combination can be as follows:

The field of view covered by the human eyes is generally about 180° in the lateral direction (X-axis) and 160° in the longitudinal direction (Z-axis). Covering such a large field of view with multiple undistorted cameras requires each camera to cover a different angle. Since the clear focus field of the human eye is only about 100°, the video captured by the closed wearable panoramic camera system of the present invention does not necessarily need to reach the maximum field of view of the above-mentioned person. We will set the field of view to be covered as Ωx (transverse) and Ωz (longitudinal). The angle of view of each camera is ax1, ax2, az1, az2 and so on to axn, azn, and the number of camera components that make up the device is n. When looking down from the top of the device (Z-axis viewing angle), we hope that the angle covered by nx cameras reaches Ωx on the X-axis, so

The angle of view of each camera does not have to be the same, but if the camera used has the same field of view on the X-axis (assuming all are ax), then we can

This means that if a device has a larger number of lenses, the minimum viewing angle required for each lens is narrower; in other words, if the angle of view of a single lens is wider, the number of lenses that the device needs to cover Ω is less. These two inequalities indicate that the fields of view covered by the nx cameras can overlap, and the device coverage field angle Ωx (Ωz is the same) is obtained only if they do not overlap each other at all.

Similarly, when looking from the side of the device (X-axis viewing angle), we hope that the angle covered by the nz cameras reaches Ωz on the Z-axis, so

If the camera used has the same field of view on the Z axis (assuming az), then we can

When we get the number of nx cameras required for the X-axis and the number of nz cameras required for the Z-axis, we know the total number of cameras required for the device, n ≥ n _x × n _z .

Since the fields of view of multiple cameras in a single device may overlap, objects captured in the overlapping area of the field of view can be calculated by the relative position of the object in the images captured by the multiple cameras, combined with the distance and angle between the cameras. The distance between the device and the device. Distance data will help with object recognition, environmental modeling and more.

An exemplary arrangement in which the camera assembly includes two or more cameras in accordance with a preferred embodiment of the present invention is shown in FIGS. 1-7. Specifically, the camera assembly includes 2, 3, 4, 5, respectively. The preferred arrangement of 6, 6, and 8 cameras.

Hereinafter, the arrangement of the cameras in the present invention will be described by taking a conventional double lens as an example, and referring to FIGS. 1-7.

A typical two-lens arrangement is to arrange the two lenses laterally in front of each other, in which case the total field of view is substantially equal to the single lens field of view (assuming the two lenses are the same). The closed wearable panoramic camera system of the present invention can also arrange two lenses at an angle in addition to the general arrangement to cover more fields of view than a single lens.

An operation method of a closed wearable panoramic camera system according to a preferred embodiment of the present invention is specifically described below with reference to FIG. 8, the method comprising the steps of:

1. After the closed wearable panoramic camera system is activated via the body switch or remote control, each camera simultaneously captures images at a consistent frame rate;

2.2 or more video streams are passed through the built-in video processor, preferably using a dynamic video stitching algorithm to stitch video frames from several video streams at the same time point into a larger field of view of the undistorted video frame to achieve shooting The effect of a larger field of view video;

3. The built-in encoding processor or computing processor encodes the stitched video in H.264 or other formats, including the Stereoscopic format or other possible formats required by the virtual reality playback device;

4. The built-in encoding processor or computing processor can perform secondary processing on the output format to adjust the fixed or active field of view, virtual lay length, virtual orientation, etc. viewed by the virtual reality device;

5. The encoded video is saved to the built-in storage or plug-in storage medium, or exported to the Internet or playback device via cable (HDMI/USB Type-c, etc.) or wireless (such as Wi-Fi/cellular mobile network adapter). , external storage devices and other computer/storage devices;

6. The closed wearable panoramic camera system can also selectively output information captured by two or more camera components directly to the Internet, playback device, and external storage device in an unprocessed format via a data transmission interface. And other computer/storage devices to process, analyze, and view;

7. The video information processed by or without the second and third steps can also be analyzed and processed by the built-in computing processor to meet the purposes of graphic recognition, depth of field analysis, visual scene modeling and the like. The built-in computing processor can also combine the real-time detection or storage information of the built-in gravity sensing, acceleration sensing, compass and gyro chip according to claim 1, and perform real-time or video information with or without splicing and encoding processing. Delayed processing to achieve digital image stabilization, adjust video image orientation, and other human-computer interactions.

The invention has been described in terms of the preferred embodiments of the present invention, and it is to be understood that the details of the invention may be modified, substituted and changed. Within the scope of the appended claims.

Claims

A closed wearable panoramic camera and processing system comprising:

a camera assembly comprising two or more undistorted cameras, each undistorted camera simultaneously capturing video images at a consistent frame rate;

A built-in processor includes a video image processor, a computing processor, an encoding processor, a built-in and/or an external storage medium, and the video processor captures the same time point from each undistorted camera through a video stitching algorithm The video frames of each video stream are merged into an undistorted video of a larger field of view, and the computing processor or the encoding processor encodes the stitched video, wherein the computing processor can perform video signals that are not processed by the video processor. Analytical processing to meet the needs of graphics recognition, depth of field analysis, visual scene modeling, and gesture recognition operations;

Built-in gravity sensing, acceleration sensing, compass and gyro chip for detecting the action during shooting and wearing, to anti-shake the shooting content, determine the shooting direction and / or for other human-computer interaction purposes;

a virtual reality display, a near-eye display, or a general liquid crystal display, receiving and processing a processed video signal transmitted from a built-in processor;

The built-in power supply supplies power to the components of the enclosed wearable panoramic camera system to ensure proper system operation.
The system of claim 1 wherein the two or more undistorted cameras are arranged at different angles.
The system of claim 1 wherein the two or more undistorted cameras are arranged in a staggered manner, and wherein the undistorted camera is a conventional camera, an array camera or a photon counting type light field camera.
The system of claim 1 wherein, after encoding the stitched video, the computing processor or encoding processor is capable of reprocessing the format of the output video to adjust the fixed or active field of view of the virtual reality display. , virtual lay length, and/or virtual orientation.
The system of claim 1 wherein the encoded video is saved to a built-in and/or external storage medium after encoding the stitched video.
The system of claim 1 wherein said built-in processor further comprises a data transfer interface that enables real-time export of video signals processed by the built-in processor by wire or wirelessly.
The system of claim 1 wherein the built-in power source is a rechargeable power source and the built-in power source further comprises a charging interface.
A method of operating a closed wearable panoramic camera system, comprising:

After the closed wearable panoramic camera system is activated, each of the two or more undistorted cameras simultaneously capture images at a consistent frame rate;

Two or more video streams from each camera pass through a built-in video processor, and use a dynamic video stitching algorithm to splicing video frames from several video streams taken at the same time point into an undistorted video frame with a larger field of view;

A built-in encoding processor or computing processor encodes the stitched video;

The built-in encoding processor or computing processor performs secondary processing on the output format to adjust a fixed or active field of view, virtual lay length, and virtual orientation viewed by the virtual reality device;

The finished encoded video is saved to internal storage and/or plug-in storage media, and/or exported to the Internet, playback devices, external storage devices, and other computer/storage devices in real time via wire or wireless.
The operating method according to claim 8, wherein the information captured by the two or more undistorted cameras is wired or wirelessly output to the Internet, a playback device, an external storage device, and other computers in an unprocessed format via a data transmission interface. / Storage devices, which are processed, analyzed, and viewed.
The operating method according to claim 8, wherein the built-in computing processor analyzes and processes the video information that has undergone or is not stitched and encoded to satisfy the purposes of pattern recognition, depth of field analysis, and/or visual scene modeling, and /or

The built-in computing processor combines built-in gravity sensing, acceleration sensing, real-time detection or storage of compass and gyroscope chips, and performs real-time or post-processing of video information that has been or is not stitched or encoded to capture the content. The purpose of digital image stabilization, adjustment of video image orientation, and/or other human-computer interaction.