KR101806840B1 - High Resolution 360 degree Video Generation System using Multiple Cameras - Google Patents

Abstract

The present invention relates to a high resolution 360-degree video generating system using a plurality of cameras. The video generating system according to the present invention includes: a plurality of camera modules; a plurality of embedded boards which control the plurality of corresponding camera modules among the plurality of camera modules, respectively; and a matching image generating unit connected to the plurality of embedded boards through a network, wherein each of the plurality of embedded boards firstly matches images from the plurality of corresponding camera modules and transmits a corresponding compressed video file, and the matching image generating unit matches the compressed video files received from the plurality of embedded boards. Accordingly, the present invention can increase the resolution of the matched image.

Description

Technical Field [0001] The present invention relates to a high resolution 360 degree video generation system using multiple cameras,

In particular, the present invention relates to a moving image generating system capable of providing high-resolution 360-degree panoramic images in real time by using a plurality of cameras but halving transmission data and reducing the amount of computation.

Recent exhibitions of CP +, CES, and other various IT companies are showing VR (Virtual Reality) related technologies and interest in VR is increasing rapidly. Panoramic technologies used in fields such as CCTV and action cam as realistic image technologies that provide a wide angle of view have also been studied to produce 360-degree realistic moving images that can be displayed on VR devices.

Dedicated devices for 360-degree video shooting have begun to be commercialized, but since a small number of cameras equipped with a wide-angle lens capture images and match them, distortions occurring when using a wide-angle lens can not be avoided. In addition, a resolution of at least 4K is required for realistic VR image reproduction in FHD (Full High-Definition) and UHD (Ultra HD) displays, which are recently installed in mobile devices. On the other hand, Even if a high-performance camera is used, the production of high-resolution images is limited. There is a separate professional VR production equipment for high-resolution images, but it is expensive because it uses dedicated camera modules and components. Also, since most of these devices use post-processing software to match images after shooting, they can not be checked and matched at the same time as shooting.

In order to produce a real-time panoramic image, many improvements are needed, but the first thing to do is to optimize the matching method. Conventional image matching methods were focused on still images rather than real - time images, and they were difficult to real - time image matching because they were based on image quality rather than execution time. Some of the existing technologies have improved the speed of existing image synthesis methods by suggesting a software structure for real-time panoramic video creation, or by using GPU (Graphics Processing Unit) or pipeline structure modification. However, There is a difficulty in real-time matching of images.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to increase the resolution of a matching image by using a plurality of cameras (e.g., 14 HD-class general angle-of-view cameras) (Eg, 7), and the result is encoded, compressed, and transmitted to the server, thereby halving the amount of data to be transmitted. In the data processing apparatus of the server, So that it is possible to provide a high-resolution 360-degree panoramic image in real time.

According to an aspect of the present invention, there is provided a motion picture generation system including: a plurality of camera modules; A plurality of embedded boards each controlling a corresponding plurality of corresponding camera modules among the plurality of camera modules; And a matching image generation unit connected to the plurality of embedded boards through a network, wherein each of the plurality of embedded boards firstly matches images from the corresponding camera modules and transmits the corresponding compressed moving image file, The matching image generating unit matches the compressed moving image files received from the plurality of embedded boards.

The plurality of camera modules are arranged in a circle at equal intervals so as to be able to capture 360 degrees, and the matching image generating unit provides 360-degree panorama images of high resolution in real time.

The plurality of camera modules are fixed and the image matching in each of the plurality of embedded boards and the matching in the matching image generating unit are performed by using a homography matrix calculated in advance without matching and detecting the feature points, .

Each of the plurality of embedded boards stores compressed moving picture files in a memory for a time determined to be proportional to the matching time in the matching image generating unit, and transmits corresponding compressed moving pictures to the matching image generating unit.

According to another aspect of the present invention, there is provided a method for generating a panoramic moving picture, the method comprising: a plurality of embedded boards, each of which controls a plurality of camera modules corresponding to the plurality of camera modules, Transmitting the compressed moving picture file by first matching images; And matching the compressed moving image files received from the plurality of embedded boards in a matching image generating unit connected to the plurality of embedded boards via a network.

According to the moving image generating system according to the present invention, the resolution of the matching image is increased by using a plurality of cameras (e.g., 14 HD-class general angle-of-view cameras), and when the fixed camera is used, The computation can be simplified by omitting unnecessary steps such as detection, matching, and the like.

In addition, according to the moving picture generating system of the present invention, the two images are firstly matched in each of the embedded boards (e.g., seven), and the result is encoded, compressed, and transmitted to the server, thereby halving the amount of transmitted data, It is possible to reduce the amount of computation by homography matrix calculation, image warping, blending, and the like every frame, and to easily perform image matching in real time.

According to the motion picture generation system of the present invention, a data processing apparatus (e.g., a PC (Personal Computer)) of a server performs homography matrix calculation, image warping, blending Resolution 360-degree panoramic image in real time.

1 shows a pyramid structure of multi-band blending according to an embodiment of the present invention.
2 is a diagram for explaining the relationship between the number of photographed images and the resolution.
FIG. 3 is a diagram for explaining a 360-degree panoramic moving image generating system 100 according to an embodiment of the present invention.
4 is a flowchart illustrating an operation of each embedded board according to an embodiment of the present invention.
5 is a flowchart illustrating an operation in a matching image generating unit (e.g., a PC) according to an embodiment of the present invention.
FIG. 6 is a diagram for explaining the concept of a matching process in a matching image generating unit (for example, a PC) in the present invention.
FIG. 7 shows an example of an implementation system using camera modules and embedded boards according to the present invention.
FIG. 8 is a graph showing an FPS according to time of a matching image generated by a matching image generating unit (e.g., PC) in the system implemented as shown in FIG.
9 is an example of comparing the resolution of a matched output panoramic image with respect to a 640 * 480 input image (a) and a 1280 * 720 input image (b) in Table 2.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are denoted by the same reference symbols as possible. In addition, detailed descriptions of known functions and / or configurations are omitted. The following description will focus on the parts necessary for understanding the operation according to various embodiments, and a description of elements that may obscure the gist of the description will be omitted. Also, some of the elements of the drawings may be exaggerated, omitted, or schematically illustrated. The size of each component does not entirely reflect the actual size, and therefore the contents described herein are not limited by the relative sizes or spacings of the components drawn in the respective drawings.

First, the principle of homography, image warping, and image blending for image matching will be briefly described.

≪ Calculation of Homography >

The homography matrix H _AB may be a 3 * 3 matrix representing the transformation relationship between two planes in the projection relationship as shown in [Equation 1]. When a planar object existing on a three-dimensional space is photographed at two different positions A and B, a relationship between a coordinate X ₁ viewed from position A and a coordinate X ₂ viewed from position B Satisfies the expression (2). The homography matrix H _AB can be computed using a pair of feature point matching of two images, where at least four matching pairs are required. In the present invention, a SURF (Speeded-Up Robust Features) method is used to extract feature points and match them. However, when using a fixed camera (e.g., 14), unnecessary steps such as feature point detection and matching required for the calculation of the homography matrix H _AB may be omitted to simplify the calculation.

[Equation 1]

&Quot; (2) "

<Image Warping>

Each input image (I ₁ , I ₂ , I ₃ , ..) obtained from a plurality of cameras (for example, 14) has the same size but has a different coordinate system. Therefore, in order to generate a panoramic image, the input images must be represented by one pixel coordinate system. If one of the input images is determined, the rest of the images can be regarded as images obtained by applying the respective homography matrix H _AB . After successfully calculate the homography matrix H _AB, it is possible to apply the inverse matrix (H _BA) in H _AB in the other image to obtain a uniform image based on the coordinate system of the reference image. At this time, each image I ₁ , I ₂ , I ₃ , .. is transformed into a coordinate system of one reference image (example I ₁ ) using the homography matrix H _AB relation between neighboring positions as shown in [Equation 3] Can be converted into an image (I ₁ , I ₂ ^' , I ₃ ^' , ..) to be expressed.

&Quot; (3) &quot;

<Image Blending>

After the images are aligned through image warping, they are arranged into one image, and image blending is performed to make the overlap region around the boundary of the matching image visually appear natural. Since the images arranged in one image through the warping process are not appropriately processed for the overlap region, correction is required through the blending process. In the present invention, a multi-band blending method is used. Multiband blending methods use Gaussian and Laplacian pyramids.

1 shows a pyramid structure of multi-band blending according to an embodiment of the present invention.

For example, as shown in FIG. 1, when three bands are used, a Gaussian image G (2), G (1), and G (0) are obtained by down- D (2) and D (3) are obtained by up-sampling 2 times again G (0) with the base image, and the differences of the image values at the same image size are calculated. Images L (1), L (2), and L (3).

It is possible to obtain natural images in the overlap region around the boundary between two matched images using the weight window of each step by acquiring the Gaussian images and the Laplacian images with respect to the two input images.

&Lt; 360 degree panoramic moving picture generating system of the present invention >

In the present invention, as a system for photographing 360-degree panoramic images of high resolution, more than ten camera modules are used to obtain high-resolution results.

2 is a diagram for explaining the relationship between the number of photographed images and the resolution.

When the camera performance is the same as FHD, as compared with the case where two cameras are photographed and photographed with a camera having a wide angle (e.g., 200 degrees) lens having a large field of view (FoV) As shown in (b) of FIG. 6, when seven photographs are taken with a camera having a general angle-of-view lens having an FoV of about 65 degrees, the resolution is higher when the same range is photographed as 360 degrees.

However, if the number of input images increases, a general panorama image matching method that performs both homography matrix calculation, image warping, and blending processes can not perform real-time image matching because of a large amount of computation. Thereby causing a bottleneck phenomenon.

FIG. 3 is a diagram for explaining a 360-degree panoramic moving image generating system 100 according to an embodiment of the present invention.

3, a moving image generating system 100 according to an exemplary embodiment of the present invention includes an image acquiring unit 110 including a plurality of camera modules 111 and a plurality of embedded boards 112, a network 120 , And a matching image generating unit 130. [

The network 120 may be a dedicated line for connecting the image acquiring unit 110 and the matching image generating unit 130 or may support a wireless Internet communication such as wired Internet communication, WiFi, WiBro, WCDMA, LTE, It may be a wired or wireless communication network.

The matching image generating unit 130 may be a server on a network or a dedicated data processing apparatus such as a PC (Personal Computer).

A plurality of camera modules 111 (for example, 14 cameras) are arranged in a circle at equal intervals so as to be able to capture 360 degrees. Each of the plurality of camera modules 111 may be a 65-degree angle of view HD-class camera module.

A plurality of embedded boards 112 may be provided corresponding to a plurality (e.g., seven) of the plurality of camera modules 111 (e.g., 14) (for example, Each of the two camera modules controls two camera modules, primarily encodes and compresses the matching results of the images from the two camera modules, and transmits the result to the matching image generating unit 130, thereby halving the amount of transmitted data and preventing the bottleneck of data transmission / reception . Accordingly, it is possible to reduce the amount of computation by homography matrix calculation, image warping, blending, and the like for every frame in the matching image generating unit 130, thereby facilitating image matching in real time. That is, the matching image generating unit 130 may provide a high-resolution 360-degree panorama image in real time by performing homography matrix calculation, image warping, and blending processing only on half-moving images (e.g., seven matched images).

Each of the matching image generating unit 130 and the plurality of embedded boards 112 may be implemented by hardware such as a semiconductor processor, software such as an application program, or a combination thereof in order to perform such functions. And a modem for transmitting and receiving data to and from each other.

4 is a flowchart illustrating an operation of each embedded board 112 according to an embodiment of the present invention.

First, the embedded boards 112 (for example, seven) transmit a predetermined signal to the matching image generating unit 130 (e.g., PC) through the network 120 and try to connect to the matching board 130 (S110). When the embedded boards 112 are connected to the matching image generating unit 130 (e.g., PC) (S120), the embedded boards 112 wait for a start signal START from the matching image generating unit 130 (e.g., PC) (S130).

When the embedded signals 112 are received from the matching image generating unit 130 (e.g., PC), the embedded boards 112 control the camera modules 111 (e.g., 14 cameras) The control unit 112 receives the image (data) of each frame captured by the corresponding two camera modules (S140).

Each embedded board 112 primarily matches the images through the image warping using the homography matrix as described above with respect to the two images (S150). When the camera modules 111 are fixed, the homography matrix can be calculated and prepared according to the position coordinates. In some cases, when the camera modules 111 are not fixed, such as a movable type, each embedded board 112 calculates the homography matrix at that time using position coordinate information or the like received from the corresponding camera module, You may.

Each embedded board 112 compresses data in a manner such that only the changed part is compared with the previous frame, such as the H.264 video compression method, and stores the compressed video file in a storage unit such as a memory (S160).

Each embedded board 112 registers and compresses two images from two camera modules for a predetermined time, stores the compressed moving picture files in storage means such as a memory for a predetermined period of time (S170) To the matching image generating unit 130 (e.g., PC) (S180). Here, the time k for storing the files to be transferred in the storage means such as a memory by each embedded board 112 may be determined in proportion to the time required for image matching in the matching image generating unit 130 (e.g., PC). For this, the matching image generating unit 130 (e.g., PC) may notify the corresponding message to transmit the compressed moving picture files to each embedded board 112 when the image matching is performed by a predetermined number of frames.

As described above, according to the present invention, since the image matching is primarily performed in each embedded board 112 and transmitted, the number of images to be matched in the matching image generating unit 130 (e.g., PC) can be reduced by half.

Also, the time taken to transfer the image from the embedded board 112 to the matching image generating unit 130 (e.g., PC) is affected by the size of data to be transmitted, and the matching image generating unit 130 , PC), there is a bottleneck in data reception and a memory problem when the amount of data to be transmitted is large. Accordingly, since the image matching is primarily performed in each embedded board 112 as in the present invention, the number of images to be matched in the matching image generating unit 130 (e.g., PC) can be reduced by half, The matching result of the image is encoded, compressed, and transmitted to the matching image generating unit 130 (e.g., PC), thereby reducing the amount of transmitted data by half and preventing the bottleneck phenomenon of data transmission / reception. , PC) can reduce the amount of computation by homography matrix calculation, image warping, blending, and the like every frame, and enables image matching easily in real time.

5 is a flowchart illustrating an operation of the matching image generating unit 130 (e.g., a PC) according to an exemplary embodiment of the present invention.

First, the matching image generating unit 130 (e.g., a PC) attempts connection with the embedded boards 112 (e.g., seven) through the network 120 while receiving a predetermined signal (S210). When the matching image generating unit 130 (e.g., PC) is connected to the embedded boards 112 (S220), the matching signal generating unit 130 transmits a start signal (START) to the embedded boards 112 to control the photographing synchronization ).

The camera modules 111 (e.g., 14 cameras) start shooting and the matching image generating unit 130 (e.g., PC) receives each frame of one frame from each embedded board 112, (Step S240). The compressed moving picture files are transmitted to the mobile terminal 112 in units of a predetermined time (k). This method is intended to cope with the problem that buffering is limited due to a faster image receiving speed than an image matching speed as the number of cameras increases.

The matching image generating unit 130 (e.g., a PC) decodes the received compressed moving image files to acquire an image in units of frames (S250), and performs image warping using the precomputed homography matrix as described above (S260). The panorama image (data) is generated and displayed on the display device in real time (S270). The embedded board 112 may transmit position coordinate information and the like received from the camera module 111 and may be transmitted to the matching image generating unit 130 , PC) may receive it and then calculate the homography matrix and then match the images.

FIG. 6 is a diagram for explaining the concept of a matching process in the matching image generating unit 130 (for example, a PC) in the present invention.

As shown in FIG. 6, in the present invention, four types of threads are implemented so that each function can have independence. There is a queue between the thread and the thread that serves as a buffer to exchange data.

The first thread is a receiving thread, which is generated as many as the number of embedded boards, receives compressed moving picture files from each embedded board 112 on the client side, and transmits the corresponding files in a frame unit to the control thread.

The second thread is a control thread, and collects files of corresponding frames to be matched with one of sequentially received compressed moving picture files, and passes them to a third thread.

The third thread is a Render thread, which is created by n (natural numbers) to speed up and parallelize the work. Here, n is set according to the processor performance of the matching image generating unit 130 (e.g., PC). The matching thread applies warping and blending to the received frames to match in one frame. The matched frame is transferred sequentially to the display queue that each matching thread has.

The last thread is a display thread, which takes the registered images in order from the display queue and displays them on the display device in real time.

FIG. 7 shows an example of an implementation system using the camera modules 111 and the embedded boards 112 according to the present invention. The specifications of the client-side camera modules 111 and the embedded boards 112 used in the implementation and the specifications of the matching image generating unit 130 (e.g., PC) are summarized in Table 1 below.

[Table 1]

8 is a graph showing FPS (Frame per Sec., Frame generation per second) according to time of a matching image generated by the matching image generating unit 130 (e.g., PC) in the system implemented as shown in FIG.

8, (a) shows a case where a compressed moving picture file is transmitted for each frame without performing image matching in the embedded boards 112, (b) shows a case in which each embedded board 112 performs image matching And transmits compressed video files for k hours. However, it is assumed that each camera module 111 generates an image having a resolution of 640 * 480 and the matching is performed using a matching thread of n = 8 in the matching image generating unit 130 (e.g., PC).

As shown in FIG. 8, when a compressed moving picture file is transmitted for each frame without performing image matching on the embedded boards 112, a performance of a minimum of 1 FPS, a maximum of 8 FPS, and an average of 5.92 FPS is shown. (B), when the compressed moving picture file is transmitted for k hours by image matching of two images as shown in FIG. 4 (b), the maximum is from 4 FPS to 9 FPS and the average is 7.08 FPS. Was improved by 20%.

[Table 2] shows the output panorama image matched with the average matching time in the matching image generating unit 130 (e.g., PC) according to the input image resolution from the embedded boards 112 in the system implemented as shown in FIG. The resolution of the system is summarized. The average matching time is the time measured using only a single Render thread. When using 640 * 480 input image, it is possible to match about 3.2 times faster than 1280 * 720 input image, but resolution is reduced by 37%. 9 is an example of comparing the resolution of a matched output panoramic image for a 640 * 480 input image (a) and a 1280 * 720 input image (b) in Table 2.

[Table 2]

Table 3 below shows the results of comparing the panorama matching methods of the present invention with the viewing angle, camera number, camera FOV, resolution, real-time matching, and FPS in the system of the present invention.

[Table 3]

In the case of [6-8], which is used as a portable VR imaging device, real-time matching is impossible. In the case of [10] with the best matching speed, 6 cameras can match 4K resolution images at 30 frames per second. However, it has the disadvantage of using expensive cameras and specialized equipment such as DSP (Digital Signaling Processor) and FPGA (Field-ProgrammableGateArray). [17] is able to match images with 3K resolution at 1-2 frames per second, but has narrower and slower matching speeds than other methods. [18] is similar to [10] However, it has a drawback that it is impossible to produce a high-resolution image. In the system of the present invention, since a large number of cameras are used, the matching speed is slightly slower than in [10] and [18] in terms of the computation speed, but it is possible to match 6K of high resolution images with 14 cameras.

As described above, in the high-definition 360-degree image generation system 100 according to the present invention, a plurality of embedded boards 112 corresponding to two general angle-of-view camera modules are connected to the network- And PC, and generates panoramic moving images by matching the images transmitted from the respective boards with the matching image generating unit 130 (e.g., a PC). In this case, unnecessary work in the matching process is eliminated, And transmitted and stored the captured images in the form of encoded video files for a certain period of time. The high quality 360 degree image generating system 100 according to the present invention can generate a 360 degree panorama image from about 14 frames of 640 * 480 resolution input image at about 7 frames per second.

As described above, the present invention has been described with reference to particular embodiments, such as specific elements, and specific embodiments and drawings. However, it should be understood that the present invention is not limited to the above- Those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the essential characteristics of the invention. Therefore, the spirit of the present invention should not be construed as being limited to the embodiments described, and all technical ideas which are equivalent to or equivalent to the claims of the present invention are included in the scope of the present invention .

Camera module 111,
Embedded board 112
The image-
Network 120,
The matching image generating unit 130 generates a matching image,

Claims (8)

A plurality of camera modules;
A plurality of embedded boards each controlling a corresponding plurality of corresponding camera modules among the plurality of camera modules; And
And a matching image generator connected to the plurality of embedded boards through a network,
Wherein each of the plurality of embedded boards firstly matches images from the plurality of corresponding camera modules and transmits a corresponding compressed moving image file, and the registered image generating unit receives the compressed moving image files received from the plurality of embedded boards In addition,
Wherein the plurality of camera modules are arranged in a circle at predetermined intervals so as to be capable of shooting 360 degrees, and the matching image generating unit provides a high-resolution 360-degree panorama image in real time,
The plurality of camera modules are fixed so as to reduce the amount of data transferred by the primary matching, and the image matching on each of the plurality of embedded boards and the matching image generation using the homography matrix calculated in advance, And the image matching is performed in the image processing unit. delete delete The method according to claim 1,
Wherein each of the plurality of embedded boards stores the compressed moving picture files in a memory for a time determined to be proportional to the matching time in the matching image generating unit and transmits the compressed moving picture files to the matching picture generating unit. . Each of a plurality of embedded boards, each of which controls a corresponding plurality of corresponding camera modules among the plurality of camera modules, firstly matches images from corresponding camera modules and transmits corresponding compressed video files; And
And matching the compressed moving image files received from the plurality of embedded boards in a matching image generating unit connected to the plurality of embedded boards via a network,
Wherein the plurality of camera modules are arranged in a circle at predetermined intervals so as to allow 360-degree photographing, and provide a high-resolution 360-degree panoramic image in real time through the registered image generating unit,
The plurality of camera modules are fixed so as to reduce the amount of data transferred by the primary matching, and the image matching on each of the plurality of embedded boards and the matching image generation using the homography matrix calculated in advance, Wherein the image matching is performed to simplify the calculation. delete delete 6. The method of claim 5,
Wherein each of the plurality of embedded boards stores the compressed moving picture files in a memory for a time determined to be proportional to the matching time in the matching image generating unit and transmits the compressed moving picture files to the matching picture generating unit. Way.

Images