KR20220086888A - Novel view synthesis system for plenoptic images - Google Patents

Abstract

The present invention relates to a system for generating a mid-viewpoint image of a plenoptic image, wherein a plenoptic image is input and n images are parallelly moved in a direction to reduce the parallax between viewpoint images constituting the input plenoptic image. A parallel movement module, each of the images translated from the n image translation modules is input, and n intermediate view generating modules and the n intermediate view generating modules that respectively generate an intermediate view image with respect to each input image and an image synthesizing module for generating a new mid-view image by combining n mid-view images output from .
According to the present invention, there is an effect that a high-quality mid-view image can be generated from a plenoptic image with a large disparity.

Description

{Novel view synthesis system for plenoptic images}

The present invention relates to a new mid-view image generation technology of a plenoptic image obtained by photographing from a camera array or micro-lens-based camera widely used for applications such as 3D image storage.

A plenoptic image refers to a bundle of images simultaneously captured from multiple view points. And, the images constituting this bundle are called view images, and these view images are images taken simultaneously from several viewpoints.

1 shows a plenoptic imaging system.

In FIG. 1, (a) is a camera array-based system, and (b) illustrates a microlens-based system.

There are two main methods for obtaining a plenoptic image.

Referring to FIG. 1 , a method (a) of arranging several cameras in a two-dimensional arrangement and taking pictures at the same time, and a microlens array (small lenses in a two-dimensional array form) in front of the image sensor inside the camera even though it is one camera There is a method (b) to achieve the same effect as installing multiple cameras by installing Either method is the same in that the camera is positioned at different positions and the effect of shooting is obtained. Therefore, such an imaging system is collectively called a plenoptic camera.

Each viewpoint image has its own viewpoint, and the greater the difference between viewpoints, the greater the need for an intermediate viewpoint (that is, a viewpoint that is not photographed). This is because the 3D information in the video can be extracted by utilizing the viewpoint images, and can be utilized for more realistic 3D content creation and reproduction.

In the prior art, a problem occurs when the disparity of the plenoptic image is large. Compared to the small disparity estimation, large disparity estimation causes inaccuracy, and thus the quality of generating an intermediate view image is poor.

The parallax of the plenoptic image is greatly affected by the preset camera viewpoint. Here, the parallax refers to a distance (distance between pixels) between a point in 3D space when a point in the three-dimensional space is captured by a camera from a different viewpoint, and appears as a single point in each camera image. Since parallax includes three-dimensional distance information, it is an essential element for three-dimensional reconstruction of an object and generation of an unphotographed mid-view image.

A representative example of the prior art is the method of Calantari. This method is based on a deep learning network, and generates mid-view images by receiving four different viewpoint images. This method generates an excellent mid-view image for an image with a small disparity, but has a disadvantage in that a proper mid-view image is not generated for an image with a large disparity.

Republic of Korea Patent Registration 10-1580284

The present invention has been devised to solve the above problems. In the present invention, the method for generating a mid-view image of a plenoptic image with a small disparity is repeatedly applied, and then the results are combined, so that even a plenoptic image with a large disparity is well performed. An object of the present invention is to provide a system for generating a mid-viewpoint image of a working plenoptic image.

Objects of the present invention are not limited to the objects mentioned above, and other objects not mentioned will be clearly understood by those skilled in the art from the following description.

The present invention for achieving the above object relates to a system for generating a mid-viewpoint image of a plenoptic image, wherein each image in a direction of reducing the parallax between the inputted plenoptic image and the viewpoint images constituting the inputted plenoptic image n image translation module for translating the and an image synthesizing module for generating a new mid-view image by combining the n mid-view images output from the n mid-viewpoint generating modules.

는 화소단위의 평행이동량을 나타내며,

은 인접한 4장의 시점 영상으로서, 각각 좌측상단, 우측상단, 좌측하단, 우측하단에 위치한 시점에서 촬영된 영상이라고 할 때, 상기 영상 평행 이동 모듈에서 다음 수학식 1을 적용하여 시차가 작아지는 방향으로 플렌옵틱 영상의 시점 영상들을 평행이동시키고,

represents the amount of parallel movement in units of pixels,

is an image of four adjacent viewpoints, respectively, in a direction in which the parallax is decreased by applying the following Equation 1 in the image parallel movement module when the images are taken from viewpoints located at the upper left, upper right, lower left, and lower right corners, respectively. Translate the viewpoint images of the plenoptic image,

,

,

,

(수학식 1)

,

,

,

(Equation 1)

는

을 시차가 감소하는 방향으로 각각

/2만큼씩 평행이동한 영상을 나타낸다. At this time,

Is

in the direction of decreasing parallax, respectively

Shows the image translated by /2.

An interval of the amount of parallel movement performed by the image parallel movement module may be set to be less than or equal to a maximum parallax allowed by the midpoint generation module.

는 모듈 B의 k번째 모듈의 출력이고, M은 비용계산을 위한 마스크의 크기이고,

는

를 최소화하는 모듈 번호 k값을 의미할 때, In order to combine the mid-viewpoint images generated by the mid-viewpoint generation module in the image synthesis module,

is the output of the k-th module of module B, M is the size of the mask for cost calculation,

Is

When it means the module number k value that minimizes

(수학식 2)

(Equation 2)

, 여기서

(수학식 3)

, here

(Equation 3)

번째 중간시점 생성 모듈의 출력 화소

를 조합함으로써 최종적인 중간시점 영상

을 산출할 수 있다. By applying

The output pixel of the second midpoint generation module

The final mid-view image by combining

can be calculated.

According to the present invention, in order to solve the problem that the existing mid-view image generating technique for plenoptic images does not work well for plenoptic images with large disparity, the disparity of the input image of the existing mid-view image generating technique can be reduced. A number of plenoptic images are generated through parallel movement to ensure that By combining them, there is an effect of generating a mid-view image of a plenoptic image with a large disparity.

As a result, the present invention can extend the existing method for generating a midpoint having a limited disparity range to a large disparity range, and by expanding the number of n, which is a detailed module constituting the module, the disparity tolerance range can be expanded as much as desired. there is

1 shows a plenoptic imaging system.
2 illustrates a system for generating a mid-viewpoint image of a plenoptic image proposed by the present invention.
FIG. 3 illustrates an image generated through a mid-viewpoint imaging method for a plenoptic image with a large parallax.

Advantages and features of the embodiments disclosed herein, and methods for achieving them will become apparent with reference to the embodiments described below in conjunction with the accompanying drawings. However, the embodiments to be proposed in the present disclosure are not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments are provided to those of ordinary skill in the art. It is only provided to fully indicate the category.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail.

The terms used in this specification have been selected as currently widely used general terms as possible while considering the functions of the disclosed embodiments, but may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the detailed description part of the corresponding specification. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the entire specification, rather than the simple name of the term.

References in the singular herein include plural expressions unless the context clearly dictates the singular.

In the entire specification, when a part "includes" a certain element, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated. Also, as used herein, the term “unit” refers to a hardware component such as software, FPGA, or ASIC, and “unit” performs certain roles. However, "part" is not meant to be limited to software or hardware. A “unit” may be configured to reside on an addressable storage medium and may be configured to refresh one or more processors. Thus, by way of example, “part” includes components such as software components, object-oriented software components, class components and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functionality provided within components and “parts” may be combined into a smaller number of components and “parts” or further divided into additional components and “parts”.

In addition, in the description with reference to the accompanying drawings, the same components are given the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In describing the present invention, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

2 illustrates a system for generating a mid-viewpoint image of a plenoptic image proposed by the present invention. In FIG. 2, a mid-viewpoint image generating system for outputting a mid-viewpoint image by inputting the plenoptic image of the present invention is shown.

Referring to FIG. 2 , the mid-viewpoint image generating system of the plenoptic image includes n image translation modules (A), n mid-viewpoint generating modules (B), and an image synthesis module (C).

When a plenoptic image is input, the n image translation module A translates each image in a direction that reduces the parallax between viewpoint images constituting the input plenoptic image.

When each of the images translated by the n image translation module A is input, the n mid-viewpoint generating module B generates mid-view images for each inputted image, respectively.

The image synthesizing module (C) generates a new midpoint image by combining the n midview images output from the n midpoint generating modules.

In the present invention, the interval of the amount of parallel movement performed by the image parallel movement module (A) may be set to be less than or equal to the maximum parallax allowed by the mid-viewpoint generation module (B).

2, the mid-viewpoint imaging system of the plenoptic image proposed by the present invention is provided with n image translation modules (A) that translate each image in a direction to reduce the parallax between input images, n mid-viewpoint generation modules (B) of a plenoptic image with small disparity are provided, and a new mid-view image is generated by combining n mid-viewpoint images output from the mid-viewpoint generation module (B). It is composed of an image synthesis module (C).

The image parallel movement module (A) is a module that performs an image translation operation. When four adjacent viewpoint images among viewpoint images of a plenoptic image are input, the image translation module (A) is translated in a direction in which the parallax between each input image becomes smaller. make it That is, according to the following equation.

[Equation 1]

,

,

,

,

,

,

는 평행이동량(화소단위)을 나타내며,

은 인접한 4장의 시점 영상으로서, 각각 좌측상단, 우측상단, 좌측하단, 우측하단에 위치한 시점에서 촬영된 영상이며,

는

을 시차가 감소하는 방향으로 각각

/2만큼씩 평행이동한 영상을 나타낸다.In [Equation 1],

represents the amount of translation (in pixels),

is an image taken from the viewpoints located in the upper left, upper right, lower left, and lower right corners, respectively, as four adjacent viewpoint images,

Is

in the direction of decreasing parallax, respectively

Shows the image translated by /2.

=0, 20, 40, ..., 180가 됨에 따라 최대 허용 시차는 180이 된다. 이는 제안된 방법이 최대 180화소의 큰 시차를 가진 플렌옵틱 영상의 중간시점 영상 생성이 가능하다는 것을 의미한다. The total number n of the image translation modules A is 2 or more, and may vary according to the maximum allowable disparity. For example, if n=10 and the translation interval is 20 (in pixels), the translation

= 0, 20, 40, ..., 180, the maximum allowable disparity becomes 180. This means that the proposed method can generate mid-view images of plenoptic images with a large parallax of up to 180 pixels.

On the other hand, the translation interval (the interval of s value) of the image translation module (A) is linked with the maximum disparity allowed by the midpoint generation module (B), and the translation interval is allowed by the midpoint generation module (B) It should be set to less than the maximum time difference.

The mid-viewpoint generation module (B) generates n mid-viewpoint images by inputting n plenoptic images generated by the translation in the image translation module (A) as input.

The mid-viewpoint generating module B is a mid-viewpoint generating image module that receives a plenoptic image with a small disparity as an input, and may use a technique for generating a mid-viewpoint image with respect to a plenoptic image with a small disparity. For example, you can use the calantari technique based on a deep learning network, which works well for images with a parallax of +/- 21 pixels.

3 exemplifies an image generated through a mid-viewpoint imaging method for a plenoptic image with a large parallax.

3 is an example of a result image obtained by generating an intermediate view image using the calantari technique by receiving a plenoptic image having a parallax in the range of 100 to 200 pixels. It can be seen that the generation performance drops sharply.

In order to describe the operation of the image synthesis module (C), first, the operation characteristics of the mid-viewpoint generation module (B) as follows.

Each of the mid-view images output from the mid-view generating module (B) shows excellent image quality for pixels whose parallax is reduced by the translation of the image translation module (A), whereas in other cases, it shows poor image quality. . That is, the image output from each of the mid-viewpoint generating module (B) has a mixed region for outputting excellent image quality and a region for outputting poor image quality, and the amount of parallel movement of each input image of the mid-viewpoint generating module (B) is mixed. Each output of the mid-viewpoint generating module (B) also has a characteristic that all regions showing excellent image quality are different according to the difference.

By using these characteristics, the image synthesis module (C) selects and synthesizes only regions with excellent image quality in the images (total n images) output from each of the mid-viewpoint generation modules (B), thereby generating a new mid-viewpoint image. print out

In the image synthesis module (C), in determining whether the region is of excellent image quality, it is determined whether the image quality is excellent in units of pixels. [Equation 2] is a cost calculation formula for this determination, using the characteristic that pixels with excellent image quality maintain similarly excellent image quality even when the parallel movement values performed by the image parallel movement module (A) are immediately adjacent to each other. will be.

[Equation 2]

는 모듈 B의 k번째 모듈의 출력이고, M은 비용계산을 위한 마스크의 크기를 의미한다. In [Equation 2]

is the output of the k-th module of module B, and M is the size of the mask for cost calculation.

[Equation 2] is the (k-1), k, (k+1)-th translation image of the image translation module (A), respectively, to the three outputs of the three intermediate viewpoint generation module (B). This is the formula for calculating the error.

를 최소화하는 k를 찾고, 찾아진 k를 기반으로 화소를 조합함으로써 최종적인 중간시점 영상을 생성한다. 이 과정은 [수학식3]으로 표현될 수 있다. As the image synthesis module (C) has a characteristic of maintaining similarly excellent image quality when the translation values performed by the image translation module (A) are immediately adjacent to each other,

A final mid-view image is generated by finding k that minimizes , and combining pixels based on the found k. This process can be expressed by [Equation 3].

[Equation 3]

, 여기서

, here

는

를 최소화하는 모듈 번호 k값을 의미하며, 해당되는

번째 중간시점 생성 모듈(B)의 출력 화소

를 조합함으로써 최종적인 중간시점 영상

을 얻는다. In [Equation 3]

Is

It means the module number k value that minimizes

The output pixel of the second midpoint generation module (B)

The final mid-view image by combining

to get

In the present invention, in order to solve the problem that the existing mid-viewpoint image generating technique for a plenoptic image does not work well for a plenoptic image with a large disparity, A plurality of plenoptic images are generated through parallel movement, the existing method is applied to the plurality of generated plenoptic images to obtain the corresponding number of mid-view images, and pixels judged to have excellent image quality are selected for each pixel. By combining, we propose a method to achieve midpoint generation of large parallax plenoptic images.

As a result, the present invention proposes a method of expanding the existing method for generating a midpoint having a limited disparity range to a large disparity range, and by expanding the number of detailed modules n constituting the module, the allowable disparity range can be expanded as much as desired. have.

The present invention has been described above using several preferred embodiments, but these embodiments are illustrative and not restrictive. Those of ordinary skill in the art to which the present invention pertains will understand that various changes and modifications can be made without departing from the spirit of the present invention and the scope of the rights set forth in the appended claims.

A image translation module
B midpoint generation module
C image synthesis module

Claims (4)

an n image translation module to which a plenoptic image is input and which translates each image in a direction to reduce a parallax between viewpoint images constituting the input plenoptic image;
n mid-viewpoint generation modules for receiving each of the images translated by the n-image translation module and generating mid-view images with respect to each inputted image; and
An image synthesizing module for generating a new mid-view image by combining n mid-view images output from the n mid-viewpoint generating modules
A mid-viewpoint image generation system of a plenoptic image comprising a.
The method according to claim 1,

represents the amount of parallel movement in units of pixels,

is an image from four adjacent viewpoints.
In the image parallel movement module, applying Equation 1 below to translate the viewpoint images of the plenoptic image in a direction in which the parallax becomes smaller,

,

,

,

(Equation 1)
At this time,

Is

in the direction of decreasing parallax, respectively

A mid-viewpoint image generating system of a plenoptic image, characterized in that it represents an image translated by /2.
3. The method according to claim 2,
An intermediate view image generating system of a plenoptic image, characterized in that the interval of the amount of parallel movement performed by the image parallel movement module is set to be less than or equal to the maximum parallax allowed by the intermediate view generating module.
4. The method according to claim 3,
In order to combine the mid-viewpoint images generated by the mid-viewpoint generation module in the image synthesis module,

is the output of the k-th module of module B, M is the size of the mask for cost calculation,

Is

When it means the module number k value that minimizes

(Equation 2)

, here

(Equation 3)
By applying

Output pixel of the second midpoint generation module

The final mid-view image by combining

A mid-viewpoint image generation system of a plenoptic image, characterized in that for calculating .

Images