KR20220071935A - Method and Apparatus for Deriving High-Resolution Depth Video Using Optical Flow - Google Patents

Abstract

The present disclosure relates to a method and apparatus for deriving a high-resolution depth image using an optical flow. According to an embodiment of the present disclosure, the method for deriving a depth image using an optical flow comprises: estimating an optical flow between a first viewpoint image and a second viewpoint image; generating a ray direction for each pixel of the first viewpoint image based on the estimated optical flow; and deriving a depth image of the first viewpoint image using the generated ray direction, wherein the ray direction is expressed as a direction vector in a 3D space, and the depth image may be derived based on parameters of a camera that has captured the first viewpoint image. According to the present disclosure, it is possible to efficiently and quickly derive a high-resolution depth image using an optical flow and provide an optical flow-based 3D coordinate estimation scheme.

Description

Method and Apparatus for Deriving High-Resolution Depth Video Using Optical Flow

The present disclosure relates to a method and apparatus for estimating a high-resolution depth image using an optical flow.

Recently, the world is rapidly changing in the face of the strong wave of the 4th industrial revolution. The change that started with so-called smartization is transforming all fields innovatively. Due to the 4th industrial revolution, we are now living in an era of complete digitalization and artificial intelligence where we can operate devices and collect data anytime, anywhere by connecting to mobile and internet. Core technologies leading the 4th industrial revolution era include augmented reality and virtual reality.

Augmented reality (AR) is a system that augments the real experience by adding additional virtual information to the information in the real world. Or, it refers to a technology that stimulates the five senses of the human body by providing an experience or environment that cannot be obtained, so that it can be experienced as if it were real. Recently, virtual reality technology is developing into a future computing environment technology that communicates through free actions and five senses across virtual reality and reality, and is expected to change people's lives and affect the development of a wide range of industries.

The virtual reality service is evolving in the direction of providing a service that maximizes the sense of immersion and realism by generating omnidirectional images in the form of live images or CG (Computer Graphics) and playing them on HMD (Head Mounted Display) and smartphones.

Accordingly, in recent AR/VR services, high-resolution and high-quality images are increasingly required, and virtual viewpoint images must be provided in the virtual space provided by VR/AR, which also requires high-resolution/high-quality rendering technology. do. In this case, the quality of the virtual view image may be affected depending on the quality of the depth image, and a lot of time and money are incurred to improve the quality of the depth image. Therefore, there is a need for a technique for a method for estimating an accurate depth image.

As a depth image estimation method, a method is mainly used to find the corresponding point of the left and right images, obtain the displacement, and then convert it back to the depth (distance). have. In this case, when the block matching method is used, when the difference between the corresponding pixels between the left and right images does not occur significantly, the depth estimation may be inaccurate.

In addition, according to the stereo matching-based depth estimation method that can scan the corresponding points of each image at high speed in the existing pinhole camera model, the image distorted through the fisheye lens is an epipolar geometry constraint. Since it cannot be applied, using the existing method may lead to inaccurate depth estimation.

It is an object of the present disclosure to provide a high-resolution depth image estimation technique using an efficient and fast optical flow.

It is an object of the present disclosure to provide a depth estimation technique based on optical flow calculation to enable free and more accurate depth estimation.

An object of the present disclosure is to more accurately estimate a depth image from a lens regardless of whether a photographing image is distorted.

Other objects and advantages of the present disclosure may be understood by the following description, and will become more clearly understood by the examples of the present disclosure. It will also be readily apparent that the objects and advantages of the present disclosure may be realized by the means and combinations thereof indicated in the appended claims.

According to an embodiment of the present disclosure, a method for estimating a depth image using an optical flow includes estimating an optical flow between a first viewpoint image and a second viewpoint image; generating a ray direction for each pixel, and estimating a depth image of the first viewpoint image using the generated ray direction, wherein the ray direction is expressed as a direction vector in a three-dimensional space, The depth image may be estimated based on a camera parameter capturing the first viewpoint image.

Meanwhile, the estimated optical flow may be estimated based on deep learning.

Meanwhile, the camera parameter may include a center position of the camera.

Meanwhile, the estimating of the depth image may include obtaining coordinates in the 3D space for each pixel of the first viewpoint image by using the generated ray direction.

Meanwhile, the coordinates may be obtained based on gradient descent using the minimum MSE.

According to an embodiment of the present disclosure, an apparatus for estimating a depth image using an optical flow includes a memory for storing data and a processor for controlling the memory, wherein the processor includes an optical flow between a first viewpoint image and a second viewpoint image. , generating a ray direction for each pixel of the first viewpoint image based on the estimated optical flow, and estimating a depth image of the first viewpoint image using the generated ray direction, A direction is expressed as a direction vector in a three-dimensional space, and the depth image may be estimated based on a camera parameter for capturing the first viewpoint image.

According to an embodiment of the present disclosure, a depth image estimation program using an optical flow stored in a non-transitory computer-readable medium includes, in a computer, estimating an optical flow between a first viewpoint image and a second viewpoint image, the estimated generating a ray direction for each pixel of the first viewpoint image based on an optical flow, and estimating a depth image of the first viewpoint image using the generated ray direction, wherein the ray direction is expressed as a direction vector in a three-dimensional space, and the depth image may be estimated based on a camera parameter for capturing the first viewpoint image.

According to the present disclosure, it is possible to efficiently and quickly perform high-resolution depth image estimation using an optical flow.

According to the present disclosure, it is possible to provide an optical flow-based 3D coordinate estimation technique using an image of a peripheral viewpoint.

According to the present disclosure, it is possible to optimize light rays generated from multiple optical streams.

According to the present disclosure, it is possible to freely and accurately estimate a high-resolution depth image in various lens-based environments by estimating a more accurate depth image from a lens regardless of whether a photographing image is distorted.

Effects obtainable in the embodiments of the present disclosure are not limited to the above-mentioned effects, and other effects not mentioned are the technical fields to which the technical configuration of the present disclosure is applied from the description of the embodiments of the present disclosure below. It can be clearly derived and understood by those of ordinary skill in the art. That is, unintended effects of implementing the configuration described in the present disclosure may also be derived by those of ordinary skill in the art from the embodiments of the present disclosure.

1 shows an example of generating a plenoptic point cloud.
FIG. 2 is a diagram for explaining a method of expressing attribute information allocated to a plenoptic point according to a position of a viewpoint.
3 is a view for explaining a method for generating a multi-viewpoint image according to an embodiment of the present disclosure.
4 and 5 are diagrams for explaining the concept of a conventional epipolar geometry constraint.
6 is a diagram for explaining a conventional method of estimating a depth image based on stereo matching.
7 is a view for explaining a depth image estimation method using an optical flow according to an embodiment of the present disclosure.
8 is a view for explaining an apparatus for estimating a depth image using an optical flow according to an embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present disclosure pertains can easily implement them. However, the present disclosure may be embodied in several different forms and is not limited to the embodiments described herein.

In describing an embodiment of the present disclosure, if it is determined that a detailed description of a known configuration or function may obscure the gist of the present disclosure, a detailed description thereof will be omitted. And, in the drawings, parts not related to the description of the present disclosure are omitted, and similar reference numerals are attached to similar parts.

In the present disclosure, the components that are distinguished from each other are for clearly explaining each characteristic, and the components do not necessarily mean that the components are separated. That is, a plurality of components may be integrated to form one hardware or software unit, or one component may be distributed to form a plurality of hardware or software units. Accordingly, even if not specifically mentioned, such integrated or distributed embodiments are also included in the scope of the present disclosure.

In the present disclosure, components described in various embodiments do not necessarily mean essential components, and some may be optional components. Accordingly, an embodiment composed of a subset of components described in an embodiment is also included in the scope of the present disclosure. In addition, embodiments including other components in addition to components described in various embodiments are also included in the scope of the present disclosure.

Meanwhile, in describing an embodiment of the present disclosure, when images for two or more viewpoints can be used, the current viewpoint and other viewpoints are divided in order to distinguish one viewpoint from another viewpoint. However, the current view is not fixed to a specific view, and at an arbitrary temporal moment, a target view from which a depth image is to be estimated may be referred to as a current view.

Hereinafter, the present disclosure will be described in detail with reference to the drawings.

1 shows an example of generating a plenoptic point cloud. A plenoptic point is one piece of geometry information expressed in three-dimensional coordinates such as X, Y, and Z in a three-dimensional space, and N pieces of RGB, YUV, etc. It is a data type that includes attribute information. A plenoptic point cloud is a set of plenoptic points, and may include at least one plenoptic point.

The plenoptic point cloud may be generated using a 2D image and depth information for each of the N input viewpoints. In this case, the 3D space may be defined through a space including all the generated points.

Here, the 2D image may mean images acquired by one or more cameras, such as a multi-view image and a lightfield image. In addition, the multi-view image may be composed of images simultaneously photographed by a plurality of cameras having different viewpoints in a specific area.

In this case, the defined 3D space may be divided into predetermined unit voxels, and points within the voxels may be merged to have one geometric information value. Also, at this time, all color information of the three-dimensional points is stored, and a plenoptic point cloud may be generated by using the information regarding the point generated from which point.

The color information of a viewpoint in which a 3D point is not generated may be inferred from color information of other viewpoints included in the same voxel. For example, color information of a viewpoint in which a 3D point is not generated may be derived using at least one of an average value, a maximum value, and a minimum value of color information of other viewpoints or points in the voxel. Also, for example, color information of a viewpoint in which a 3D point is not generated may be derived from color information of a viewpoint or point adjacent to the corresponding viewpoint or point.

Meanwhile, when a plurality of 3D points generated at one viewpoint are included in one voxel, a plenoptic point cloud may be generated by storing at least one of the average value, the maximum value, and the minimum value of the color values of the corresponding viewpoint. .

As another example, if multiple 3D points generated at one point are included in one voxel, a plenoptic point cloud can be created by storing the color information of the point with the smallest depth information or the largest depth information. have.

Here, a voxel having one or more attribute information may be referred to as a multi-attribute voxel. That is, the multi-attribute voxel may mean a plenoptic point.

FIG. 2 is a diagram for explaining a method of expressing attribute information allocated to a plenoptic point according to a position of a viewpoint.

와 h를 이용한 2차원 형태로 표현할 수 있다. 여기서,

는 실수 값, 정수 값 등으로 표현되는 각도를 의미하며, h는 실수 값, 정수 값 등의 표현되는 크기를 의미할 수 있다. 즉, 플렌옵틱 포인트에 할당된 속성 정보는

와 h로 표현되는 2차원 좌표 값을 가질 수 있다.As in the example of FIG. 2 , the attribute information assigned to the generated plenoptic point depends on the position of the viewpoint.

It can be expressed in a two-dimensional form using and h. here,

may mean an angle expressed by a real value, an integer value, etc., and h may mean a size expressed by a real value, an integer value, or the like. That is, the attribute information assigned to the plenoptic point is

It can have a two-dimensional coordinate value expressed by and h.

In order to efficiently remove redundancy between attribute information of plenoptic points of N views expressed in a two-dimensional form, attribute analysis may be performed. At this time, the property decomposition of the plenoptic point includes prediction between property information for property information, padding of property information, extrapolation of property information, interpolation of property information, conversion of property information ( transform) and quantization of attribute information may mean performing at least one of.

Attribute decomposition may mean decomposing attributes into view-dependent attribute information and view-independent attribute information. Here, the view-dependent attribute information may mean unique attribute information of each view, and the view-independent attribute information may refer to attribute information that is common to one or more views. For example, view-dependent attribute information may mean a specular component, and view-independent attribute information may mean a diffuse component.

3 is a view for explaining a method of generating a multi-viewpoint image according to an embodiment of the present disclosure.

First, a 3D object may be projected as a 2D image by using the geometric information of the plenoptic point cloud (S300). In addition, a 3D object (point) may be projected as a 2D image by using the occlusion pattern information of the plenoptic point cloud. For example, when occlusion pattern information corresponding to an arbitrary viewpoint among occlusion pattern information indicates meaningless attribute information, the corresponding point may not be projected to the corresponding viewpoint. The occlusion pattern information may have a first value of '1' and a second value of '0'. For example, when the occlusion pattern information of an arbitrary viewpoint has a first value, a pixel value of a position projected to the corresponding viewpoint may be determined as attribute information of the corresponding viewpoint of the corresponding point. On the other hand, when the occlusion pattern information has the second value, the point may not be projected to the corresponding viewpoint.

In this case, when only one point is projected on the current pixel, the attribute information of the current pixel may be determined using attribute information of (the point of) the current pixel ( S305 and S310 ). On the other hand, when a plurality of points are projected on the current pixel, the attribute information of the corresponding pixel may be determined using attribute information of (a point of) that is closest to the camera ( S305 and S320 ).

On the other hand, with respect to attribute information in which the projection of the plenoptic point is not performed on the current pixel, attribute information of N neighboring pixels of the current pixel may be referred to. In this case, property information of the current pixel may be determined using property information of a neighboring pixel having the closest distance to the camera ( S315 and S330 ). In this case, N may be a natural number greater than 0, for example, may have a value of 4 or 8.

In addition, if the projection of the plenoptic point is not performed on the current pixel and there is no property information or no projected points of the N neighboring pixels of the current pixel, using an interpolation method using an NxN mask on a two-dimensional image, Hole filling may be performed on the current sample (S315 and S340). In this case, N may be a natural number greater than 0, for example, may have a value of 5.

4 and 5 are diagrams for explaining the concept of a conventional epipolar geometry constraint. More specifically, FIG. 4 is a diagram for explaining an epipolar geometry constraint that exists when an image is captured from two or more viewpoints with a pinhole camera, and FIG. 5 is a point of an image according to the epipolar geometry constraint. It is a diagram for explaining an epipolar line for finding a point that can correspond to another image in .

As an example, the pinhole camera model may be applied to a general camera that can contain a perspective view on a general two-dimensional plane. When an image is taken from two or more viewpoints with such a pinhole camera, an epipolar geometry constraint may exist.

이라 할 수 있다. 차원 공간 상의 점

는 중심점

과 X를 잇는 선분 상에 있는 점들은 좌시점에서 바라보았을 때 좌측 이미지 평면에서는

로만 표현될 수 있다. 즉,

는 하나의 점

과 위치가 동일하나 깊이가 상이한 점으로 나타나나, 우 시점에서 바라보았을 때는

가 이미지 평면상에 놓이게 될 수 있다. 한편, 이미지 평면이나

는 동일한 선(line) 위에 놓이게 될 수 있다. As an example, as shown in FIG. 4 , an image for a left view and an image for a right view may exist. in this case. The center point of the image for each viewpoint

it can be said point in dimensional space

is the center point

Points on the line segment connecting X and X are on the left image plane when viewed from the left viewpoint.

can only be expressed as in other words,

is a single point

It appears as a point with the same location as the one with a different depth, but when viewed from the right

may lie on the image plane. On the other hand, the image plane or

may lie on the same line.

The part where these points lie is called an epipolar line. In other words, when based on one viewpoint, points in a three-dimensional space that satisfy certain conditions are based on another viewpoint, the epipolar geometry constraint is a situation in which it is only necessary to find points on the epipolar line of the image plane. (Epipolar geometry constraints).

When this is described with reference to FIG. 5 , the epipolar geometry constraint can be characterized in the following two ways. A point to which the red point p on the image plane of one view may correspond to the image plane of the other view may be any point on the epipolar line 1 .

Similarly, a point to which the red point p' of the image plane at different viewpoints may correspond to in the other image plane may be a point on the epipolar line 2 .

6 is a diagram for explaining a conventional method of estimating a depth image based on stereo matching.

As an example, the conventional stereo matching may be based on a multi-viewpoint image, and it is assumed that an image captured from at least two viewpoints is used. In addition, the above-mentioned epipolar line may be based on the corresponding point search process according to the search.

The process of generating an epipolar line ( S601 ) includes aligning images captured at two viewpoints, and searching for an epipolar line between images photographed at two viewpoints using a camera parameter, etc. may include the step of Here, the camera parameter may mean a parameter of a camera that captures an image for two viewpoints.

The process of searching for a corresponding point based on the generated epipolar line ( S602 ) may include a disparity estimation step, which includes a current view image (a first view image) and a current view point based on the generated epipolar line. Disparity may be estimated by finding a correspondence point between images of different viewpoints (second viewpoint images).

Thereafter, the depth image based on the camera parameter may be estimated using the obtained corresponding point ( S603 ). The estimated disparity may be converted into depth. Here, the camera parameter used may include a camera position, viewpoint-related values, and the like.

Meanwhile, the stereo matching-based depth image estimation method of FIG. 6 is most optimized in the existing pinhole camera model. As an example, when the method of FIG. 6 is applied by using an image distorted through a fisheye lens as a multi-view image, an epipolar geometry constraint cannot be applied.

7 is a view for explaining a depth image estimation method using an optical flow according to an embodiment of the present disclosure.

As an embodiment, the depth image estimation method of FIG. 7 may be performed by a depth image estimation apparatus, etc., including the apparatus of FIG. 8 .

As an example, a method of estimating a depth image using an optical flow may use a multi-view image. For example, image(s) for two or more viewpoints may be used. The multi-viewpoint image(s) used herein may be obtained by scanning one camera while moving, or may be obtained by one or more cameras.

First, the method for estimating a depth image using an optical flow may include estimating an optical flow between multi-viewpoint images ( S701 ). As an example, this may include estimating an optical flow in the image using images for two or more viewpoints. The step of estimating an optical flow between a current viewpoint (a first viewpoint) and other viewpoints may be included for images captured for two or more viewpoints. For example, artificial intelligence may be used for a method of obtaining an optical flow and may be based on deep learning, but the present disclosure is not limited thereto.

를 사용하여 각 시점 영상의 임의의 픽셀(i 픽셀)들 각각에 대한 3차원 공간상의 방향벡터

생성하는 단계를 포함할 수 있다. 또한, 이는 다시점 영상 중 특정 시점의 영상을 기준으로 다른 시점 영상에 대한 광선 방향을 생성하는 과정일 수 있다. Thereafter, a process of generating a ray direction based on the estimated optical flow ( S702 ) may be performed. As an example, in this process, the ray direction of each of arbitrary pixels (eg, i pixels) of each viewpoint image obtained from the optical flow

direction vector in 3D space for each arbitrary pixel (i-pixel) of each viewpoint image using

It may include the step of generating. Also, this may be a process of generating a ray direction for an image of a different viewpoint based on an image of a specific viewpoint among multi-view images.

, 제n 카메라 화각 등이 포함될 수 있으며, 여기서 카메라의 위치란 카메라의 시야의 중점, 즉 카메라에 의해 획득되는 영상의 중점을 의미할 수 있다. 또한, 깊이 영상은 방향 벡터

등을 더 사용하여 추정될 수 있다. 즉, 방향 벡터는 현재 시점의 영상의 임의의 픽셀에 대한 3차원 공간상의 좌표에 대한 최적해를 계산하는 데 사용될 수 있다. 일 예로서, 최적해를 구하는 방법은 최소 MSE(Mean Square Error)를 이용한 경사 하강법(gradient descent) 등 다양한 방법들을 이용할 수 있다. 이때 구해지는 최적해, 즉 3차원 공간 상의 좌표 P=(X,Y,Z)를 사용하여 현재 시점 픽셀 위치의 깊이 값이 추정되어, 이를 모든 픽셀에 반복하면, 깊이 영상이 추정될 수 있다. 일 예로서, 최적의 좌표는 하기와 같이 구해질 수 있다.Also, the step of estimating the depth image ( S703 ) may be performed using the generated ray direction. The generated ray direction may be the direction vector for each pixel mentioned above. As an example, the step of estimating the depth image may be based on the camera parameters for capturing images of the two or more viewpoints mentioned above. As an example, the camera parameter includes the position of an arbitrary nth camera.

, n-th camera angle of view, etc. may be included, and the position of the camera may mean a midpoint of a field of view of the camera, that is, a midpoint of an image acquired by the camera. Also, the depth image is a direction vector

It can be estimated using further That is, the direction vector may be used to calculate an optimal solution for coordinates in 3D space for any pixel of the image of the current view. As an example, various methods such as gradient descent using a minimum mean square error (MSE) may be used as a method of obtaining an optimal solution. At this time, the depth value of the current view pixel position is estimated using the obtained optimal solution, ie, coordinates P=(X,Y,Z) in the three-dimensional space, and repeating this for all pixels can estimate the depth image. As an example, the optimal coordinates may be obtained as follows.

[Equation 1]

Meanwhile, since the flowchart of FIG. 7 corresponds to an embodiment of the present disclosure, the present disclosure is not limited thereto. Accordingly, it will be said that the order of some steps is changed, other steps are added, some steps are removed, or it is possible to proceed simultaneously.

8 is a view for explaining an apparatus for estimating a depth image using an optical flow according to an embodiment of the present disclosure.

As an example, the apparatus for estimating a depth image using an optical flow may perform the above-mentioned depth image estimation, and the depth image estimation method of FIG. 7 is included.

As an example, the apparatus 801 for estimating a depth image using an optical flow according to the embodiment shown in FIG. 8 may include a memory 802 for storing data and a processor 803 for controlling the memory. Meanwhile, although not shown in FIG. 8 , a transceiver or a user input/output interface for transmitting/receiving data with an external device may be further included, and the present disclosure is not limited thereto.

As an example, the processor may estimate an optical flow between multi-viewpoint images, generate a ray direction for each pixel of a current view image among multi-view images based on the estimated optical flow, and use the generated ray direction to A depth image of a predetermined viewpoint image is estimated, a ray direction is expressed as a direction vector in a three-dimensional space, and the depth image may be estimated based on a camera parameter obtained by photographing the first viewpoint image. As an example, the estimated optical flow may be estimated based on deep learning. Also, the camera parameter may include a center position of a camera that has captured a multi-viewpoint image. Meanwhile, when estimating the depth image, coordinates in 3D space may be obtained for each pixel of the current view image by using the generated ray direction. Meanwhile, the coordinates may be obtained based on gradient descent using the minimum MSE. This is the same as described above with reference to other drawings.

According to the present disclosure, it is possible to estimate a more accurate depth image from the lens regardless of whether the photographing image is distorted, and the optical flow between images for a plurality of viewpoints can be obtained using images of peripheral viewpoints centered on the image of the viewpoint to be obtained. can be calculated In addition, since three-dimensional coordinate estimation based on this is possible, light rays generated from an optical flow can be optimized.

Various embodiments of the present disclosure do not list all possible combinations, but are intended to describe representative aspects of the present disclosure, and matters described in various embodiments may be applied independently or in combination of two or more.

In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In addition, it may be implemented by a combination of one or more software rather than one software, and one subject may not perform all processes. For example, a machine learning process that requires a high degree of data computing power and a large amount of memory is performed in the cloud or a server, and the user side may be implemented in a way that only uses a neural network in which machine learning has been completed, but it is self-evident that it is not limited thereto.

For implementation by hardware, one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), general purpose It may be implemented by a processor (general processor), a controller, a microcontroller, a microprocessor, and the like. For example, it may take various forms including the general-purpose processor. It is apparent that hardware may be disclosed in combination of one or more.

The scope of the present disclosure includes software or machine-executable instructions (eg, operating system, application, firmware, program, etc.) that cause an operation according to the method of various embodiments to be executed on a device or computer, and such software or and non-transitory computer-readable media in which instructions and the like are stored and executable on a device or computer.

Meanwhile, the contents described with reference to each drawing are not limited to each drawing, and as long as there is no contradiction, they may be applied complementary to each other.

Since the present disclosure described above can be various substitutions, modifications, and changes within the scope that does not depart from the technical spirit of the present disclosure for those of ordinary skill in the art to which the present disclosure pertains, the scope of the present disclosure is limited to the above It is not limited by one embodiment and the accompanying drawings.

Claims (1)

In the depth image estimation method using optical flow,
estimating an optical flow between the first viewpoint image and the second viewpoint image;
generating a ray direction for each pixel of the first viewpoint image based on the estimated optical flow; and
estimating a depth image of the first viewpoint image by using the generated ray direction;
The ray direction is expressed as a direction vector in a three-dimensional space, and the depth image is estimated based on a camera parameter for capturing the first viewpoint image.

Images