KR20180019329A - Depth map acquisition device and depth map acquisition method - Google Patents

Abstract

The present invention relates to a depth information acquisition apparatus which acquires depth information using a plurality of image information depending on a plurality of views, comprising: a feature point and corresponding point extracting unit for extracting a plurality of feature points from reference image information, and extracting a plurality of corresponding points corresponding to the feature points from each non-reference image information; and a camera parameter acquiring unit including a undistorted feature point and undistorted corresponding point acquiring unit and an estimated undistorted corresponding point acquiring unit. The undistorted feature point and undistorted corresponding point acquiring unit separately maps the feature points and the corresponding points to an undistorted image coordinate system from a distorted image coordinate system so as to acquire the undistorted feature points and the undistorted corresponding points. The estimated undistorted corresponding point acquiring unit acquires an estimated three-dimensional coordinate of the undistorted feature points, projects the estimated three-dimensional coordinate onto each of the non-reference image information, and acquires the estimated undistorted corresponding points. The camera parameter acquiring unit acquires a camera parameter minimizing a difference between the undistorted corresponding points and the estimated undistorted corresponding points.

Description

[0001] DEPTH MAP ACQUISITION DEVICE AND DEPTH MAP ACQUISITION METHOD [0002]

The present invention relates to a depth information obtaining apparatus and a depth information obtaining method.

The existing Structure-from-Motion (SfM) technology restores three-dimensional information of a scene from images with large changes in viewpoint. However, recently, research that acquires three-dimensional depth information from a plurality of images with minute viewpoint changes It is progressing.

Micro-baseline stereo (Technical report MSR-TR-2014-73, Microsoft Research, 2014) (hereafter referred to as "the paper 1") is used to approximate the camera viewpoint change in a minute three- dimensional space using homography transformation of two- And depth information was estimated by measuring the optical flow.

In the conventional SfM model, a rotation matrix (rotation matrix) is defined as a rotation angle of a camera, ), And the camera posture was estimated by modeling the three - dimensional coordinates of the feature points using inverse depth from the reference image. Acquisition of depth information was achieved by obtaining approximate depth information using plane sweeping and propagating using Markov Random Field.

High quality structure from small motion for rolling shutter cameras. (Hereinafter referred to as " IEEE International Conference on Computer Vision 2015 ") (hereinafter referred to as " Paper No. 3 ") discloses a method for expanding the method of Patent 1 and Method 2 to a rolling- The depth map is obtained by propagating the information using the color information of the reference image.

However, in the case of the paper 1, an error occurs naturally by approximating the three-dimensional camera motion in two dimensions, and the obtained depth information also has a large error. Patents 1, 2, and 3 have appropriately estimated three-dimensional camera motion through modeling using small angle approximation, but only when the intrinsic parameters of the camera are obtained through calibration As a possible technique, there is a great inconvenience that the user must perform camera calibration every time the user changes the camera focus. Also, there is a disadvantage that the result obtained from this is inaccurate (Patent 1 and Paper 2), the structure is too smooth and the distinction is poor (Thesis 3).

1. U.S. Patent Publication No. 8760500 (Apr. 24, 2014)

1. Micro-baseline stereo (Technical report MSR-TR-2014-73, Microsoft Research, 2014) 2. 3D Reconstruction from Accidental Motion (IEEE Computer Vision and Pattern Recognition 2014) 3. High quality structure from small motion for rolling shutter cameras. (IEEE International Conference on Computer Vision 2015)

A technical problem to be solved is to provide a depth information acquiring apparatus and a depth information acquiring method which can acquire camera parameters and depth information from a plurality of image information having a minute view change without performing a calibration separately by the user.

According to an aspect of the present invention, there is provided a depth information acquisition apparatus for acquiring depth information using a plurality of image information according to a plurality of viewpoints, the apparatus comprising: a plurality of feature points extraction means for extracting a plurality of feature points from reference image information, A feature point extracting unit for extracting a plurality of corresponding points corresponding to feature points, a camera parameter acquiring unit including a non-distortion corresponding point acquiring unit and an estimated non-distortion corresponding point acquiring unit. Wherein the non-distorted feature point and the ratio distortion corresponding point obtaining unit maps the plurality of feature points and the plurality of corresponding points respectively to an undistorted image coordinate system in a distorted image coordinate system to generate a plurality of non-distorted feature points and a plurality of non- Distortion corresponding point obtaining unit obtains the estimated three-dimensional coordinates of the plurality of non-distorted feature points, projects the estimated three-dimensional coordinates to each of the non-reference image information, And the camera parameter obtaining section may obtain a camera parameter that minimizes a difference between the plurality of non-distortion corresponding points and the plurality of estimated non-distortion corresponding points.

Wherein the depth information acquisition device maps each of the non-reference image information to the reference image coordinate system according to a plurality of inverse depth information to remove distortion from the plurality of image information using the camera parameter, A brightness profile obtaining unit for obtaining a plurality of brightness profiles for each inverse depth information of each pixel, and a plurality of brightness profiles for inverse depth information of each pixel to obtain specific inverse depth information of each pixel And an inverse depth information obtaining unit.

Wherein the depth information acquiring device comprises: an outlier depth information removing unit that removes specific inverse depth information exceeding a threshold from a plurality of specific inverse depth information for a plurality of pixels; Depth information to the depth information.

Wherein the feature point and corresponding point extracting unit extracts a plurality of backward correspondence points by tracing back the reference image information from the plurality of corresponding points and detects an anteropposite corresponding point having an interval greater than or equal to a threshold value from corresponding feature points of the plurality of backward corresponding points, The feature points corresponding to the anomaly inverse opposite points and the corresponding points can be removed from the plurality of feature points and the plurality of corresponding points.

Wherein the inverse depth information obtaining unit calculates a variance amount of the plurality of brightness profiles with respect to each inverse depth information of each pixel and outputs the inverse depth information having the smallest amount of variance to the specific inverse depth information .

Wherein when the ratio of the dispersion amount of the plurality of brightness profiles to the specific depth information to the average value of the plurality of brightness profiles with respect to the specific depth information is greater than the threshold value, Specific depth information can be removed.

The refiner may refer to color information of the reference image information to propagate inverse depth information on the pixel from which the specific inverse depth information is removed.

According to another aspect of the present invention, there is provided a depth information acquisition method for acquiring depth information using a plurality of image information according to a plurality of viewpoints, comprising: extracting a plurality of feature points from reference image information; Extracting a plurality of corresponding points corresponding to the feature points; mapping the plurality of feature points and the plurality of corresponding points from the distortion image coordinate system to the non-distortion image coordinate system to obtain a plurality of non-distorted feature points and a plurality of non- Obtaining the estimated three-dimensional coordinates of the plurality of non-distorted feature points, projecting the estimated three-dimensional coordinates to each of the non-reference image information to obtain a plurality of estimated non-distortion corresponding points, Obtaining a camera parameter that minimizes the difference of the plurality of estimated non-distorted corresponding points It includes.

Wherein the step of removing distortion from the plurality of image information using the camera parameter maps each of the non-reference image information to a reference image coordinate system according to a plurality of inverse depth information, Acquiring a plurality of brightness profiles for the inverse depth information, and obtaining specific inverse depth information of each pixel using the plurality of brightness profiles for each inverse depth information of each pixel.

The depth information obtaining method includes the steps of removing specific inverse depth information exceeding a threshold among a plurality of specific inverse depth information for a plurality of pixels and removing inverse depth information for a pixel from which the specific inverse depth information is removed And may further include a propagating step.

The step of extracting the plurality of corresponding points may include the steps of tracing back the reference image information from the plurality of corresponding points and extracting a plurality of backward corresponding points

Detecting an anomaly inverse correspondence point having an interval equal to or greater than a threshold value with corresponding feature points of the plurality of inverse correspondence points and removing the feature points corresponding to the anomaly inverse opposite point from the plurality of feature points and the plurality of correspondence points .

Wherein the step of acquiring specific inverse depth information of each pixel includes the steps of calculating a variance amount of the plurality of brightness profiles for each inverse depth information of each pixel, As specific inverse depth information.

Wherein the step of removing the specific inverse depth information exceeding the threshold value is performed such that the ratio of the variance amount of the plurality of brightness profile to the specific inverse depth information to the average value of the plurality of brightness profiles, If the threshold value is exceeded, the specific inverse depth information can be removed.

The step of propagating the inverse depth information may propagate inverse depth information on the pixel from which the specific inverse depth information is removed by referring to the color information of the reference image information.

The depth information acquiring apparatus and depth information acquiring method according to embodiments can acquire camera parameters and depth information from a plurality of image information having a minute view change without performing a calibration separately.

FIG. 1 is a view for explaining a depth information obtaining apparatus according to an embodiment of the present invention.
2 is a diagram for explaining the operation of the minutiae and the corresponding point extraction unit.
3 is a diagram for explaining the operation of the camera parameter obtaining unit.
4 to 6 are views for explaining a brightness profile obtaining unit and an inverse depth information obtaining unit.
Figs. 7 to 10 are views for explaining the abnormal depth information removing unit and the refinement unit. Fig.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The embodiments may be implemented in various different forms and are not limited to the embodiments described herein.

In order to clearly illustrate the embodiments, parts not related to the description are omitted, and the same reference numerals are used for the same or similar components throughout the specification. Therefore, reference numerals of the components used in the previous drawings can be used in the following drawings.

FIG. 1 is a view for explaining a depth information obtaining apparatus according to an embodiment of the present invention.

Referring to FIG. 1, a depth information obtaining apparatus 10 according to an embodiment of the present invention includes a feature point and corresponding point extracting unit 11, a camera parameter obtaining unit 14, a distortion removing unit 15, a brightness profile obtaining unit 16, an inverse depth information obtaining unit 17, an inverse depth information removing unit 18, and a refiner 19. The camera parameter obtaining section 14 includes non-distortion feature points and non-distortion correspondence point obtaining section 12 and estimated non-distortion correspondence point obtaining section 13. [

The depth information acquisition apparatus 10 acquires depth information using a plurality of image information corresponding to a plurality of viewpoints. The plurality of image information includes reference image information and remaining non-reference image information.

The depth information acquisition apparatus 10 may be a computing device capable of operating a program such as a desktop, a notebook, and a server. The depth information obtaining apparatus 10 may include at least one processor and a memory. The depth information acquisition method according to an embodiment of the present invention may be pre-programmed in the memory. The processor can drive such a program. On the other hand, such a program may be imported remotely from an external server or the like. On the other hand, the distinction between the processor and the memory may be ambiguous in the depth information acquiring apparatus 10, and it is possible for a person skilled in the art to determine the proportion of software and hardware for implementing the depth information acquiring method.

Although the depth information obtaining apparatus 10 may implement the depth information obtaining method of the present invention by one processor and one memory, the depth information obtaining apparatus 10 may use a plurality of processors and a plurality of memories, or a combination of various software and hardware, Can be implemented. 1, the depth information obtaining apparatus 10 includes a plurality of function units 11, 12, 13, 14, 15, 16, and 14 logically divided according to the depth information obtaining method, 17, 18, 19). It is optional for those skilled in the art to integrate or separate each functional unit into one or a specific number and implement it in hardware and software.

Hereinafter, the operation of the depth information obtaining apparatus 10 will be described with reference to FIGS.

2 is a diagram for explaining the operation of the minutiae and the corresponding point extraction unit.

The feature point and corresponding point extraction section 11 extracts a plurality of feature points from the reference image information and extracts a plurality of corresponding points corresponding to the plurality of feature points from each of the non-reference image information.

Also, in order to eliminate the erroneously estimated corresponding points, the feature point and corresponding point extracting section 11 traces back the reference image information from a plurality of corresponding points to extract a plurality of backward corresponding points, And the feature points and the corresponding points corresponding to the anomaly inverse opposite points can be removed from the plurality of feature points and the plurality of corresponding points.

Hereinafter, the process described above with reference to FIG. 2 will be described in detail. For convenience of explanation, description will be made on the basis of any one feature point 201 and any one non-reference image information 210 of the reference image information 200, rather than the entire feature point and the entire non-reference image information.

First, the minutiae and corresponding point extraction section 11 extracts Harris corners from the reference image information 200 and acquires them as a plurality of minutiae. The plurality of feature points include any one feature point 201. [ A KLT tracker (Kanade-Lucas-Tomashi tracker) is applied to the obtained plurality of feature points to forwardly track each of the non-reference image information, thereby extracting a plurality of corresponding points for each of the non-reference image information. For example, the corresponding point 211 corresponding to the feature point 201 may be obtained for the non-reference image information 210. [

The minutiae and corresponding point extraction unit 11 can obtain a reverse correspondence point 202 corresponding to the corresponding point 211 in the reference image information 200 by applying the KLT tracker to the corresponding point 211 in the reverse direction. The feature point and corresponding point extraction unit 11 can determine the outward correspondence point 202 as the outlier correspondence point if the distance d between the feature point 201 and the corresponding point 202 is equal to or larger than the threshold value. When the backward corresponding point 202 is determined to be an anomalous backward corresponding point, the feature point 201 and the corresponding point 211 are judged to be erroneously estimated, so that they can be removed from a plurality of feature points and a plurality of corresponding points, respectively.

Where the interval d may be in pixels and the threshold may be 0.1 pixels. The specific values of the threshold value can be selected by those skilled in the art.

Through the above-described process, the minutiae and corresponding point extraction section 11 can acquire a plurality of minutiae and a plurality of corresponding points.

3 is a diagram for explaining the operation of the camera parameter obtaining unit.

First, the camera internal parameter to be acquired includes one focal length f and two radial distortion parameters k ₁ and k ₂ , and the camera's optical point of interest and the center of the radial distortion Is assumed to be at the center of the image.

을 왜곡 영상 좌표계에서 비왜곡 영상 좌표계로 변환하는 함수

는 아래의 예시적인 수학식 1에 따를 수 있다.The non-distorted feature point and non-distorted corresponding point obtaining unit 12 maps a plurality of feature points and a plurality of corresponding points to an undistorted image coordinate system in a distorted image coordinate system to obtain a plurality of non- The non-distortion correspondence point of? Any point

A function of transforming a distortion image coordinate system into a non-distortion image coordinate system

Lt; RTI ID = 0.0 > (1) < / RTI >

[Equation 1]

로 표현할 수 있다. 본 실시예에서 i=0일 때의 영상 정보를 기준 영상 정보로 한다. 왜곡 영상 좌표계의 좌표

를 비왜곡 영상 좌표계의 좌표로 변환하면

로 표현할 수 있다.i is the number of the image information, and j is the number of the feature point, the coordinates of the distortion image coordinate system are represented by

. In the present embodiment, the video information when i = 0 is set as reference video information. Coordinates of the distortion image coordinate system

Into coordinates of the non-distorted image coordinate system

.

Through the above calculation process, the non-distorted feature point and non-distorted corresponding point obtaining unit 12 obtains the non-distorted feature point 201u with respect to the reference image information 200u, (211u).

Next, the estimated non-distorted corresponding point obtaining section 13 obtains the estimated three-dimensional coordinates of the plurality of non-distorted feature points, projects the estimated three-dimensional coordinates onto each of the non-reference image information, Non-distortion correspondence point.

3, an estimated three-dimensional coordinate 301 of the non-distorted feature point 201u is obtained, and the estimated three-dimensional coordinate 301 is projected to the non-reference image information 210u to obtain an estimated non-distortion corresponding point 211u ' Can be obtained.

는 기준 영상 정보(200u)의 비왜곡 특징점(201u)의 좌표

와 역 깊이(inverse depth) 정보

를 이용하여 아래 수학식 2와 같이 모델링할 수 있다. 이때 역 깊이 정보

는 추정된 삼차원 좌표(301)

자체의 역 깊이 정보이다.The estimated three-dimensional coordinate (301)

Distortion feature point 201u of the reference image information 200u

And inverse depth information

The following equation (2) can be used. At this time,

Dimensional coordinates 301,

Depth information of its own.

&Quot; (2) "

를 비기준 영상 정보(210u)에 투사하여 추정된 비왜곡 대응점(211u')인

을 획득한다. 이때 사용되는 함수

는 아래의 예시적인 수학식 3에 따를 수 있다.The estimated non-distortion corresponding point obtaining section 13 obtains the estimated three-dimensional coordinate 301

Is projected on the non-reference image information 210u to be the estimated non-distortion corresponding point 211u '

. Functions used at this time

Lt; RTI ID = 0.0 > (3) < / RTI >

&Quot; (3) "

와 같이 계산될 수 있다. 함수

는 추정된 삼차원 좌표(301)인

를 회전 벡터

및 변위 벡터

를 갖는 i번째 영상 정보(여기서 비기준 영상 정보(210u))에 투사시키는 함수이다. 이때,

에 대응되는 작은 각도 근사된 회전 행렬(small angle approximated rotation matrix)은 함수

로 표현하였으며, 아래의 예시적인 수학식 4와 같이 정의될 수 있다.At this time,

Can be calculated as follows. function

Dimensional coordinate 301

Rotation vector

And a displacement vector

To the ith image information (here, the non-reference image information 210u). At this time,

A small angle approximated rotation matrix corresponding to < RTI ID = 0.0 >

And can be defined by the following exemplary equation (4).

&Quot; (4) &quot;

는

의 구성요소들이고,

는 벡터-매트릭스 전화 함수(vector-to-matrix function)로서 회전 벡터

를 작은 각도 근사된 회전 행렬로 변환시킨다.

The

&Lt; / RTI &gt;

Is a vector-to-matrix function,

To a small angle-approximated rotation matrix.

Next, the camera parameter obtaining section 14 obtains camera internal parameters and camera external parameters that minimize the difference between the plurality of non-distortion corresponding points and the plurality of estimated non-distortion corresponding points, using the following exemplary equation (5) can do.

&Quot; (5) &quot;

에 초점 거리

가 곱해지는 것은 단위를 픽셀 단위로 변환하기 위해서이다.

는 후버 손실 함수(Huber loss function)이고,

는 카메라 내부 파라미터의 집합이고,

은 비기준 영상들에 대한 카메라 회전 벡터들의 집합이고,

는 비기준 영상들에 대한 카메라 변위 벡터들의 집합이고,

는 복수의 특징점들의 역 깊이 정보들의 집합이다. N은 복수의 영상 정보의 개수이고, M은 복수의 특징점의 개수이다.argmin can be a function of nonlinear optimization technique such as LM method (Levenberg-Marquardt method). The estimated non-distortion corresponding point 211u '

Focus distance on

The multiplication is to convert the units into pixels.

Is a Huber loss function,

Is a set of camera internal parameters,

Is a set of camera rotation vectors for non-reference images,

Is a set of camera displacement vectors for non-reference images,

Is a set of inverse depth information of a plurality of feature points. N is the number of the plurality of image information, and M is the number of the plurality of feature points.

The camera parameter acquiring unit 14 can acquire the camera internal parameter K, the camera external parameters R and T, and W using Equation (5). Distortion feature point and non-distortion correspondence point acquisition section 12 refers to K, R, T, and W estimated at the present point in the calculation process of camera parameter acquisition section 14 to obtain a plurality of non-distortion feature points and a plurality of non- Distortion correspondence point, and the estimated non-distortion correspondence point obtaining section 13 updates the plurality of estimated non-distortion correspondence points.

4 to 6 are views for explaining a brightness profile obtaining unit and an inverse depth information obtaining unit.

First, before operation of the brightness profile obtaining section 16, the distortion removing section 15 may operate. The distortion removing unit 15 may remove distortion from a plurality of pieces of image information using camera internal parameters. Accordingly, the brightness profile obtaining unit 16 can operate using a plurality of pieces of image information from which distortion has been removed.

The brightness profile obtaining unit 16 may obtain a plurality of brightness profiles for each inverse depth information of each pixel by mapping each of the non-reference image information to a reference image coordinate system according to a plurality of inverse depth information. At this time, the brightness profile obtaining unit 16 may map each of the non-reference image information to the reference image coordinate system according to the plurality of inverse depth information using the obtained camera parameters.

은 아래의 예시적인 수학식 6으로 표현될 수 있다. 본 실시예에서 역 깊이 w_k의 k는

로 가정한다. 즉 256개의 가상 평면이 있는 것으로 가정한다.Specifically, assuming a plurality of virtual planes spaced by the inverse depth w _k from the reference image information 410 and moving the non-reference images to the reference image coordinate system through homography transformation to pass through these virtual planes . The homography matrix used at this time

Can be expressed by the following exemplary equation (6). In this embodiment, _k of the inverse depth w k is

. That is, there are 256 virtual planes.

&Quot; (6) &quot;

는 카메라 내부 행렬(intrinsic matrix)로,

이며, (p_u,p_v)는 이미지 중심 좌표이다. 수학식 6의 변수들은 이미 획득된 카메라 파라미터를 이용할 수 있다.

Is a camera intrinsic matrix,

(P _u , p _v ) is the image center coordinate. The variables in equation (6) can use the already obtained camera parameters.

에 대한 N 개의 픽셀 값의 집합을 밝기 프로파일(intensity profile)이라고 할 때, 좌표

인 픽셀의 역 깊이 정보 w_k에 대한 밝기 프로파일인

는 아래 수학식 7과 같이 표현할 수 있다.If all of the non-reference image information and reference image information moved to the reference image coordinate system are overlapped, each pixel has N pixel values from N image information for each depth depth. location

And a set of N pixel values for a pixel is called an intensity profile,

Lt; _{RTI ID = 0.0} &gt; _wk &lt; / _RTI &gt;

Can be expressed as Equation (7) below.

&Quot; (7) &quot;

는 i번째 영상 정보를 k 번째 가상 평면을 통해 기준 영상 좌표계로 변환시켰을 때, 좌표

의 픽셀 값을 의미한다. 이때 k 번째 가상 평면의 깊이가 좌표가

인 픽셀에 해당하는 삼차원 공간 상의 지점의 깊이와 유사한 경우, 밝기 프로파일

은 일관적인 값들을 갖게 된다.here

When the i-th image information is converted into the reference image coordinate system through the k-th virtual plane,

. In this case, the depth of the kth virtual plane is

If it is similar to the depth of a point on a three-dimensional space corresponding to an in-pixel,

Will have consistent values.

Referring to FIG. 5, a brightness profile 421 corresponding to each inverse depth information w _k is shown graphically for three exemplary pixels (P2, P2, P3) of the reference image information 410 of FIG. 4 . Also shown is a graph of a horizontal gradient 422 and a vertical gradient 423 of the brightness profile.

The inverse depth information obtaining unit 17 can obtain specific inverse depth information of each pixel by using a plurality of brightness profiles for each inverse depth information of each pixel. At this time, the inverse depth information obtaining unit 17 calculates a variance amount of a plurality of brightness profiles with respect to each inverse depth information of each pixel, and outputs inverse depth information having the smallest amount of variance to specific inverse depth information . Here, the amount of dispersion may include a variance value of the brightness profile, a variance value of the brightness gradient of the brightness profile, and a variance value of the brightness gradient of the brightness profile.

라고 하고, 밝기 프로파일의 분산 값을

라고 하고, 가로 방향 그래디언트의 분산값을

라고 하고, 세로 방향 그래디언트의 분산값을

라고 한다면, 아래 수학식 8과 같이 표현할 수 있다.

,

는 가로 방향 및 세로 방향 그래디언트 각각의 가중치 값으로서 당업자에 의해 적절히 선택될 수 있다.The amount of dispersion

And the variance value of the brightness profile is

And the variance value of the horizontal gradient is

And the variance value of the longitudinal gradient is

, It can be expressed as Equation (8) below.

,

May be appropriately selected by those skilled in the art as the weight values of the respective horizontal and vertical gradients.

&Quot; (8) &quot;

는 아래 수학식 9와 같이 표현될 수 있다. VAR는 분산(variance)을 구하는 함수이다.The variance value of the brightness profile

Can be expressed by Equation (9) below. VAR is a function to obtain variance.

&Quot; (9) &quot;

는 아래 수학식 10과 같이 표현될 수 있다.The variance of the horizontal gradient of the brightness profile.

Can be expressed as Equation (10) below.

&Quot; (10) &quot;

는 아래 수학식 11과 같이 표현될 수 있다.The variance of the vertical gradient of the brightness profile.

Can be expressed by Equation (11) below.

&Quot; (11) &quot;

이 가장 낮은 역 깊이 정보를 해당 픽셀에 대한 특정 역 깊이 정보로서 획득할 수 있다. 도 5를 참조하면, 픽셀(P1)의 경우 k=a 일때 그래프(426b)의 분산량이 최소이므로 픽셀(P1)에 대한 특정 역 깊이 정보 w_a가 획득된다. 픽셀(P2)의 경우 k=b 일때 그래프(427b)의 분산량이 최소이므로 픽셀(P2)에 대한 특정 역 깊이 정보 w_b가 획득된다. 픽셀(P3)의 경우 k=c 일때 그래프(428b)의 분산량이 최소이므로 픽셀(P3)에 대한 특정 역 깊이 정보 w_c가 획득된다. k=a일 때 픽셀(P1)이 가장 선명하고(426a), k=b일 때 픽셀(P2)이 가장 선명하고(427a), k=c일 때 픽셀(P3)이 가장 선명한 것(428a)을 확인할 수 있다.The inverse depth information obtaining unit 17 calculates the inverse depth information

This lowest inverse depth information can be obtained as specific inverse depth information for that pixel. Referring to FIG. 5, the specific depth information w _a for the pixel P ₁ is obtained since the variance amount of the graph 426 b is minimum when k = a for the pixel P 1. For pixel P2, the specific depth information w _b for pixel P2 is obtained since the variance of the graph 427 b is at a minimum when k = b. For the pixel P3, the specific depth information w _c for the pixel P3 is obtained since the variance amount of the graph 428b is minimum when k = c. the pixel P1 is the most vivid when k = a 426a and the pixel P2 is the most vivid 427a when k = b and the pixel P3 is the most vivid 428a when k = can confirm.

라고 하면, 아래의 예시적인 수학식 12와 같이 표현할 수 있다.The above-described process is a process of extracting specific inverse depth information

, It can be expressed as the following exemplary equation (12).

&Quot; (12) &quot;

Referring to FIG. 6, a depth map 430 using a plurality of specific depth information obtained for a plurality of pixels is shown. Therefore, the depth information obtaining apparatus 10 of the present invention can confirm that camera parameters and depth information can be obtained from a plurality of image information having a change in the micro-viewpoint without performing a calibration separately by the user through the above-described process .

Figs. 7 to 10 are views for explaining the abnormal depth information removing unit and the refinement unit. Fig.

Referring again to FIG. 6, it can be seen that the smoothness of the edge of the depth map 430 is reduced due to outlier depth information. FIG. 7 shows a rough depth map 510 that has undergone a process similar to that of FIG. The enlarged portion 511b of the portion 511a of the depth map 510 and the enlarged portion 512b of the portion 512a will be referred to.

The outlier depth information removing unit 18 can determine the specific inverse depth information exceeding the threshold among the plurality of specific inverse depth information for the plurality of pixels as the abnormal inverse depth information and remove the abnormal inverse depth information.

의 특정 역 깊이 정보에 대한 복수의 밝기 프로파일의 평균값

에 대한 비율

이 임계치를 초과하는 경우, 초과한 특정 역 깊이 정보를 이상 역 깊이 정보로 판단하고 이를 제거할 수 있다. 비율

은 아래 수학식 13과 같이 표현할 수 있다.Specifically, the anomaly depth information removing unit 18 calculates the amount of variance of the plurality of brightness profiles with respect to the specific inverse depth information

The average value of the plurality of brightness profiles for the specific depth information

Percentage to

If the threshold value is exceeded, the specific inverse depth information is judged to be abnormal inverse depth information and can be removed. ratio

Can be expressed by Equation (13) below.

&Quot; (13) &quot;

들 중 상위 10 퍼센트에 해당하는 값일 수 있다. 즉, 비율

이 지나치게 큰 경우 해당하는 특정 역 깊이 정보를 이상 역 깊이 정보로 판단하고 이를 제거할 수 있다. 이러한 임계치는 당업자의 선택사항일 수 있다. 도 8를 참조하면, 이상 역 깊이 정보가 제거된 깊이 맵(520, 521, 522)이 도시된다.The threshold is the ratio of the total image information

Of the top 10 percent. That is,

If the depth information is too large, the corresponding specific depth information can be determined as the abnormal depth information and can be removed. These thresholds may be optional to those skilled in the art. Referring to FIG. 8, depth maps 520, 521 and 522 are shown in which the outlier depth information is removed.

The refinement section 19 may propagate inverse depth information for a pixel from which specific inverse depth information has been removed from a plurality of pixels. Specifically, referring to the color information of the reference image information 540, the inverse depth information may be propagated to the pixel from which the specific inverse depth information has been removed. Referring to FIG. 9, a smoother depth map 530, 531, 532 is shown filled with propagated inverse depth information.

It is to be understood that both the foregoing general description and the following detailed description of the present invention are illustrative and explanatory only and are intended to be illustrative of the invention and are not to be construed as limiting the scope of the invention as defined by the appended claims. It is not. Therefore, those skilled in the art will appreciate that various modifications and equivalent embodiments are possible without departing from the scope of the present invention. Accordingly, the true scope of the present invention should be determined by the technical idea of the appended claims.

10: Depth information acquisition device
11: minutiae and corresponding point extraction section
12: non-distortion characteristic point and non-distortion correspondence point acquisition unit
13: Estimated non-distortion corresponding point obtaining unit
14: Camera parameter acquisition unit
15: Distortion elimination
16: Brightness profile acquisition unit
17: Inverse depth information acquisition unit
18: Abnormal depth information removal
19: Purification section

Claims (14)

A depth information acquiring apparatus for acquiring depth information using a plurality of image information according to a plurality of viewpoints,
A feature point extracting unit that extracts a plurality of feature points from the reference image information and extracts a plurality of corresponding points corresponding to the plurality of feature points from each of the non-reference image information; And
A non-distorted corresponding point obtaining unit, and an estimated non-distortion corresponding point obtaining unit,
The non-distorted feature point and non-distorted corresponding point obtaining unit maps the plurality of feature points and the plurality of corresponding points respectively to an undistorted image coordinate system in a distorted image coordinate system to generate a plurality of non- Acquires a non-distortion correspondence point,
The estimated non-distorted corresponding point obtaining unit obtains estimated three-dimensional coordinates of the plurality of non-distorted feature points, projects the estimated three-dimensional coordinates to each of the non-reference image information, calculates a plurality of estimated non- Lt; / RTI &gt;
Wherein the camera parameter obtaining section obtains a camera parameter that minimizes a difference between the plurality of non-distortion corresponding points and the plurality of estimated non-
Depth information acquisition device. The method according to claim 1,
A distortion removing unit for removing distortion from the plurality of image information using the camera parameters;
A brightness profile obtaining unit that maps each of the non-reference image information to a reference image coordinate system according to a plurality of inverse depth information to obtain a plurality of brightness profiles for each inverse depth information of each pixel; And
And an inverse depth information obtaining unit for obtaining specific inverse depth information of each pixel using the plurality of brightness profiles for each inverse depth information of each pixel
Depth information acquisition device. 3. The method of claim 2,
An abnormal depth information removing unit that removes specific inverse depth information exceeding a threshold among a plurality of specific inverse depth information for a plurality of pixels; And
Further comprising a refinement section for propagating inverse depth information on a pixel from which the specific inverse depth information has been removed
Depth information acquisition device. The method according to claim 1,
The feature point and corresponding point extraction unit
Extracting a plurality of backward corresponding points by tracing back the reference image information from the plurality of corresponding points,
Detecting an anomaly inverse point having an interval greater than or equal to a threshold value from the corresponding minutiae of the plurality of inverse correspondence points,
And removing, from the plurality of feature points and the plurality of corresponding points, feature points and corresponding points corresponding to the anomaly-
Depth information acquisition device. 3. The method of claim 2,
The inverse depth information obtaining unit
Calculating a variance amount of the plurality of brightness profiles for each inverse depth information of each pixel,
Acquiring the inverse depth information having the lowest amount of dispersion as the specific inverse depth information for the pixel,
Depth information acquisition device. The method of claim 3,
The abnormal depth information removing unit
Depth information of the plurality of brightness profiles to the average value of the plurality of brightness profiles with respect to the specific depth information of the dispersion amount of the plurality of brightness profiles with respect to the specific depth depth information exceeds the threshold value,
Depth information acquisition device. The method of claim 3,
The purification unit
Depth information on a pixel from which the specific inverse depth information has been removed by referring to color information of the reference image information,
Depth information acquisition device. A depth information acquiring method for acquiring depth information using a plurality of image information according to a plurality of viewpoints,
Extracting a plurality of feature points from the reference image information and extracting a plurality of corresponding points corresponding to the plurality of feature points from each of the non-reference image information;
Obtaining a plurality of non-distorted feature points and a plurality of non-distorted corresponding points by mapping the plurality of feature points and the plurality of corresponding points, respectively, from a distortion image coordinate system to an unoriented image coordinate system;
Obtaining the estimated three-dimensional coordinates of the plurality of non-distorted feature points, projecting the estimated three-dimensional coordinates to each of the non-reference image information to obtain a plurality of estimated non-distortion corresponding points; And
Obtaining a camera parameter that minimizes the difference between the plurality of non-distorted corresponding points and the plurality of estimated non-distorted corresponding points
Depth information acquisition method. 9. The method of claim 8,
Removing distortion from the plurality of image information using the camera parameters;
Mapping each of the non-reference image information to a reference image coordinate system according to a plurality of inverse depth information to obtain a plurality of brightness profiles for each inverse depth information of each pixel; And
Further comprising obtaining the specific inverse depth information of each pixel using the plurality of brightness profiles for each inverse depth information of each pixel
Depth information acquisition method. 10. The method of claim 9,
Removing specific inverse depth information that exceeds a threshold among a plurality of specific inverse depth information for a plurality of pixels; And
Further comprising propagating inverse depth information for a pixel from which the specific inverse depth information has been removed
Depth information acquisition method. 9. The method of claim 8,
Wherein the extracting of the plurality of corresponding points comprises:
A step of tracing back the reference image information from the plurality of corresponding points to extract a plurality of backward correspondence points;
Detecting an anomaly opposite point having an interval equal to or greater than a threshold value with corresponding minutiae of the plurality of reverse counterparts; And
And removing, from the plurality of feature points and the plurality of corresponding points, feature points and corresponding points corresponding to the abnormal backward corresponding points.
Depth information acquisition method. 10. The method of claim 9,
Wherein acquiring specific inverse depth information of each pixel comprises:
Calculating a variance amount of the plurality of brightness profiles for each inverse depth information of each pixel; And
And obtaining the inverse depth information having the smallest variance as the specific inverse depth information for the pixel
Depth information acquisition method. 11. The method of claim 10,
Wherein removing the specific inverse depth information exceeding the threshold comprises:
Depth information of the plurality of brightness profiles to the average value of the plurality of brightness profiles with respect to the specific depth information of the dispersion amount of the plurality of brightness profiles with respect to the specific depth depth information exceeds the threshold value,
Depth information acquisition method. 11. The method of claim 10,
Wherein propagating the inverse depth information comprises:
Depth information on the pixel from which the specific inverse depth information has been removed by referring to the color information of the reference image information,
Depth information acquisition method.

Images