KR101027004B1 - Apparatus and method for 3d estimating fusing optical flow and stereo matching - Google Patents

Abstract

A three-dimensional structure estimating apparatus that combines an optical flow and stereo matching according to an embodiment of the present invention includes a register storing a binocular difference and an optical flow for a parent node (t-1) of left and right input images input at time t And a time update for calculating a binomial difference prediction value of the time t estimated by the parent node based on the left image and optical flow values of the left and right images of the parent node, And a measurement value updater for updating the register by calculating binocular disparity and optical flow for the left and right input images using the optical flow value, the binocular disparity value, and the corresponding points in the parent node calculated by the time update unit.

As described above, the present invention estimates the binocular difference value and the corresponding points of the current frame using the motion and frame of the previous time, which is the parent node, and calculates the actual correct corresponding points using the current frame and the estimated corresponding point and binocular difference prediction value Thereby making it possible to reduce the search range of the optical flow and the binocular, as well as improve the computation speed and accuracy.

Binocular, optical flow, stereo matching

Description

FIELD OF THE INVENTION [0001] The present invention relates to an apparatus and method for estimating a three-dimensional structure that combines optical flow and stereo matching,

The present invention relates to a three-dimensional structure estimation that combines optical flow and stereo matching, and more particularly, to a method and apparatus for estimating a three-dimensional structure by combining optical flow and stereo matching, Dimensional structure estimating apparatus and method.

In general, the human visual system includes two eyes, which allow us to know the distance and distance information of things through the two eyes. When applied to a camera image, two images of a certain space can be used to obtain distance information.

Obtaining distance information through the image is the ultimate goal of stereo matching in the visual system. One image is a projection of a three-dimensional space into a two-dimensional space. In this process, the distance information is lost. Therefore, it is difficult to reconstruct a three-dimensional space from a single image.

However, when there are two images obtained from different positions on the object, it becomes possible to reconstruct the 3D space. If a point on the actual space is formed on two images, the position of the point on the real three-dimensional space can be found by using the information of the two images and the simple geometry. Here, a point formed in the two-dimensional image in the three-dimensional space is referred to as a corresponding point of each of the left and right images, and the process of finding corresponding points on the basis of the left or right image is referred to as stereo matching.

The disparity of two images can be obtained through stereo matching. That is, when two cameras are on the same plane and the optical axes are parallel to each other, the disparity of the two images is obtained by dividing the left and right images Can be calculated by the difference of the corresponding points.

On the other hand, an optical flow is required to restore a three-dimensional space based on left and right images provided by two cameras. Here, a light flow is a velocity distribution of a motion in an image generated by gradually changing a lightness type , That is, a change value of an image with time.

Conventional techniques for reconstructing a three-dimensional space by combining optical flow and stereo matching techniques calculate the light flow and the binocular by using an inaccurate relationship between each other. One prior art technique calculates light flow using feature points of images sequentially in time, and uses the calculated light flow value as information on a moving component of stereo matching.

Other prior art techniques take advantage of the interrelationships between discontinuous characteristics of the binocular and light flow, and use the relationship between discontinuity and occlusion in stereo matching techniques as the limiting term for binocular disparity calculations.

Conventional optical flow and binocular difference calculation techniques use only one cue to guess the information of the opponent cue instead of using mutually strong constraint terms and do not take advantage of restrictions on geometric properties However, there is a problem that is not calculated in the time characteristic region.

The present invention estimates the binocular difference value and the corresponding points of the current frame using the motion and frame of the previous time which is the parent node and calculates the correct corresponding points using the current frame and the corresponding corresponding point and the binocular prediction value, Dimensional structure estimating apparatus and method that can combine optical flow and stereo matching, which can reduce the search range of an image, and improve computation speed and accuracy.

The objects of the present invention are not limited to those mentioned above, and other objects not mentioned may be clearly understood by those skilled in the art from the following description.

A three-dimensional structure estimating apparatus that combines an optical flow and stereo matching according to an embodiment of the present invention includes a register storing a binocular difference and an optical flow for a parent node (t-1) of left and right input images input at time t A pairwise predicted value of the time t estimated by the parent node on the basis of the left image and a corresponding optical flow value of the left and right images of the parent node, A binomial difference and an optical flow for the left and right input images are calculated using the optical flow value, the binocular difference predicted value, and the corresponding points in the parent node calculated by the time update unit, And updates the measured value.

The measurement value updating unit of the three-dimensional structure estimating apparatus that combines the optical flow and the stereo matching according to the embodiment of the present invention minimizes the difference between the pixels of the left and right input images existing on the same line on the basis of the input left image A binocular difference which minimizes a difference between the binocular difference predictive value and a binocular bin which can be selected at the time t and a binocular binocular binarization which calculates a binocular difference of the time t based on a discontinuity penalty to a previous pixel in any pixel A difference between a binocular difference of the specific pixel at the time t and a binocular difference value of the binocular difference predicted by the time update unit at the time t; And a light flow update module for calculating the light flow at the time t using the pixel position at which the difference becomes minimum The.

The three-dimensional structure estimation method of combining the optical flow and the stereo matching according to the embodiment of the present invention in another aspect estimates corresponding points of the left and right input images of time t from the optical flow of the pre-stored time t-1, Estimating the estimated binaural difference value of the time t from the estimated corresponding points and the optical flow at the time t-1; and estimating the optical flow at the estimated time t-1, And calculating a binocular difference and an optical flow value at the time t using the estimated corresponding point.

In the embodiment of the present invention, the binocular difference prediction value of the time t is a binocular difference predicted value of the time t estimated at the time t-1 based on the left input image in the left and right input frames.

The calculating of the binocular disparity and the optical flow value at the time t in the embodiment of the present invention may include calculating the binocular disparity and the optical flow value of the first binocular Searching for a second binocular difference that minimizes a difference between the binocular difference predictive value and a binocular difference that can be selected at the time t; And calculating a discontinuous penalty value to the binocular at the time t by minimizing the energy function composed of the first binocular, the second binocular, and the penalty value.

In the embodiment of the present invention, the step of calculating the optical flow value at the time t may include the steps of: calculating a brightness difference value between left and right images for a neighboring pixel of a specific pixel in the left or right input image; Finding a position of a pixel at which a difference between a binocular difference and a binocular prediction value of a specific pixel is minimized; and calculating a value for minimizing the energy function by using the brightness difference value and the position of the pixel as an energy function, And calculating the optical flow value.

In the present invention, binocular difference predictions and corresponding points of a current frame are estimated using motion and frames of a previous time, which is a parent node, and by calculating actual correct correspondences using the current frame and an estimated corresponding point and a binocular difference prediction value, The search range of the flow and the binocular can be reduced, and the operation speed and accuracy can be improved.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, features and advantages of the present invention will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: FIG. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. The following terms are defined in consideration of the functions of the present invention, and these may be changed according to the intention of the user, the operator, or the like.

The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. These embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art to which the present invention pertains. Only. Therefore, the definition should be based on the contents throughout this specification.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

In the embodiment of the present invention, the binocular difference value and the corresponding points of the current frame are estimated using the motion and frame of the previous time, which is the parent node, and the actual corresponding points are calculated using the current frame and the estimated corresponding point and binocular difference prediction value A three-dimensional structure estimating apparatus and method combining optical flow and stereo matching will be described.

Prior to the description of the embodiments of the present invention, the optical flow in the present invention means a change value of an image according to time, that is, time t, t-1, t-2 , ..., ... ,,,,,,.

The light flow for a normal left / right image, i.e., an image photographed by left and right cameras, is defined by the following equation (1).

는 좌,우 영상에 대한 각각의 대응점들의 광 흐름을 나타내며, u,v는 각각 수평, 수직 성분을 표현한다. 또한

은 양안차 d 의 시간의 흐름에 따라 유도된 값을 의미한다.here

Represents the light flow of the corresponding points to the left and right images, and u and v represent the horizontal and vertical components, respectively. Also

Means a value derived in accordance with the flow of time of the binocular difference d .

에 의해 아래의 수학식2와 같이 표현되며, 이를 광 흐름-양안차 식으로 불린다. 즉, 시간에 따 라 발생된 광 흐름에 대해 미분을 했을 경우 양안차와 동일한 값을 갖는다.The relationship of the optical flow-binocular disparity derived by the present invention is expressed by Equation

Is expressed by the following equation (2), which is called optical flow-binocular equation. That is, if the light flow generated according to the time is differentiated, it has the same value as the binocular difference.

In Equation (2), F is a fundamental matrix, and x _l and x _r denote corresponding points of the left and right images, respectively.

At time t-1, pairs of corresponding points are calculated by the pixels of all the left and right images, and the parent nodes at time t-2 for all the pixels are calculated as valid values for the calculation of the estimates at time t-1 . Here, the parent node refers to a corresponding point at a previous time (t-1) corresponding to a time t for images sequential in time.

1 is a block diagram illustrating an apparatus for estimating a 3D structure using an optical flow and a stereo matching technique according to an embodiment of the present invention.

1, an apparatus for estimating a three-dimensional structure using an optical flow and a stereo matching technique according to an embodiment of the present invention includes left and right cameras 100 and 110, a measurement value update unit 120, a time update unit 130, And a register 140.

The corresponding points for the binocular and optical flows are defined as respective states, and if the calculated binocular and optical flow values are defined as the output of the three-dimensional structure estimator, it is stored in the register 140 do.

In the present invention, each state and result value are defined by the following Equation (3).

states :

states:

output:

는 이전 시간(t-2)에 위치한 x^l, x^r의 부모 노드를 나타내는데, 즉 이전 시간에 위치한 프레임 값을 의미한다. 아래 첨자 t-1은 시간 t-1에서 얻은 값들을 의미한다.In Equation (3)

^Represents the parent node of x ^l , x ^r located at the previous time (t-2), that is, the frame value located at the previous time. The subscript t-1 means the values obtained at time t-1.

)으로부터 시간 t의 대응점들(

)의 값을 추정한다. 이를 시간-업데이트 스테이지(time-update stage)라고 정의한다. Before estimating the output value from the new frame (left and right images) f ¹ , f ^r at time t, the time update unit 130 updates the motion

) To the corresponding points of time t

). ≪ / RTI > This is defined as a time-update stage.

The measured value update unit 120 obtains the estimated binocular difference and the optical flow of the time t using the output of the time update unit 130 and the new frames f ^l and f ^{r of} time t, 140).

)으로부터 추정된 시간 t의 대응점들(

)의 값과 시간 t의 추정된 양안차 및 광 흐름은 아래의 수학식 4에 의해 계산된다.In this way, the time update unit 130 updates the motion of the time t-

) Corresponding to the estimated time t (

) And the estimated binocular and optical flow of time t are calculated by the following equation (4).

는 시간 t-1의 모션(

)으로부터 추정된 시간 t의 좌우 영상에서 대응점을 의미하며,

는 좌측 영상을 기준으로 시간 t-1에서 추정된 시간 t의 양안차 값(예측 값)이다. 또한

는 시간 t-1에서 좌우측 영상 각각에서 계산된 시간 t-1 프레임의 모션(광 흐름) 값을 의미한다.In Equation (4)

Lt; RTI ID = 0.0 > t-1 < / RTI &

) Corresponding to the time t,

(Predicted value) of the time t estimated at time t-1 on the basis of the left image. Also

(Light flow) value of the time t-1 frame calculated at each of the left and right images at time t-1.

The measurement value update unit 120 updates the binarization and motion (light flow) values at time t using the output of the time update unit 130 and the left and right input images input from the left and right cameras 100 and 110 at time t . Here, the motion, the binocular difference predicted value and the estimated motion at time t-1 calculated by the time update unit 130 are used as a cue for finding a new accurate motion and an eye-difference value.

)은 일반적인 스테레오 매칭기술에 더하여 시간 t에서 더 정확한 양안차

를 얻을 수 있으며, 모션의 대응점들(

)은 상기의 수학식 4에 의해 결정된다. 최적의 대응점은 반드시 앞선 프레임(앞선 시간의 영상)에서 얻은 추정된 위치로부터 가까운 곳에 존재하기 때문에 최적의 결과를 얻기 위해 수학식 1을 만족하는 점들의 주위를 검색한다. 즉, 도 2에 도시된 바와 같이, 최적의 대응점을 찾기 위해서 양안차 검색 범위와 광 흐름 검색 범위를 설정한 후 설정된 범위를 검색한다.The binaural difference predicted value at the time t-1 estimated by the time update unit 130

), In addition to the usual stereo matching technique,

And the corresponding points of motion (

) Is determined by Equation (4) above. Since the optimal correspondence point is located close to the estimated position obtained in the preceding frame (image of the preceding time), it searches around the points satisfying the equation (1) to obtain the optimum result. That is, as shown in FIG. 2, a binocular search range and an optical flow search range are set in order to search for an optimal correspondence point, and then a set range is searched.

The measurement value update unit 120 is composed of two modules, that is, a binocular key update module 121 and an optical flow update module 122.

The binocular eye updating module 121 finds a binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular.

은 좌측 영상을 기준으로 동일 라인(y축)상에 존재하는 좌, 우 영상의 픽셀들의 차를 최소화 하는 양안차 d를 찾는 것을 의미하며, 두 번째 항인

은 시간 t-1의 영상에서 예측, 추정한 시간 t에서의 양안차 예측 값(

)과 시간 t에서 선택할 수 있는 양안차의 차를 최소화 하는 양안차 d를 찾고자 하는 것을 의미한다. 그리고 마지막 항인

은 픽셀 x에서 x-1로의 불연속 패널티 값을 의미하는데, 이때 이웃하는 픽셀의 양안차 값이 연속적이지 못할 경우 그 값은 더 커진다.In Equation (5), subscript t denotes time t, superscripts l and r mean left and right, and f (x ^l ) denotes a pixel value of an image based on the left image. Accordingly,

Means to find the binocular difference d that minimizes the difference between the pixels of the left and right images existing on the same line (y-axis) on the basis of the left image,

Is predicted from the image at time t-1, and the binocular difference predicted value at time t

) And the binocular difference d that can be selected from the time t is minimized. And the last mortal

Means a discontinuous penalty value from pixel x to x-1, where the value is greater if the binocular value of neighboring pixels is not continuous.

The optical flow update module 122 finds the position of the pixel that minimizes the energy function of Equation 6 below and calculates the pixel position difference between the previous frame (image at time t-1) and the current frame (image at time t) Outputs the motion as a result value.

은 어떤 픽셀(x)의 이웃 픽셀(x')에 대한 좌, 우 영상의 밝기(intensity) 차이 값을 의미하며, 두 번째 항인

은 시간 t에서 어떤 픽셀(x)의 양안차

와 시간 업데이트부(130)에서 갱신된 시간 t-1에서 추정된 양안차 예측값(

)의 차가 최소가 되는 픽셀위치(x')를 의미한다.In Equation (6) above,

Represents the intensity difference value of the left and right images for a neighboring pixel x 'of a certain pixel x,

(X) < / RTI > at time < RTI ID = 0.0 &

And a binocular difference predicted value (a) estimated at time t-1 updated by the time update unit 130

) Is the minimum pixel position (x ').

The motion (light flow) calculated from the above is expressed by Equation (7) below.

If all corresponding points are found through the above process, the corresponding points of the previous frame (frame at time t-1) are stored in the register 140 as a parent node, as shown in Equation (8) below.

The result of Equations (7) and (8) can be applied to Equation (3) to calculate the final output value, that is, the binocular and optical flow, (140).

The operation of the three-dimensional structure estimating apparatus using the optical flow and the stereo matching technique as described above will be described with reference to FIGS. 2 to 3. FIG.

FIG. 3 is a flowchart illustrating a process of calculating a binocular disparity and an optical flow with respect to left and right input images according to an embodiment of the present invention, and sequentially calculates a disparity map and an optical flow.

As shown in FIG. 3, for the left and right input images f ¹ and f ^r given at times t and t-1, the initialization is performed for the entire x _{0 | 0} as shown in the following Equation 9 (S300) .

)으로부터 시간 t의 대응점들

을 추정함과 더불어 좌측 영상을 기준으로 시간 t-1에서 시간 t의 양안차 예측 값(

)을 추정(S302)한다. 추정된 양안차 예측 값(

)과 시간 t의 대응점들(

)은 측정 값 업데이트부(120)에 제공된다.Then, the recursive operation is started, that is, the time update unit 130 uses the above-described equation (4) to calculate the motion

) To the corresponding points of time t

And estimates binocular disparity at time t from time t-1 based on the left image

(S302). The estimated binocular differential value (

) And the corresponding points of time t (

Is provided to the measurement value update unit 120. [

)과 시간 t의 대응점들(

) 및 좌우 카메라(100, 110)로부터 제공받은 좌우측 입력 영상(f^l, f^r)을 이용하여 시간 t의 양안차와 광 흐름을 순차적으로 계산한다. 즉, 측정 값 업데이트부(120)의 양안차 업데이트 모듈(122)은 시간 t의 좌우측 입력 영상의 전체 픽셀x에 대해, 양안차 예측 값((

) 및 좌우측 입력 영상(f^l, f^r)을 이용해 시간 t의 좌우측 입력 영상에 대한 양안차(

)를 결정(S304)한다. 즉, 상기의 수학식 5에 기재된 에너지 함수를 최소화하는 양안차를 찾아 레지스터(130)에 저장한다.Then, the measurement value update unit 120 updates the measurement value of the binaural difference predicted value

) And the corresponding points of time t (

And the left and right input images f ^l and f ^r provided from the left and right cameras 100 and 110 to sequentially calculate binocular disparity and optical flow at time t. That is, the binocular difference update module 122 of the measurement value update unit 120 updates the binocular difference prediction value ((

) And the left and right input images (f ¹ , f ^r )

(S304). That is, a binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular nucleus

)를 찾는 방법은 기존의 스테레오 매칭 기법 belief propagation(BP) 또는 dynamic programing(DP)을 이용해 찾는다.The binocular function minimizing by using the energy function described in Equation (5)

) Can be found using conventional stereo matching techniques such as belief propagation (BP) or dynamic programming (DP).

Then, the optical flow update module 124 calculates the optical flow at time t (S306) using the above Equations (6) and (7) and stores it in the register 130.

Therefore, the calculated parent node, optical flow, and binocular difference values are stored in the register 130, and these values become the results in the time t frame.

In the embodiment of the present invention, the binocular difference and the optical flow are sequentially calculated. However, it is also possible to calculate them at the same time.

First, for the left and right input images (f ^l , f ^r ) given at times t and t-1, the initialization is performed for the entire x _{0 | 0 as} in the following equation (10).

)으로부터 시간 t의 대응점들

을 추정함과 더불어 좌측 영상을 기준으로 시간 t-1에서 시간 t의 양안차 예측 값(

)을 추정한다. As a result of the time update process as in S202,

) To the corresponding points of time t

And estimates binocular disparity at time t from time t-1 based on the left image

).

Then, the measurement update unit 130 obtains the binocular and optical flow values from the values minimizing the energy function of Equation (11) and stores it in the register 140 at the same time.

은 어떤 픽셀(x)의 이웃 픽셀(x')에 대한 좌, 우 영상의 밝기(intensity) 차이 값을 의미하고,

은 좌측 영상을 기준으로 동일 라인(y축)상에 존재하는 좌, 우 영상의 픽셀들의 차를 최소화하는 양안차 d를 찾는 것을 의미하며,

은 시간 t-1프레임에서 예측, 추정한 시간 t에서의 양안차 예측 값

과 시간 t에서 선택할 수 있는 양안차의 차를 최소화 하는 양안차 d를 찾고자 하는 것을 의미한다. 이를 통해 시간 t에서의 양안차(d)와 좌우측 입력 영상에 대한 광 흐름(

)을 산출할 수 있다.In Equation (11)

Means an intensity difference value of the left and right images for a neighboring pixel x 'of a certain pixel x,

Means to find a binocular difference d that minimizes a difference between pixels of left and right images existing on the same line (y-axis) on the basis of the left image,

Is a predicted value at time t-1 frame, a binocular difference predicted value at time t

And a binocular difference d that minimizes the difference between the binocular disparities that can be selected at time t. Thus, the binocular disparity (d) at time t and the optical flow for left and right input images

) Can be calculated.

According to embodiments of the present invention, stereo matching can estimate the candidate values of the corresponding points of the light flow and calculate the actual correct corresponding points using the actual image frame and the candidate values, thereby effectively reducing the search range of the light flow and the binocular Not only the operation speed and accuracy can be improved.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, You will understand. For example, a person skilled in the art can change the material, size and the like of each component according to the application field, or can combine or replace the disclosed embodiments in a form not explicitly disclosed in the embodiment of the present invention. It is to be understood that the invention is not limited to the disclosed embodiments. Therefore, it should be understood that the above-described embodiments are to be considered in all respects as illustrative and not restrictive, and that these modified embodiments are included in the technical idea described in the claims of the present invention.

FIG. 1 is a block diagram illustrating an apparatus for estimating a three-dimensional structure using an optical flow and a stereo matching technique according to an embodiment of the present invention,

2 is a view showing a relationship between an optical flow and a binocular according to the present invention,

3 is a flowchart illustrating a process of calculating a binocular disparity and an optical flow with respect to left and right input images according to an exemplary embodiment of the present invention.

Description of the Related Art

100: Left camera 110: Right camera

120: measured value update unit 130: time update unit

140: Register

Claims (11)

A register in which a binocular difference and an optical flow for a parent node t-1 of left and right input images input at time t are stored, The parent node estimates the binocular difference value of the time t estimated by the parent node and the optical flow values of the left and right images of the parent node based on the corresponding image estimated from the left and right input images of the time t, A time update unit, A measurement value update unit for calculating the binocular disparity and the optical flow for the left and right input images using the optical flow value, the binocular disparity prediction value, and the corresponding points in the parent node calculated by the time update unit, And a stereo matching unit for combining the optical flow and the stereo matching. The method according to claim 1, The time update unit, The optical flow value, the binocular difference predicted value and the corresponding points at the parent node are calculated by the following equation

(

Lt; RTI ID = 0.0 > t-1 < / RTI &

) From the left and right images of the estimated time t,

Is a binocular difference predicted value at a time t estimated at a time t-1 based on the left image,

Is the optical flow value of the time t-1 frame calculated at each of the left and right images at time t-1) A three - dimensional structure estimator combining optical flow and stereo matching. The method according to claim 1, Wherein the measurement value updating unit comprises: A binocular difference that minimizes a difference between pixels of the left and right input images existing on the same line with respect to the input left image, A binocular eye updating module for calculating a binocular difference of the time t based on a discontinuous penalty from any pixel to a previous pixel, The difference between the brightness difference of the left and right images for a neighboring pixel of a specific pixel in the left and right input images of the time t and the difference between the binocular difference of the specific pixel at the time t and the binocular difference predicted by the time update unit becomes minimum And a light flow update module for calculating the light flow of the time t using the pixel position A three - dimensional structure estimator combining optical flow and stereo matching. The method of claim 3, Wherein the binocular < RTI ID = 0.0 > A binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binocular binoculars

(

Means to find a binocular difference d that minimizes a difference between pixels of left and right images existing on the same line (y-axis) on the basis of the left image,

Is predicted from the image at time t-1, and the binocular difference predicted value at time t

) And a binocular which minimizes the difference between the binocular disparities that can be selected at time t d) A three - dimensional structure estimator combining optical flow and stereo matching. 5. The method of claim 4, The binocular eye update module may detect a binocular binocular by minimizing the energy function using a stereo matching technique of BP or DP A three - dimensional structure estimator combining optical flow and stereo matching. The method of claim 3, The optical flow update module includes: Finds a position of a pixel that minimizes the energy function of the following equation, calculates a position difference between the frame of the parent node at the found pixel and the frame at the time t, and calculates an optical flow value at the time t

(

Is the intensity difference value of the left and right images for a neighboring pixel (x ') of a certain pixel (x)

(X) < / RTI > at time < RTI ID = 0.0 &

And binocular differential predictor ((

(X ') at which the difference between the pixel position A three - dimensional structure estimator combining optical flow and stereo matching. Estimating corresponding points of the left and right input images at time t from the optical flow at the pre-stored time t-1, Estimating the estimated binocular differential value of the time t and the optical flow at the time t-1 from the estimated corresponding points, Calculating a binocular disparity and an optical flow value at the time t using the optical flow at the estimated time t-1, the binocular difference prediction value, and the estimated corresponding point, A three - dimensional structure estimation method combining optical flow and stereo matching. 8. The method of claim 7, The binocular difference prediction value at the time t is a binocular difference predicted value at the time t estimated at the time t-1 based on the left input image in the left and right input frames A three - dimensional structure estimation method combining optical flow and stereo matching. 8. The method of claim 7, The step of calculating the binocular magnification and the optical flow value at the time t may include: Searching for a first binocular difference that minimizes a difference between pixels of the left and right input images existing on the same line based on the left input image; Searching for a second binocular difference that can minimize a difference between the binocular difference predictive value and a binocular difference that can be selected at the time t, Calculating a discontinuous penalty value from a particular pixel of the left or right input image frame to a neighboring pixel; Calculating a value for minimizing an energy function composed of the first binocular binocular, the second binocular bin and the penalty value as a binocular of the time t A three - dimensional structure estimation method combining optical flow and stereo matching. 8. The method of claim 7, The step of calculating the binocular magnification and the optical flow value at the time t may include: Calculating a brightness difference value between left and right images for a neighboring pixel of a specific pixel in the left or right input image; Finding a position of a pixel at which the difference between the binocular difference and the binocular difference predicted value of the specific pixel is minimum at the time t; Calculating a light flow value at the time t by minimizing the energy function by using the brightness difference value and the position of the pixel as an energy function; A three - dimensional structure estimation method combining optical flow and stereo matching. 8. The method of claim 7, The step of calculating the binocular magnification and the optical flow value at the time t may include: The binocular difference and the optical flow value of the time t are calculated from the values minimizing the energy function expressed by the following equation

(

Is the intensity difference value of the left and right images for a neighboring pixel (x ') of a certain pixel (x)

A binocular difference d for minimizing a difference between pixels of the left and right images existing on the same line (y-axis) on the basis of the left image,

The binocular difference prediction value

And a binocular that minimizes the difference between the binocular eyes that can be selected at time t d) A three - dimensional structure estimation method combining optical flow and stereo matching.

Images