KR20080001794A - Apparatus and method for estimating fundamental matrix using robust correspondence point - Google Patents

Abstract

An apparatus and a method for estimating a fundamental matrix using a robust corresponding point are provided to estimate an optimum fundamental matrix with high precision, thereby improving reliability in various fields such as multi-view image generation, camera calibration and three dimensional image reconfiguration performed from a stereo image. A feature point extracting unit(120) extracts feature points by measuring a variation amount of brightness of an image. A candidate point selecting unit(130) selects candidate points, where a score by an autocorrelation function satisfies a predetermined range, from the feature points. A candidate sorting unit(140) calculates a median vector of a motion vector for candidate points within each area after dividing each image into four, and removes candidate points deviated from a predetermined threshold value for a size and a direction of the median vector. A fundamental matrix selecting unit(150) partitions each image into 8x8 areas, selects 8 areas from the partitioned areas, selects total 8 couples of corresponding points by arbitrarily extracting one corresponding point within each of the selected areas, and calculates a fundamental matrix from the selected corresponding point. After forming epipolar geometry for the calculated fundamental matrix, the fundamental matrix selecting unit selects a fundamental matrix where a distance error between an epipolar plane component and the corresponding point is the minimum.

Description

Basic matrix estimation apparatus and method using robust correspondence point {APPARATUS AND METHOD FOR ESTIMATING FUNDAMENTAL MATRIX USING ROBUST CORRESPONDENCE POINT}

1 is an exemplary view showing the concept of epipolar geometry,

2 is a schematic structural diagram of a basic matrix estimating apparatus according to the present invention;

3 is an exemplary diagram illustrating a motion vector for a candidate point in a quadrant divided image;

4 is an exemplary diagram illustrating 8 × 8 region division including corresponding points;

5 is a schematic flowchart of a basic matrix estimation method according to the present invention;

Figure 6a is an image showing the results of the method according to the invention and the RANSAC method for the 'building' image,

6B is a table showing the results of the process in FIG. 6A;

7A is an image showing results of a method according to the present invention and a RANSAC method for a 'castle' image,

7B is a table showing the results of the process in FIG. 7B.

** Description of symbols for the main parts of the drawing **

100: basic matrix estimation apparatus 110: image input means

120: feature point extraction means 130: candidate point selection means

140: candidate point selection means 150: basic matrix selection means

The present invention relates to an apparatus and method for estimating a basic matrix using robust correspondence points, and in particular, an optimal basic matrix by extracting robust correspondence points through epipolar geometry from two un-calibrated left and right images. An apparatus and method for estimating the same are provided.

Correspondence point extraction from natural images can be used in a variety of ways, including multi-view image generation [1], camera calibration [2], and three-dimensional image reconstruction from stereo images [3]. Plays an important role in the field

In the case of the stereo image, as a technique for obtaining 3D information from an arbitrary image of the overlapped image, it is variously used in fields such as virtual reality as well as robot navigation. When any one point in the three-dimensional space is respectively projected on the target stereo image, a corresponding point appears between the two images, and their relationship can be described by epipolar geometry. Using epipolar geometry, we can derive a fundamental matrix containing the relative geometric information of two cameras, and find the corresponding point in the other image in the other image. Will be able to perform

을 서로 다른 위치의 카메라(

,

)에 의한 이미지 평면(

,

)에 투영하면, 각각의 영상에 점

과

로 맺히며, 이러한 두 점을 대응점이라고 한다. 공간상의 점

과 두 카메라가 이루는 평면은 에피폴라 평면이며, 에피폴라 평면과 영상 평면이 교차하는 선을 에피폴라 선(

,

)이라고 하고, 두 카메라를 연결하는 선분과 각 영상 평면의 교점을 에피폴(

,

)이라고 한다. 여기서 공간상의 점 M은 투영된 점

,

는 에피폴라 선상에 존재하며, 이를 에피폴라 구속 조건이라고 한다. Referring to FIG. 1, the epipolar geometry is briefly described. First of all, the relationship between the images obtained from two cameras may be defined as a point-to-line correspondence, not a point-to-point correspondence. . Random point in space

At different locations

,

Image plane by

,

), A dot on each image

and

These two points are called correspondence points. Point in space

The plane formed by the two cameras is the epipolar plane, and the epipolar line (the line where the epipolar plane and the image plane intersect)

,

), And the intersection of the two cameras and the intersection of each image plane is called epipole (

,

It is called). Where point M in space is the projected point

,

Is on the epipolar line and is called epipolar constraint.

The algebraic expression for this is the basic matrix (F) mentioned above, and satisfies the condition as shown in Equation 1 below.

................................. [수학식 1]

............... Equation 1

Currently, the problem of accurately finding the correspondence between two images remains an important challenge in the field of computer vision. The limitation of this task is that there is only geometric information between two images without any known information about the image. There are two types of image matching methods. One is template matching, which is an algorithm that correlates brightness values between images and finds similar pixel values [4]. The other is a feature matching algorithm that extracts feature information between two images [5].

To date, various algorithms for estimating the basic matrix have been studied. For example, a direct method, an iterative method, and a robust method are known.

The direct method can quickly estimate the basic matrix by the least-square method for seven or eight pairs of matching points, but has a disadvantage of being affected by image noise. The iterative method is based on the optimization method, and it has more disadvantages than the linear method, but it takes longer to execute, and it is not possible to reduce the effects of position error and mismatch error. On the other hand, the relatively robust method proposed recently has the advantage of reducing the effect of outliers due to erroneous matching errors or positional errors due to quantization of the target image, a representative example of which is RANSAC proposed by M.Fischer et al. [6].

According to the RANSAC method, the epipolar geometry is directly constructed based on the corresponding candidate points, and the wrong correspondence points are removed one by one. However, in this method, in determining some thresholds included in the algorithm, the values must be determined experimentally according to the distribution of the feature points obtained earlier and the positions of the corresponding candidate points, and among the calculated values, The results vary greatly depending on the situation, and there are many restrictions in the practical application.

Z. Zhang also includes an algorithm that removes invalid candidates among initial matched candidate points using neighboring correspondence points and mutual distance information, but one candidate point that has a large error because it uses mutual distance information. Relatively unstable results such as the removal of all the corresponding candidate points for the pair are shown [7].

Statement

[1] R. Hartly and A. Zisserman, Multiple View Geometry in Computer Vision, Cambridge University Press, 2000.

[2] Z. Zhang, "A Flexible New Techinque for Camera Calibration," IEEE Transaction on Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1-20, 1998.

[3] R. Cipolla and E. Boyer, "3D Model Acquisition from Uncalibrated Images," IAPR Workshop on Machine Vision Applications, pp. 559-568, 1998.

[4] P. Fua, "A Parallel Stereo Algorithm that Produces Dense Depth Maps and Preserves Image Features," Machine Vision and Applications, vol. 6, no. 1, 1993.

[5] R. Horaud and T. Skordas, "Stereo Correspondence Through Feature Grouping and Maximal Cliques," IEEE Trans. PAM, vol. 11, no 12, pp. 1310-1180, 1989.

The present invention was devised to solve the above-mentioned problems, and a main object of the present invention is to minimize the extraction of wrong correspondence points from two compared images, thereby to estimate an optimal basic matrix with higher accuracy than the conventional one. The present invention provides an apparatus and method.

2 is a view showing a schematic configuration of the basic matrix estimation apparatus 100 of the present invention, the image input means 110, the feature point extraction means 120, the candidate point selection means 130, the candidate point selection means ( 140) and basic matrix selecting means (150).

The image input unit 110 receives an un-calibrated left and right image 10, and the feature point extracting unit 120 measures a change in brightness of the left and right images processed to include only brightness information. Extract.

Here, feature point extraction is performed through Equation 2 below according to the conventional Harris edge search method [Reference papers: C. Harris and M. Stephens, "A combined corner and edge detector", 4th Alvey Vision Conference, pp. 147-151, 1988.

.......................... [수학식 2]

........... [ Equation 2 ]

는 다음과 같은 행렬 요소를 갖는다.Of the above equation

Has the following matrix elements:

에서

와

는

와

방향으로의 미분을 나타낸다.Brightness video

in

Wow

Is

Wow

The derivative in the direction is shown.

로 설정하고, 두 번째 영상에서 중심점 주위로 사각형의 검색 영역의 크기를

로 설정한 후, 첫 번째 영상에서의 후보점

과 두 번째 영상에서의 검색 영역 안에 놓여있는 후보점

사이의 상관성을 검사한다.The candidate point selecting unit 130 selects candidate correspondence points (hereinafter, referred to as 'candidate points') whose correlation degree satisfies a predetermined range from the feature points extracted from the left and right images. The size of the correlation window for applying the function is

To set the size of the rectangle's search area around the center point

After setting to, the candidate point in the first image

Candidates placed in the search region in the second and second images

Check the correlation between.

Therefore, the aforementioned correlation function is defined by Equation 3 below.

[ Equation 3 ]

Here, score means correlation.

이고,

ego,

는 점 (

,

)에서의 평균을 나타내며,

는 (

,

)의 이웃하는

에서 영상

의 표준 편차를 나타내고, 상기

는 아래와 같이 정의된다.

Points (

,

) Mean,

Is (

,

Neighbors

Video from

Represents the standard deviation of

Is defined as

와

는 각각 좌우 영상의 해당 화소 값을 나타내고,

과

은 상관관계를 계산하는 영역의 크기를 나타내며, 일반적으로

=7,

=7을 사용한다. Also,

Wow

Respectively represent corresponding pixel values of the left and right images,

and

Represents the size of the area for which the correlation is calculated,

= 7,

Use = 7.

Since the range of the correlation has been normalized as shown in Equation 3 defined above, a value between -1 and 1 can theoretically be 1 if the correlation between the two images is exactly the same. It will have a value. In the present invention, by setting the threshold value to 0.8, candidate points having a correlation value of 0.8 or more and 1 or less will be selected from feature points of the extracted left and right images, but the present invention is not limited thereto.

At this time, since Equation 3 performs calculation only on initial candidate points (feature points), not on all pixel values in the image, the processing speed can be improved compared to the conventional method.

For reference, more detailed technical information regarding the above-described correlation function and correlation window can be found in Z. Zhang's paper [Z. Zhang, R. Deriche, O. Faugeras, and Q.-T. Luong, "A Robust Technique for Matching Two Uncalibrated Images Through the Recovery of the Unknown Epipolar Geometry," Artificial Intelligence Journal, vol. 78, pp. 87-119, 1995].

Meanwhile, the candidate point selecting means 140 performs a function of removing wrong candidate points from the selected candidate points as described above, which is based on a median vector based on a motion vector of the selected candidate points. Correct candidate points are selected by calculating a median vector and removing candidate points that deviate from a threshold for the magnitude and direction of the calculated median vector.

In detail, as shown in FIG. 3, the motion vector represented in each region is found by dividing the left and right images into four equal parts. In this case, since the representative value of the motion vector should be the highest value in the corresponding region, the representative value of the motion vector is taken as an intermediate value rather than an average value. Based on the median vector found as the representative value of each region, candidate points that deviate from a predetermined threshold for size and direction are removed. In the present invention, it is preferable to find a predetermined threshold experimentally for each image.

라하고, 해당 영역에 포함된 각 후보점들의 움직임 벡터를

라 하면,

의 크기에 해당하는

성분과 방향에 해당하는

성분에 대해서 아래와 같이 두 가지의 경우(조건식 ①,②)에 잘못된 후보점으로 판별한다.The median vector of each interval

The motion vector of each candidate point included in the corresponding area

Say,

Corresponds to the size of

Corresponding to components and directions

In the following two cases (components ① and ②), components are identified as wrong candidate points.

> 크기 임계값①

> Size threshold

> 방향 임계값②

Direction threshold

)로 선택한다.Next, the basic matrix selecting means 150 calculates a basic matrix from the selected candidate points, that is, corresponding points through an eight-point algorithm, but has a minimum distance error between corresponding corresponding points on the epipolar plane based on the epipolar geometry. Is the optimal base matrix (

Select with).

The basic matrix selecting means 150 uses the RANSAC (RANdom SAmple Consensus) method, but there are differences as follows.

As shown in FIG. 4, a rectangular border including corresponding points is covered with an image, the border is divided into 8 × 8 areas, and eight areas are randomly selected, and one corresponding point is randomly selected within each selected area. The pairs of matching points are obtained and the basic matrix is computed through the 8-point algorithm. Basic technical details on this can be found in the paper by L. Higgins [L. Higgins, "A Computer Algorithm for Reconstructing a Scene from Two Projections," Nature, vol. 293, pp. 133-135, 1981.

Among the basic matrices repeatedly calculated by this process, the matrix that minimizes the value of [Equation 4] below to form the geometry of the epipolar plane and calculate the distance error between the epipolar plane segment and the corresponding point is optimal. It is selected as a matrix.

......................... [수학식 4]

......................... [Equation 4]

Based on the above description, the basic matrix estimation method of the present invention can be arranged as shown in FIG. 5.

First, a feature point is extracted by measuring an amount of change in brightness of a left and right image processed to include only brightness information (S501). This is accomplished by the Harris edge search method of [Equation 2].

Based on the correlation function shown in Equation 3, a candidate point satisfying a predetermined range of scores is selected from the extracted feature points (S502).

Each of the left and right images is divided into four sections (S503), and a median vector is calculated based on the motion vectors of the candidate points selected in step S502 for each region (S504), and then the median vector calculated for each region is calculated. The candidate points deviating from the threshold for the size and the direction are removed (S505). At this time, the threshold may be set differently for each image, and wrong candidate points are determined by the two conditional expressions described above.

Next, a rectangular border including corresponding points is set for each of the left and right images, and the border is divided into 8 × 8 areas (S506), and eight areas of the partitioned areas (64) are arbitrarily selected (S507). One corresponding point is arbitrarily selected within each selected area, and a total of eight pairs of corresponding points are selected (S580).

Based on the selected eight pairs of matching points, a basic matrix is calculated through an eight-point algorithm (S509), and an epipolar geometry is formed to calculate a distance error between the epipolar plane line segment and the corresponding corresponding point (S510). Error calculation is performed using Equation 4.

The above-described steps S507 to S510 are repeatedly performed to select a basic matrix having a minimum distance error (S511).

[ Experimental example ]

For simulation of the above-mentioned technical ideas, IBM-PC of Pentium IV 1.4GHz, 256M DDR specification is used. In this experimental example, the epipolar plane segment is plotted against the optimal elementary matrix and the mean of the corresponding distance errors is presented numerically. In addition, the results of applying the RANSAC method, which is commonly used for performance comparison, are presented.

FIG. 6A shows a result image by the method according to the present invention and a RANSAC method for a 'building' image, and FIG. 6B summarizes the result of processing by the quantum method as a table. In addition, Figure 7a shows the result image by the method according to the invention and the RANSAC method for the 'castle' image, Figure 7b summarizes the results of the processing by the quantum method as a table.

As a result, it can be seen that the method proposed by the present invention has a larger number of finally obtained correspondence points and a smaller distance error even though erroneous correspondence points are controlled than the conventional RANSAC method.

According to the present invention as described above, there is a peculiar effect that can estimate the optimal basic matrix with a higher accuracy than the prior art. Therefore, it will act as a major technical factor to increase the reliability in various fields such as multi-image generation, camera calibration, and 3D image reconstruction from stereo images.

As described above and described with reference to a preferred embodiment for illustrating the technical idea of the present invention, the present invention is not limited to the configuration and operation as shown and described as described above, it is a deviation from the scope of the technical idea It will be understood by those skilled in the art that many modifications and variations can be made to the invention without departing from the scope of the invention. Accordingly, all such suitable changes and modifications and equivalents should be considered to be within the scope of the present invention.

Claims (9)

An apparatus for estimating a basic matrix from a comparative left and right image, Feature point extracting means for extracting feature points by measuring an amount of change in brightness of the image; Candidate point selecting means for selecting candidate points whose scores satisfy a predetermined range from the feature points according to a correlation function; Candidate point selecting means for dividing each image into quadrants to calculate a median vector of motion vectors for candidate points in each region, and removing candidate points that deviate from predetermined thresholds for the magnitude and direction of the median vector; And Each image is divided into 8x8 areas, eight areas are selected from the partitioned areas, and one corresponding point is randomly extracted within each selected area to select a total of eight pairs of matching points, and a basic matrix is calculated from the selected points. Basic matrix selecting means for forming an epipolar geometry with respect to the calculated basic matrix to select a basic matrix having a minimum distance error between the epipolar plane segment and the corresponding point; Basic matrix estimation apparatus using a robust corresponding point, characterized in that it comprises a. The method according to claim 1, The feature point extraction means, A basic matrix estimation apparatus using robust correspondence points, which is characterized by extracting feature points using Harris edge search method. The method according to claim 1, The correlation coefficient base matrix estimation apparatus using a robust corresponding point, characterized in that calculated by the following equation. [ Equation ]

here,

,

only,

Points (

,

) Mean,

Is (

,

Neighbors

Video from

Represents the standard deviation of

Wow

Respectively represent corresponding pixel values of the left and right images,

and

Is the size of the area for which the correlation is calculated. The method according to claim 1, And a range of the correlation is 0.8 or more and 1 or less. The method according to claim 1, And the candidate point selecting means removes the candidate point when the following two conditions are satisfied. [Condition; ①

> Size threshold ②

Direction threshold only,

Is the median vector,

Is the motion vector of the candidate point. As a method of estimating the basic matrix from the compared left and right images, A first step of extracting feature points by measuring an amount of change in brightness of the image; A second step of selecting candidate points having a correlation degree corresponding to a predetermined range according to a correlation function from the extracted feature points; Dividing each image into four equal parts to calculate a median vector of motion vectors for candidate points in each region, and removing candidate points that deviate from predetermined thresholds for the magnitude and direction of the median vector; Each image is divided into 8x8 areas, eight areas are selected from the partitioned areas, and one corresponding point is randomly extracted within each selected area to select a total of eight pairs of matching points, and a basic matrix is calculated from the selected points. A fourth process of doing; A fifth step of forming an epipolar geometry with respect to the calculated basic matrix to calculate a distance error between the epipolar plane segment and the corresponding point; And A sixth process of repeating the fourth process and the fifth process to select a basic matrix having a minimum distance error; Basic matrix estimation method using a strong corresponding point, characterized in that it comprises a. The method according to claim 6, Feature point extraction of the first process, A method of estimating the basic matrix using robust correspondence points, using the Harris edge search method. The method according to claim 6, The score is calculated by the following equation, the basic matrix estimation method using a strong corresponding point. [ Equation ]

here,

,

only,

Points (

,

) Mean,

Is (

,

Neighbors

Video from

Represents the standard deviation of

Wow

Respectively represent corresponding pixel values of the left and right images,

and

Is the size of the area for which the correlation is calculated. The method according to claim 6, The third process is a method of estimating a basic matrix using robust correspondence points, wherein candidate points are removed when the following two conditions are satisfied. [Condition; ①

> Size threshold ②

Direction threshold only,

Is the median vector,

Is the motion vector of the candidate point.

Images