KR101995466B1 - Stereo image matching based on feature points - Google Patents

Abstract

The present invention relates to a method and an apparatus for matching images of a stereo camera using feature points. The method is configured to acquire slopes of line segments which connect feature point pairs with each other matched after image matching using the feature point pairs, and to determine, as a matching error, feature point pairs with slopes which differ by a predetermined value or more from the most frequent slope or feature point pairs with slopes which differ by a predetermined value or more from an average value of the slopes of line segments connecting the feature point pairs with each other so as to exclude feature point pairs with the determined matching error from the feature point pairs and thus, it is possible to accurately match stereo images.

Description

[0001] STEREO IMAGE MATCHING BASED ON FEATURE POINTS [0002]

The present invention relates to a method of matching a stereo image captured by a stereo camera, and more particularly, to a method of matching a stereo image using a slope between detected feature points in left and right images.

2. Description of the Related Art [0002] In recent years, studies on autonomous vehicles have been actively developed, and various methods and devices for recognizing the autonomous vehicles have been developed.

Monocular cameras, stereo cameras, RADAR, and LIDAR have been studied as sensors for recognizing the driving environment in autonomous vehicles. Although a monocular camera can recognize a near object, there is a limit to accurately recognize the distance between a camera and an object. LIDAR can secure the disadvantage of a monocular camera, but it is expensive to install and the appearance of the vehicle is inevitable. It is difficult to apply it to a commercial vehicle.

On the other hand, stereo cameras can solve the problem of distance recognition, which is a disadvantage of monocular cameras, but they can be easily installed in vehicles at a lower cost than LIDAR.

A method of recognizing a driving environment using a stereo camera is a technique imitating a method of recognizing the shape and distance of a person through two eyes. That is, when two left and right cameras serve as two eyes of a human, respectively, and the image of the three-dimensional space is read as a two-dimensional image, the stereo camera's recognition system displays two left and right images through two left and right two- Allows you to recognize the distance between the object and the camera.

Hereinafter, an environment recognition method using a stereo camera will be described with reference to FIG. 1 showing an environment recognition system using a general stereo camera.

In general, an environment recognition system using a stereo camera includes an image sensing unit 100, an image analysis unit 122, an image matching unit 124, and a depth map generation unit (first and second cameras 102 and 104) And an image processor 120,

The two cameras 102 and 104 simultaneously capture the front object 130 at the same time. The left image picked up by the first camera 102 and the right picked up by the second camera 104 are two-dimensional images and analyzed by the image analyzer 122.

The image analysis unit 122 executes another analysis method according to the matching method executed by the image matching unit 124. [ In general, the matching method is classified into three types, namely, an area based matching method, a phase based matching method, and a feature point based matching method. Since the area-based matching method is sensitive to various kinds of image distortion, it is difficult to set the matching window size according to the influence of image noise, contrast difference, and the like. The phase-based matching method is advantageous in that it is strong against high-frequency image noise, but a disadvantage due to a low band pass output signal is known. On the other hand, the feature point based matching method is a method of utilizing the structural information of the space obtained from the image, and is effectively used for matching the images obtained through the stereo camera. Since the present invention relates to an image matching method based on a minutiae point, an image analysis on a minutiae point will be mainly described below.

The image matching unit 124 matches feature points analyzed by the image analysis unit 122 with each other. That is, the feature points extracted from the left image captured by the first camera 102 are matched with the extracted feature points extracted from the right image captured by the second camera 104 to detect the pair of feature points. The matched pair of feature points represent the same point of the object 130 captured by the first and second cameras 102 and 104.

The depth map creating unit 126 creates a depth map by detecting disparity from pairs of minutiae matched by the image matching unit 124 and calculating the three-dimensional depth of the object 130 represented by the minutiae points . Through the above-described process, the autonomous vehicle can recognize the driving environment three-dimensionally.

The image matching performed by the image matching unit 124 may cause a large number of errors due to an error of the calibration process of the first and second cameras 102 and 104 and an error of the image sensor of each camera. The present invention proposes a method that can more accurately match a stereo image by reducing such stereo matching errors.

A stereo camera image matching method using feature points according to an embodiment of the present invention includes a first step of extracting a plurality of first feature points from a first image picked up by a first image pickup device of a stereo camera, A second step of extracting a plurality of second feature points from the second image picked up by the first feature point, a third step of matching the plurality of first feature points and the plurality of second feature points, a step of arranging the first image and the second image side by side A fourth step of calculating a slope of a line segment connecting the first feature point and the second feature point of each of the plurality of pairs of feature points matched in the state, and determining a matching error according to the slope of the line segment, And a fifth step of removing the feature points from the plurality of matching point pairs.

The stereo camera image matching method using feature points according to another embodiment of the present invention further includes a step of obtaining a histogram of a slope in a fourth step and a slope having a most frequent number is obtained through a histogram in a fifth step, The feature point pairs of a slope deviating from a predetermined angle from many slopes are excluded from a plurality of matched pairs of feature points.

The stereo camera image matching method using feature points according to another embodiment of the present invention includes a sixth step of obtaining an average slope in a fourth step and a fifth step of matching a pair of feature points of a slope out of a predetermined angle with an average slope, And is excluded from a plurality of pairs of feature points.

In the stereo camera image matching method using the feature points according to another embodiment of the present invention, after the feature point pairs are removed in the fifth step, the mean slope is updated based on the remaining feature point pairs, and the fifth step is performed again using the updated average slope .

The stereo camera image matching method using feature points according to another embodiment of the present invention is characterized in that after the fifth step, in each of the matched pairs of feature points, the x coordinate value of the first feature point in the first image, The feature point pairs in which the difference between the x coordinate values of the two feature points are larger than the predetermined value are excluded from a plurality of matching feature point pairs.

The stereo camera image matching method using feature points according to another embodiment of the present invention is characterized in that the predetermined value is set to a value larger than the x coordinate deviation between the first image and the second image with respect to the object closest to the stereo camera.

As described above, according to various embodiments of the present invention, in matching images obtained through a stereo camera, a matching error is detected based on a slope of a line segment between minutiae of the left and right images, Thereby enabling more accurate stereo image matching.

Figure 1 shows a typical stereo image matching system using a stereo camera.
FIG. 2 is a diagram for explaining a process of extracting feature points that are resistant to size change.
Fig. 3 is a diagram showing the relationship between the focal points of the first and second cameras, the imaging points of the object to be imaged, and the pixel points on the left and right images.
FIG. 4 is a diagram illustrating the lines connecting the left and right images side by side and matching feature points obtained by the SIFT algorithm through image matching, and connecting the matched pairs of feature points.
5 is a flowchart illustrating a procedure of a stereo image matching method according to an embodiment of the present invention.
6 is a flowchart illustrating a procedure of a stereo image matching method according to another embodiment of the present invention.
7 is a flowchart illustrating a procedure of a stereo image matching method according to another embodiment of the present invention.

Brief Description of the Drawings The advantages and features of the embodiments disclosed herein, and how to accomplish them, will be apparent with reference to the embodiments described below with reference to the accompanying drawings. It should be understood, however, that the invention is not limited to the disclosed embodiments, but may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. But only to provide a complete picture of the categories.

The terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail.

Although the terminology used herein should be interpreted taking into account the functions of the disclosed embodiments, it is to be understood that the present invention may be embodied in many other specific forms without departing from the spirit or essential characteristics thereof. Also, in certain cases, there may be a term arbitrarily selected by the applicant, in which case the meaning thereof will be described in detail in the detailed description of the corresponding specification. Accordingly, the terms used in the present disclosure should be defined based on the meanings of the terms, not on the names of the terms, but on the entire contents of the specification.

The singular expressions herein include plural referents unless the context clearly dictates otherwise.

When an element is referred to as "including" an element throughout the specification, it is to be understood that the element may include other elements as well, without departing from the spirit or scope of the present invention. Also, as used herein, the term "part " refers to a hardware component such as software, FPGA or ASIC, and" part " However, "part" is not meant to be limited to software or hardware. "Part" may be configured to reside on an addressable storage medium and may be configured to play back one or more processors. Thus, by way of example, and not limitation, "part (s) " refers to components such as software components, object oriented software components, class components and task components, and processes, Subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functions provided in the components and "parts " may be combined into a smaller number of components and" parts " or further separated into additional components and "parts ".

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to clearly explain the present invention in the drawings, parts not related to the description will be omitted.

In general, a Scale Invariant Feature Transform (SIFT) algorithm is known as an algorithm widely used as a feature point extraction method in the conventional image analysis unit 122. SIFT is a feature point extraction method that is robust to changes in size and rotation.

FIG. 2 is a diagram for explaining a process of extracting feature points that are resistant to size change. First, a Gaussian filter is applied to the left image and the right image picked up by the first camera 102 and the second camera 104, respectively, to generate a Gaussian pyramid (image pyramid).

The Gaussian filter is expressed by the following equation (1).

(1)

(One)

     In the Gaussian filter, sigma denotes the layer of the Gaussian pyramid, and the Gaussian pyramid generated by applying the Gaussian filter to the sensed image is indicated by (a) in Fig. The image of each level of the Gaussian pyramid is expressed by the following equation (2).

(2)

(2)

The higher the value of the Gaussian pyramid (the larger the σ value), the lighter the image and the blurry the image becomes.

To extract feature points that are strong in size, DoG (Difference of Gaussian) is generated, which is expressed by Equation (3) below. The extreme value of DoG can be obtained by comparing each sample point with points in 26 adjacent layers and adjacent points in the same layer.

(3)

(3)

On the other hand, in order to extract the feature points strong against the change in rotation, a predetermined number of pixels around the feature points obtained as described above are selected to calculate the gradient distribution characteristic for the pixels of the left and right images.

The magnitude (m (x, y)) and the direction (? (X, y)) of the gradient are expressed by the following equation (4).

(4)

(4)

In this way, the magnitude and direction of the gradient for a predetermined number of pixels (for example, 16X16) around the minutiae are obtained, and the direction of the gradient for each pixel is quantized. A histogram may be generated for the direction of the quantized gradient and a gradient direction of the largest value on the histogram may be set for the direction of the corresponding feature point.

The feature point is specified by its coordinate value (x, y), scale value (?) And gradient direction (?), And in order to obtain a feature point strong or unrelated to the direction change, the axis is rotated by the gradient direction obtained above You can give it.

The above describes a general method for obtaining strong feature points for size and direction changes according to the SIFT algorithm. Hereinafter, a general stereo image matching method will be described with reference to FIG.

Fig. 3 is a diagram showing the relationship between the focal points of the first and second cameras, the imaging points of the object to be imaged, and the pixel points on the left and right images.

O _L represents the focus of the first camera, and O _R represents the focus of the second camera. X is the first camera and the second represents the image pick-up point of the imaging object of the camera, X _L is the pixel that the image pick-up point X appears is projected on the left side of the image plane, X _R is displayed is projected on the image plane, the right side is the image pickup point X .

O _L , O _R , and X form a triangle constituting a three-dimensional space formed by the first and second cameras and the object to be imaged. The line segment formed by the intersection of this triangle and the left image plane is called an epipolar line of X _L , and a line segment (not shown) formed by the triangle and the right image plane is called an epipolar line of X _R.

Although the depth of the imaging point X changes to X ₁ , X ₂ , and X ₃ , it is projected to X _L on the left image plane, but projected to other points on the right image plane and these points appear on the epipolar line X _L Lt; / RTI >

The relationship between the pixel point X _L projected on the left image plane and the pixel point X _R projected on the right image plane is expressed by the following equations (5) and (6) through the matrix F obtained through the internal and external parameters of the first and second cameras It is obtained together.

(5)

(5)

That is, if a pixel point position to be projected on the matrix F and the left or right image plane is known, a pixel point projected on the right or left image plane corresponding thereto can be obtained.

However, it is difficult to accurately calculate the position of the pixel point corresponding to the measurement error and the inaccuracy of the camera position and direction. Therefore, the pixel point corresponding to X _L can be detected through the search in the vicinity of the aforementioned epipolar line in the right image plane.

The above description has described the relationship between the pixel point projected on one of the left and right image planes and the pixel point on the other image plane corresponding thereto and a method for obtaining the corresponding pixel point pair.

In order to match the first feature point on the left image picked up by the first camera and the second feature point on the right image picked up by the second camera corresponding thereto in the same way, the first feature point and the periphery of the epipolar line of the first feature point And if the difference is smaller than the predetermined value, it is detected as a corresponding second feature point of the first feature point. However, due to the hardware error and the measurement error of the stereo camera, there are still errors in the pairs of the feature points detected by the above-described method.

FIG. 4 is a diagram illustrating the lines connecting the left and right images side by side and matching feature points obtained by the SIFT algorithm through image matching, and connecting the matched pairs of feature points. In FIG. 4, the red line segment represents a line segment connecting pairs of feature points where a matching error occurs.

According to the stereo image matching method of the embodiment of the present invention, a plurality of first feature points are extracted from a first image (left image) captured by a first camera, and a second image captured by a second camera A plurality of first feature points and a plurality of second feature points are matched with each other and a plurality of feature point pairs of a plurality of feature point pairs matched in a state in which the first image and the second image are arranged side by side, A slope of a line segment connecting one feature point and a second feature point is calculated to determine whether or not a matching error has occurred according to the slope of the calculated line segment, and the pair of feature points having a matching error is excluded from a plurality of matched pairs of feature points.

The image captured by the stereo camera is substantially the same in size as the object to be imaged on the left and right images and the slope of the line segment corresponding to the corresponding (matched) pair of feature points is substantially similar. In an embodiment of the present invention, If there is a difference between the slopes of the line segments connecting the pair of minutiae points having different slopes of the line segments connecting the pair of minutiae points, the matching error is judged and the pair of minutiae where the error occurs is excluded.

5 is a flowchart illustrating a procedure of a stereo image matching method according to an embodiment of the present invention.

In step 502, a plurality of first feature points are extracted from the left image and a plurality of second feature points are extracted from the right image. The method of extracting minutiae points is various and is not limited to a specific method.

In step 504, the first feature point and the second feature point are matched to detect a plurality of matching feature point pairs. In step 506, the slope of a line segment connecting left and right side images and minutiae points of matched pairs of minutiae on the right image is obtained, and a slope value Sm having the highest frequency is obtained from the histogram of slopes.

In step 508, the slope Si is initialized. In step 510, the difference between the slope Si and the most frequently used slope Sm is compared with the predetermined value? _{0. If the} difference is larger than the predetermined value, Are excluded from matched pairs of feature points. I is incremented in step 514, i is compared to the total number of feature point pairs in step 516, and steps 510 to 516 are repeated if i is less than the total number, and if the gradient comparison is complete for all feature point pairs, (Step 518).

6 is a flowchart illustrating a procedure of a stereo image matching method according to another embodiment of the present invention.

In step 602, a plurality of first feature points are extracted from the left image and a plurality of second feature points are extracted from the right image. The method of extracting minutiae points is various and is not limited to a specific method.

In step 604, the first feature point and the second feature point are matched to detect a plurality of matching feature point pairs. In step 606, the slope of a line segment connecting the left image and the minus side image and the minutiae of the matched pair of the feature points on the right image is obtained, and the average value Sa of the slopes is obtained.

The difference between the slope Si and the average slope Sa is initialized at step 608 and the difference between the slope Si and the average slope Sa is compared with the predetermined value? _{0. If the} difference is greater than the predetermined value at step 610, Exclude from feature point pairs. I is incremented in step 614, i is compared to the total number of feature point pairs in step 616, and steps 610 through 616 are repeated if i is less than the total number, and if the gradient comparison is complete for all feature point pairs, (Step 618).

7 is a flowchart illustrating a procedure of a stereo image matching method according to another embodiment of the present invention.

In step 702, a plurality of first feature points are extracted from the left image and a plurality of second feature points are extracted from the right image. The method of extracting minutiae points is various and is not limited to a specific method.

In step 704, the first feature point and the second feature point are matched to detect a plurality of matching feature point pairs. In step 706, the slope of a line segment connecting left and right side-by-side images and the minutiae points of the matched pair of minutiae points on the right image is obtained, and the average value Sa of the slopes is obtained.

The difference between the slope Si and the average slope Sa is initialized at step 708 and the difference between the slope Si and the average slope Sa is compared with the predetermined value? _{0. If the} difference is greater than the predetermined value, Exclude from feature point pairs. In step 714, the average slope is again calculated from the remaining minutiae except the pair of minutiae which are matching errors, and the average slope is updated. This is to correct this because the average slope obtained in the previous step is contaminated due to the slope of the pair of feature points, which is a matching error. The average slope may be updated each time a matching error is detected or the average slope may be updated each time the cumulative number of matching errors becomes equal to or greater than a predetermined value. I is incremented in step 716, i is compared to the total number of feature point pairs in step 718, and steps 710 to 718 are repeated if i is less than the total number, and if the gradient comparison is complete for all feature point pairs, And terminates (step 720).

The stereo image matching method according to another embodiment of the present invention may further include adding the x coordinate value of the first feature point and the second feature point of each of the matched pairs of feature points, In addition to the detection of a matching error on the side of the slope between the pairs of minutiae, by additionally excluding the pair of minutiae having the difference between the x coordinate values larger than the predetermined value from the matched pairs of minutiae points, By further filtering the matching error, it is possible to more accurately match the stereo image.

On the other hand, if the difference between the x coordinate values of the first and second characteristic points is a reference for judging the matching error to be a value larger than the x coordinate deviation between the first image and the second image for the closest object picked up by the stereo camera It is desirable that the normal matching of the proximate object having the largest deviation of the x coordinate value is not judged as an error.

Another embodiment of the present invention may be an apparatus for implementing the above-described stereo image matching method. A stereo image matching apparatus according to an embodiment of the present invention includes one or more processors for executing the steps of the above-described stereo matching method on the one or more processors.

Another embodiment according to the present invention may be provided as a recording medium in which the above-described stereo image matching method is implemented as a program for a computer to be executed and the program is stored in a readable nonvolatile memory.

100 Stereo camera
102 First camera
104 Second camera
120 image processing unit
122 image analysis section
124 image matching unit
The depth map creating unit

Claims (12)

delete A first step of extracting a plurality of first feature points from a first image picked up by a first image pickup apparatus of a stereo camera;
A second step of extracting a plurality of second feature points from a second image captured by the second imaging device of the stereo camera;
A third step of matching the plurality of first feature points with the plurality of second feature points;
A fourth step of calculating a slope of a line segment connecting the first feature point and the second feature point of each of a plurality of pairs of feature points matched in a state where the first image and the second image are arranged side by side; And
And a fifth step of judging whether or not a matching error has occurred according to the slope of the line segment and excluding a pair of feature points having a matching error from the matched pairs of feature points,
The fourth step includes obtaining a histogram of the slope,
The fifth step may include obtaining a slope with the most frequent number through the histogram and excluding a pair of feature points having a slope deviated from a predetermined angle from a slope having the most frequent number from the plurality of matched pairs of feature points,
After the fifth step, a pair of feature points having a difference between an x-coordinate value of the first feature point in the first image and an x-coordinate value of the second feature point in the second image is greater than a predetermined value in each of the matched pairs of feature points Further excludes from the matched pairs of feature points,
Wherein the predetermined value is set to a value larger than an x-coordinate deviation between the first image and the second image with respect to an object closest to the stereo camera.
A first step of extracting a plurality of first feature points from a first image picked up by a first image pickup apparatus of a stereo camera;
A second step of extracting a plurality of second feature points from a second image captured by the second imaging device of the stereo camera;
A third step of matching the plurality of first feature points with the plurality of second feature points;
A fourth step of calculating a slope of a line segment connecting the first feature point and the second feature point of each of a plurality of pairs of feature points matched in a state where the first image and the second image are arranged side by side; And
And a fifth step of judging whether or not a matching error has occurred according to the slope of the line segment and excluding a pair of feature points having a matching error from the matched pairs of feature points,
The fourth step includes a sixth step of obtaining an average slope, which is an average value of slopes,
The fifth step is a matching error detecting process of eliminating, from the matched pairs of feature points, feature point pairs having an inclination that deviates from the average inclination by a predetermined angle,
Calculating the average slope based on the remaining pair of feature points after excluding the pair of feature points in the fifth step, updating the average slope, executing the fifth step again using the updated average slope,
Wherein in the fifth step, the average slope is updated whenever the cumulative number of matching errors in the matching error detection process becomes equal to or greater than a predetermined value,
After the fifth step, a pair of feature points having a difference between an x-coordinate value of the first feature point in the first image and an x-coordinate value of the second feature point in the second image is greater than a predetermined value in each of the matched pairs of feature points Further excludes from the matched pairs of feature points,
Wherein the predetermined value is set to a value larger than an x-coordinate deviation between the first image and the second image with respect to an object closest to the stereo camera.
delete delete delete delete delete delete delete delete delete

Images