KR20200082854A - A method of matching a stereo image and an apparatus therefor - Google Patents

Abstract

The present invention relates to a stereo image matching method and a device thereof. According to one embodiment of the present invention, the method comprises the steps of: differentiating a current image in a vertical and horizontal direction; frequency-analyzing the differentiated image by using a Fourier transform; measuring energy of a specific frequency through the frequency analysis; detecting a low-texture region based on the energy; and matching a stereo image based on a characteristic indicator with respect to the detected low-texture region.

Description

Matching method of stereo image and device performing same {A METHOD OF MATCHING A STEREO IMAGE AND AN APPARATUS THEREFOR}

The present invention relates to a method for matching stereo images and an apparatus for performing the same, and more particularly, to a method for matching stereo images in a low-texture region and an apparatus for performing the same.

3D reconstruction of stereoscopic images through registration of stereoscopic images can be used for virtual reality content, obstacle detection of autonomous vehicles, robot navigation, and motion recognition. Matching of the stereo image may be performed by estimating the parallax between each image, and parallax along with the pixel value of the image may be estimated to determine depth information. There are two methods for estimating parallax of a stereoscopic image, a global matching method and a local matching method.

The local matching method calculates a matching cost for a region limited by a region window among images, and calculates similarity between two stereo images to estimate parallax. The above method calculates the matching cost of each pixel in order to obtain the initial matching cost. In general, an initial matching cost calculation method of a local matching method includes a sum of absolute difference values (SAD) and a sum of squares of difference values (SSD), and a normalized correlation (NCC). However, these filters have a disadvantage that blurring occurs due to a low-pass filter. Therefore, robust mutual information and census conversion can be used for the outer region close to the object boundary in the region-based method. The method using the robust mutual information uses a probability distribution function to estimate the similarity, and the method using the consensus conversion encodes pixels in a window from a character string and compares Hamming distances to calculate the similarity Can.

In addition, the global matching method repeatedly calculates the parallax of the entire image for parallax estimation while minimizing the overall parallax error. The global matching method may generate an initial parallax map through initial matching cost calculation. Thereafter, the method may obtain parallax information by repeating a process of calculating an energy function for the matching cost of the entire image and estimating a parallax in order to obtain minimum energy.

The parallax estimation accuracy of the local matching method is drastically reduced in the low-textured region due to the ambiguous and homogeneous pixel values. The global matching method is relatively accurate because it considers characteristics of the entire image. The semi-global matching method (SGM) can limit the matching direction and reduce the computation amount. However, there is a limit to performance improvement in the low-textured area. In addition, the accuracy of estimation of 3D depth information for a low-texture image with strong homogeneity of pixels and ambiguous distinction is not high. Errors in estimating depth information can cause application failures, which can cause serious side effects. Therefore, it is important to develop a parallax estimation method of a stereo image with high accuracy even in a low-texture region.

The technical problem to be achieved by the present invention is to provide a matching method of a stereo image that requires a small amount of computation for parallax estimation with high accuracy even in a low-texture region.

According to another embodiment of the present invention, to provide a stereo image matching device having the above advantages.

A method of matching stereo images according to an embodiment of the present invention includes: differentiating a current image in vertical and horizontal directions; Frequency-analyzing the differentiated image using a Fourier transform; Measuring energy of a specific frequency through the frequency analysis; Sensing a low-textured region based on the energy; And registering a stereo image based on a characteristic indicator for the sensed low-texture region.

In one embodiment, the low-textured region may be detected when the energy is less than or equal to a predetermined value, and the characteristic indicator may be a Scale-Invariant Feature Transform (SIFT).

In addition, the step of matching the stereo image may further use a characteristic point defined by the characteristic indicators, and the step of matching the stereo image may include the characteristic point on the isopole line in the reference image and the target image. The reference image and the target image may be matched using them.

In one embodiment, in the region other than the sensed low-texture region, the method may further include registering the stereo image using a semi-global matching (SGM) method.

An apparatus for matching stereo images according to another embodiment of the present invention includes: a frequency analyzer that differentiates a current image in a vertical and horizontal direction, and frequency-analyzes the differentiated image using a Fourier transform; An energy measuring unit measuring energy of a specific frequency through the frequency analysis; A low-texture area detection unit that detects a low-texture area based on the energy; And a low-texture region stereo image matching unit that matches a stereo image based on a characteristic indicator to the detected low-texture region.

In one embodiment, the low-textured region may be detected when the energy is less than or equal to a predetermined value, and the characteristic indicator may be a Scale-Invariant Feature Transform (SIFT).

In one embodiment, the low-texture region stereo image registration performer may further use a feature point defined by the characteristic indicators, and the low-texture region stereo image match performer may perform matching in the reference image and the target image. The reference image and the target image may be matched using the characteristic points on the isopole line.

In a region other than the detected low-texture region, a high-texture region stereo image matching performing unit that matches the stereo image using a semi-global matching (SGM) method may be further included.

According to an embodiment of the present invention, by detecting a low-textured region of a current image, and applying a matching method of a stereo image optimized for the low-textured region, a computational amount is reduced and a parallax of the stereo image is accurately estimated, thereby obtaining a 3D image. It is to provide a method of matching stereo images capable of restoring.

According to another embodiment of the present invention, to provide a stereo image matching device having the above advantages.

1 is a block diagram of a stereo image matching device according to an embodiment of the present invention.
2 is a flowchart illustrating a stereo image matching method according to an embodiment of the present invention.
3 illustrates a power spectrum of a specific region in a differential image of a stereo image according to an embodiment of the present invention.
4 is a graph showing a change in detection accuracy of a low-textured region under a specific energy and threshold according to an embodiment of the present invention.
5 is an example of a stereo image matching method according to an embodiment of the present invention.
6 is a graph measuring an error rate of parallax estimation in a low-textured region after estimating parallax by a matching method of a stereo image according to an embodiment of the present invention.
7A to 7E are images showing matching of a stereo image by the above method.
8 is a graph showing detection accuracy of detecting a low-textured region using an embodiment of the present invention and an existing parallax estimation method.
9 is a graph showing an existing SGM method, an efficient large-scale stereo (ELAS) method, and parallax of an entire image according to an embodiment of the present invention, and showing their error rates.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The embodiments of the present invention are provided to more fully describe the present invention to those of ordinary skill in the art, and the following embodiments can be modified in various other forms, and the scope of the present invention is as follows. It is not limited to the Examples. Rather, these examples are provided to make the present disclosure more faithful and complete, and to fully convey the spirit of the present invention to those skilled in the art.

The same reference numbers in the drawings refer to the same elements. Also, as used herein, the term “and/or” includes any combination of any one or more of the listed items.

The terms used in this specification are used to describe the embodiments, and are not intended to limit the scope of the present invention. In addition, even if it is described in the singular in this specification, unless the context clearly indicates the singular, it may include a plurality of forms. Also, the terms “comprise” and/or “comprising” as used herein specify the shapes, numbers, steps, actions, elements, elements and/or the presence of these groups. Does not exclude the presence or addition of other shapes, numbers, actions, elements, elements and/or groups.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description of the embodiments, when it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the invention, detailed description thereof will be omitted.

Matching of stereo images is a method of estimating parallax in order to calculate 3D depth information when the reference image and the target image are at the same position. However, this method is a complicated and difficult process resulting from various problems such as occlusion, noise, and different illuminance levels in the image. In order to obtain a more accurate matching result, limiting conditions such as epipolar limitation, non-uniformity limitation, and similarity limitation are required.

There is a method of using a matching cost function based on a characteristic indicator to estimate the time difference as quickly as possible while increasing accuracy. Low-textured regions can be found by estimating a variable, pixel value change, gradient, energy, entropy, or low's convolution mask in the window. As another method for detecting the low-textured region, a random sample consensus (RANSAN) may be used to divide the region based on a gradient and a change in pixel values. This method can perform accurate detection with a small amount of computation. However, a non-uniform shape may be generated in the plane correspondence process of the RANSAC algorithm.

The stereo image matching method according to an embodiment of the present invention combines a stereo image matching method based on a characteristic indicator for a low-texture region and a SGM (Semi-global Matching) method for regions other than the low-texture region. Hybrid estimation algorithms can be provided. The algorithm of the present invention includes a method of detecting a low-textured region based on energy in a specific frequency range in a window through hierarchical frequency analysis. When the method is used in combination with a scale-invariant feature transform (SIFT) indicator, the parallax in the low-textured region can be accurately estimated.

1 is a block diagram showing a configuration of a stereo image matching apparatus according to an embodiment of the present invention, and FIG. 2 is a flowchart illustrating a stereo image matching method according to an embodiment of the present invention.

Referring to FIGS. 1 and 2 together, the stereo image matching device includes a frequency analysis unit 100, an energy measurement unit 200, a low-texture region detection unit 300, and a low-texture region stereo image matching unit ( 400), and a high-texture region stereo image matching unit 500.

The frequency analysis unit 100 may receive the current image and divide it into differentiations in the horizontal direction and the vertical direction. The divided current image may perform frequency analysis using a Fourier transform (S10). The frequency analysis may be performed on the entire current image including a low-texture region and a high-texture region.

In order to perform frequency analysis on the pixel value change, the pixel value change I'(x,y) may be obtained through Equation 1 below.

The differentiated image may be obtained by adding the differential coefficients to the horizontal and vertical directions of pixels around the current position (x,y) through Equation (1). In order to analyze the characteristics of the low-textured region, frequency analysis may be performed by performing a fast Fourier transform on the differential image I'(x,y).

3 illustrates a power spectrum of a specific region in a differential image of a stereo image according to an embodiment of the present invention. Referring to FIG. 3, it can be seen that the size of the power spectrum according to the frequencies in the low-textured region and the high-textured region is different, and both regions show a large change in the frequency range ω∈[-0.3, 0.3]. Able to know.

Thereafter, the energy measurement unit 200 may measure energy in a specific frequency range based on the frequency analysis result (S20). The specific frequency range may be [-3.0, 3.0], preferably [-0.3, 0.3]. Energy in the above range may be calculated through Equation 2 below.

However, a non-linear intensity change may occur due to a change in the intensity of the object due to the saturation or angle of the stereo camera. In order to reduce the influence of such a change, the present invention can perform a normalization process of generating a unit vector having [0,1] for energy calculated through Equation 2 as shown in Equation 3 below.

In one embodiment, there may be a problem that the low-textured region located around the boundary is not well detected by noise generated from the boundary of the region. Therefore, a local minimum filter can be used to lower the energy of the region in the region located around the boundary. In this case, the low-texture area located around the boundary can be detected more accurately.

Thereafter, the low-texture area detector 300 may detect the low-texture area in the current image based on the energy measured by the method (S30 ). When the energy is less than a threshold, the low-texture region detector 300 may determine a region of the current image as a low-textured region as in Equation 4 below.

A hierarchical method may be used to more accurately detect the low-texture region, and the hierarchical method may be used to calculate energy using a small size window Ws. Local energy can be calculated using the small size window. In one embodiment, a more accurate sensing result may be obtained by removing small and separated regions from regions located around the low-textured region L(x,y), and the low-textured region is a threshold value ( T) It represents the following measured value: The threshold value can be determined by the energy (α) and window size (Ws) of the low-textured region.

4 is a graph showing the detection accuracy of a low-textured region for each energy measured according to an embodiment of the present invention. Referring to FIG. 4, it was confirmed that the change in detection accuracy according to energy is not large except for the Lamp1 image. In the following experimental example, experiment was performed by setting α=0.2.

The low-texture region stereo image matching performer 400 may perform registration of the stereo image based on the characteristic indicators when the current image region is the low-texture region (Y of S30) (S40). The characteristic indicators may include Scale-Invariant Feature Transform (SIFT), Speed up Robust Feature (SURF), Binary Robust Independent Elementary Features (BRIEF), and Oriented FAST and Rotated BRIEF (ORB). If the characteristic indicators have a characteristic of recording in a specific format by counting spatial and frequency characteristics for changes in pixel values of the central pixel and surrounding pixels at one point of the image, It is not necessary to be a character indicator. In one embodiment, in parallax estimation using the characteristic indicators, only points recognized as characteristic points can be depicted in the image and used for registration of stereo images. Since these points are sparse, it is difficult to estimate a dense parallax for a stereo image.

However, the characteristics for the pixels of the entire image can be described, and as shown in FIG. 5, the matching method of the stereo image according to an embodiment of the present invention is the same as the characteristic points of the reference image, and is the same in the target image. The characteristic points on the epipolar lines can be precisely matched through the first nearest neighbor method. Unlike the SGM method, since the stereo image matching method of the present invention does not perform a normalization process to calculate the matching cost in various directions, there is an advantage that the amount of computation does not increase significantly.

6 is a graph measuring an error rate of parallax estimation in a low-textured region after estimating parallax by a matching method of a stereo image according to an embodiment of the present invention.

Referring to FIG. 6, in the parallax estimation method of the present invention, SIFT characteristic indicators may be used to provide the best parallax estimation accuracy in a low-texture region, and an SGM method may be used to perform final parallax estimation. To estimate the final parallax D, the SIFT characteristic indicator in the low-textured region can be used as in Equation 5 below. In addition, the high-texture region stereo image matching unit 450 may perform stereo image matching using a global matching method (SGM) (S50 ).

As shown in FIG. 6, as a result of measuring the RMAE (mean square root deviation) by matching the same image in various methods, the smallest RMAE value in the case of using the SIFT characteristic indicator according to an embodiment of the present invention for the low-textured region It can be confirmed that it has.

7A to 7E are images showing matching of a stereo image by the above method. 7A are input images used in the experiment, FIG. 7B shows a low-texture ground truth image of the FIG. 7A image, and FIG. 7C is obtained by a low-texture area detection method according to an embodiment of the present invention Video. 7D is a parallax ground truth image, and FIG. 7E shows a result image obtained by a matching method according to an embodiment of the present invention combining SGM and SIFT.

In one embodiment, an experiment was performed using a Middlebury stereo dataset to measure the performance of the stereo image matching method of the present invention. The dataset is widely used for stereo vision. In order to measure the detection accuracy of the low-texture region, it is necessary to define a region with little change in pixel value, such as a low-texture region, and a ground truth image is generated for the region. In addition, the F value (the harmonic average of recall and precision) as an index for measuring the accuracy of low-texture area detection was measured as shown in Equation 6.

8 is a graph showing detection accuracy of detecting a low-textured region using an embodiment of the present invention and an existing parallax estimation method, and [8] is "Stereo matching-based low-textured scene reconstruction for alv navigation, "In Image Analysis and Signal Processing (IASP), 2011 International Conference on.IEEE, pp.204-209,2011, the parallax estimation method shows the detection accuracy of detecting low-textured areas, [28] is "Local stereo matching using adaptive local segmentation,", ISRN Machine Vision, vol. 2012, 2012. The parallax estimation method shows the accuracy of detecting a low-textured region.

Referring to FIG. 8, it can be seen that the method according to an embodiment of the present invention shows an improved accuracy of 80% or more in a low-texture region than existing algorithms. However, since the low-texture region has high-frequency components even if it is present in the window, the method according to an embodiment of the present invention has a problem that a low-texture region smaller than the window size cannot be detected. Therefore, in some images, a method according to an embodiment of the present invention may exhibit low accuracy.

In addition, in order to confirm the accuracy of parallax estimation, the ground truth of each pixel is compared, and RMAE (mean square root error) is estimated and shown in FIGS. 9 and 1. 9 is an existing SGM method, an efficient large-scale stereo (ELAS) method, and graphs for estimating the parallax of the entire image according to an embodiment of the present invention and showing their error rates, and Table 1 shows one embodiment of the present invention. The ratio of the low-textured region of the image to which the method according to the example is applied and the estimated error reduction rate of parallax are shown.

9 and Table 1, in some images, it can be seen that the higher the ratio of the low-textured region, the higher the error reduction rate, and the error rate is lower in various images than when using the SGM method and the ELAS method. Can. In particular, it can be seen that the performance increases in proportion to the ratio of the low-textured region. The stereo image matching method according to an embodiment of the present invention has a high estimation accuracy for a low-textured region compared to other parallax estimation and stereo image matching methods. In addition, it shows the low-texture area detection accuracy of 80% or more according to the image.

The present invention described above is not limited to the above-described embodiments and the accompanying drawings, and various substitutions, modifications and changes are possible within the scope of the present invention without departing from the technical field to which the present invention pertains. It will be clear to those who have knowledge.

Claims (12)

Differentiating the current image in the vertical and horizontal directions;
Frequency-analyzing the differentiated image using a Fourier transform;
Measuring energy of a specific frequency through the frequency analysis;
Sensing a low-textured region based on the energy; And
Registering a stereo image based on a characteristic indicator for the sensed low-texture region; Stereo image matching method comprising a. According to claim 1,
The low-texture region is a stereo image matching method that is detected when the energy is below a predetermined value. According to claim 1,
The characteristic indicator is a stereo image matching method that is a Scale-Invariant Feature Transform (SIFT). According to claim 1,
In the matching of the stereo image, the stereo image matching method further using a characteristic point defined by the characteristic indicators. According to claim 1,
The matching of the stereo image comprises matching the reference image and the target image by using the characteristic points on the isoline in the reference image and the target image. According to claim 1,
And registering the stereo image in a region other than the sensed low-texture region using a semi-global matching (SGM) method. A frequency analyzer for differentiating the current image in the vertical and horizontal directions and frequency-analyzing the differentiated image using a Fourier transform;
An energy measuring unit measuring energy of a specific frequency through the frequency analysis;
A low-texture area detection unit that detects a low-texture area based on the energy; And
And a low-texture region stereo image matching unit that matches a stereo image based on a characteristic indicator to the detected low-texture region. The method of claim 7,
The low-texture region is a stereo image matching device that is detected when the energy is below a predetermined value. The method of claim 7,
The characteristic indicator is a stereo image matching device that is a Scale-Invariant Feature Transform (SIFT). The method of claim 7,
The low-texture region stereo image matching performing unit further uses a characteristic point defined by the characteristic indicators. The method of claim 7,
The low-texture region stereo image registration performing unit matches the reference image and the target image by using the characteristic points on the isopolar line in the reference image and the target image. The method of claim 7,
And a high-texture region stereo image registration performing unit that matches the stereo image in a region other than the sensed low-texture region using a semi-global matching (SGM) method.

Images